Re: Nsock read buffer

[David Fifield <david () bamsoftware com>]

On Mon, Nov 15, 2010 at 10:23:15PM -0800, David Fifield wrote:
> I started implementing this in the branch
>
> svn co --username guest --password "" svn://svn.insecure.org/nmap-exp/david/nmap-readbuf
>
> I attached a script that shows how it works. nsock_readbytes and
> nsock_readlines (and hence nmap.receive_bytes and nmap.receive_lines)
> return exactly what you ask for, not more and not less. If there are not
> enough bytes available, the functions return an error and the bytes stay
> in the buffer for the next read. If there are more than enough bytes
> available, only the first n bytes or lines are returned, and what's left
> over remains in the buffer.
>
> I think that the behavior of nsock_readbytes and nsock_readlines in my
> branch is just write. My question is, how should nsock_read work?
> nsock_read is used to mean both of these two things:
> 1. Keep reading until timeout or EOF, and return everything.
> 2. Return me some small chunk of data, up to one buffer's worth, but I
>    don't care about the exact number.
> In practice, nsock_receive does (1), except that there a built-in limit
> of 589823 bytes, after which the read always finishes. (Look for the
> comment "spews and spews data" in nsock_core.c.) When people want (2),
> they have been using nsock_readbytes(1), which only does one recv and
> only returns up to a relatively small fixed-size buffer (like recv).
>
> My personal feeling is the nsock_read should work like (2). But we still
> want a function that works like (1) and returns a big block of data all
> at once. Service scan does this for example. But even service scan has
> no use for 589823 bytes; it seems like the caller should be able to set
> this limit. "As many bytes as are available, but not more than X." This
> is pretty close to how nsock_readbytes works now, in that it quits
> reading once enough bytes are available, except that you can still get
> more than you asked for (up to one fixed-size buffer's worth more).

I've been going back and forth over the API for these read functions.
I've identified four use cases:

1. Return everything left on the socket (looped reading), with a
   sanity limit on bytes.
2. Return whatever is immediately available on the socket.
3. Return an exact number of bytes.
4. Return an exact number of lines, with a sanity limit on bytes.

In the branch, nsock_readbytes and nsock_readlines do (3) and (4).
nsock_read does (2), and there's no built-in way to do (1).

I updated some scripts that were, for example, using
socket:receive_bytes(1), to use socket:receive() instead. However I
can't guarantee that this is correct in every case.

Another question is what to do with sanity byte limits. We can't just
read forever if someone keeps writing without sending a newline. Current
nsock_readlines already has such a limit, only it is an undocumented,
unspecified internal constant. If there's no newline you get back a
partial line. An alternative is just to return an error and not violate
the contract that we will return only the requested number of lines.

Some of the migration is tricky. Here is an example from domcom-cmd.nse.
The call to receive_lines(1) is expecting to get more than one line. It
looks like it's expecting a line with "BeginData" and one with
"EndData".

  local status, line = socket:receive_lines(1)
  if ( not(status) ) then return false, "Failed to read line" end
  lines = stdnse.strsplit( "\n", line )
  for _, line in ipairs( lines ) do
    if ( not(line:match("BeginData")) and not(line:match("EndData")) ) then
      table.insert(result, line)
    end
  end

Strictly speaking, this code is incorrect. There's no reason why
receive_lines has to return everything on the socket. It just happens
that line endings fall on packet boundaries so no half-lines are
returned. Since the new receive_lines(1) will always return exactly one
line, this has to be rewritten. What's wanted here is an nsock_read that
does option (1) above. But then it becomes obvious that the code is
relying on an integer number of lines being returned, and you have to
write an explicit loop with receive_lines(1). This is what I have done
but it's more code, and frustrating for a project that was supposed to
simplify code.

David Fifield
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/