Nsock read buffer

[David Fifield <david () bamsoftware com>]

On Wed, Sep 15, 2010 at 10:28:46PM -0500, Kris Katterjohn wrote:
> On 09/15/2010 10:00 PM, David Fifield wrote:
> > I would like to eventually remove receive_buf completely. I believe that
> > such a function should be built into Nsock itself. There should be a
> > buffer associated with each iod so that you can ask for a certain number
> > of bytes (or lines, etc.) and get exactly that many, no more, with those
> > left over remaining in the buffer. The ssh pattern would be
> >
> > packet_size = socket:receive_exactly(4)
> > -- Unpack size.
> > packet = socket:receive_exactly(packet_size)
> >
> > It would be an error if either call failed; they would return an error
> > if they can't supply enough bytes. There would also be a call to return
> > everything available on the socket, clearing the buffer. There is such a
> > socket buffer (not using Nsock) in ncat/http.c.
> >
> > The main implementation difficulty that I have considered is that you
> > can't just use the select function to detect when an iod has input
> > pending, you also have to check if there is anything in the buffer. We
> > would also need new name for the new "exact" functions. There are a lot
> > of script that use receive_bytes(1) that expect to get as many bytes as
> > are available.
>
> This involves changing the behavior of existing functions but I think is a
> better solution (or at least the beginning of one):
>
> receive(): Give me whatever is available
>
> receive_bytes(n): Give me exactly n bytes or fail somehow (maybe returning an
> error but with the available bytes, or whatever is decided)
>
> receive_lines(n): Give me exactly n lines or fail like receive_bytes()

I started implementing this in the branch

svn co --username guest --password "" svn://svn.insecure.org/nmap-exp/david/nmap-readbuf

I attached a script that shows how it works. nsock_readbytes and
nsock_readlines (and hence nmap.receive_bytes and nmap.receive_lines)
return exactly what you ask for, not more and not less. If there are not
enough bytes available, the functions return an error and the bytes stay
in the buffer for the next read. If there are more than enough bytes
available, only the first n bytes or lines are returned, and what's left
over remains in the buffer.

Here's the output running against this server:
$ ncat -l -k 1234 --sh-exec 'yes ABCDEFGHIJKLMNOPQRSTUVWXYZ'
1234/tcp open  hotline syn-ack
| readbuf:
| receive_bytes(5)  -> 5 "ABCDE"
| receive_bytes(10) -> 10 "FGHIOJKLMN"
| receive_lines(1)  -> 13 "OPQRSTUVWXYZ\
| "
| receive_lines(2)  -> 56 "ABCDEFGHIOJKLMNOPQRSTUVWXYZ\
| ABCDEFGHIOJKLMNOPQRSTUVWXYZ\
| "
| receive()         -> 99
|_receive()         -> 56472

"\" at the end of a line stands for "\n". The first actual network read
returned 183 bytes, which was enough for receive_bytes(5),
receive_bytes(10), receive_lines(1), receive_lines(2), and receive().
Compare this to what the output is in the trunk (literal strings omitted
because they're too long):

1234/tcp open  hotline syn-ack
| readbuf:
| receive_bytes(5)  -> 183
| receive_bytes(10) -> 8192
| receive_lines(1)  -> 58416
| receive_lines(2)  -> 30408
| receive()         -> 31856
|_receive()         -> 34752

I think that the behavior of nsock_readbytes and nsock_readlines in my
branch is just write. My question is, how should nsock_read work?
nsock_read is used to mean both of these two things:
1. Keep reading until timeout or EOF, and return everything.
2. Return me some small chunk of data, up to one buffer's worth, but I
   don't care about the exact number.
In practice, nsock_receive does (1), except that there a built-in limit
of 589823 bytes, after which the read always finishes. (Look for the
comment "spews and spews data" in nsock_core.c.) When people want (2),
they have been using nsock_readbytes(1), which only does one recv and
only returns up to a relatively small fixed-size buffer (like recv).

My personal feeling is the nsock_read should work like (2). But we still
want a function that works like (1) and returns a big block of data all
at once. Service scan does this for example. But even service scan has
no use for 589823 bytes; it seems like the caller should be able to set
this limit. "As many bytes as are available, but not more than X." This
is pretty close to how nsock_readbytes works now, in that it quits
reading once enough bytes are available, except that you can still get
more than you asked for (up to one fixed-size buffer's worth more).

So I'm thinking of a function like
        nsock_read_upto(size_t n)
Service scan would call this with n = 10240 or thereabouts. Another
option is to add an argument to nsock_read to give it nsock_read_upto
semantics. Current nsock_read is much like nsock_read_upto(589823).

David Fifield

Attachment: readbuf.nse
Description:

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/