Nmap Development mailing list archives
Proposal for adding buffer to Nsock read functions.
From: venkat sanaka <venkatsanaka () gmail com>
Date: Thu, 13 Aug 2009 19:26:48 +0530
Hi all I have finished writing the proposal for adding a buffer to Nsock read functions. I would like to receive comments, suggestions and any alternatives for implementing this idea to Nsock library. Regards Venkat ............................................................................................................................ Nsock Buffer Management For Read Functions. Current Implementation: At present Nsock API provides three different functions to read, namely nsock_readlines,nsock_readbytes and nsock_read. nsock_readlines: This reads atleast n lines (terminated with \n, which of course includes \r\n), or read up until EOF or timeout, whichever comes first. nsock_readbytes: This works similar to nsock_readlines except that it reads atleast n bytes instead of n lines. nsock_read: This is like the read system call. You get some amount of data, but you don't have control over it. Problem with current implementation: The main problem of current implementation is lack of buffer to read functions in Nsock. I will explain the need of buffer to Nsock read functions by stating some situations, where we have difficulties in using the current Nsock read API. 1. Ncat should send one line at a time in when --delay option is given. For this Ncat should read data from the STDIN and store it in a buffer. Then one line at a time is read from the buffer and write that line to the socket for sending. This is not possible now, as there is no read function in Nsock which gives exactly one line of read data and store the rest of the read data in buffer for sending later, in the same way as above. 2. The size of the socks4 response is "7" bytes. So we need to read exactly "7" bytes from the buffer for processing the socks4 response for validity and if socks4 response is valid,rest of the buffered read data is written to the STDOUT.SO, buffer is needed here for storing the extra bytes (other than 7) from being lost. 3. Similar problems discussed in this post. http://seclists.org/nmap-dev/2009/q2/0673.html Buffering problems in ssh2.lua.This problem occured when an SSH-2 packet is split across two TCP packets.Then the read only gets the first half, and subsequent operations on the partial packet lead to the crash. So we have to do the operations like parsing or checking the validity of the packet etc; only when we received the exact amount of data needed. Untill then, the data read should be buffered. 4. When parsing a http header,one line at a time is read looking for "\r\n" or "\n". Suppose we received a partial http response like the below one in the first network read(). HTTP/1.0 200 OK \r\n Accept: Here one line at a time should be read from the buffer while parsing and store the incomplete line ("Accept: ") in the buffer from being lost, which is parsed with the received data from the subsequent network read()s. The above exmaples show the importance of having a buffer to read functions and how it can be used in some situations where we need exact amount of data to be read from network/STDIN. These applications can work around this by implementing their own buffer (as NSE does), but it will be more convenient to be built into the Nsock library itself. So the nsock_readlines and nsock_readbytes functions should give exactly specified no. of lines and bytes respectively and should also store the rest of the data from being lost in a buffer for later operations. Both of these behaviours are lacking in Nsock API. Existing similar solutions: 1. nmap.receive_buf in NSE which reads upto a particular delimiter (pattern/separator). http://nmap.org/nsedoc/modules/nmap.html#receive_buf 2. socket_buffer in http.h, which is a stateful buffer. This buffer type is used in socket_buffer_read, socket_buffer_readline, socket_buffer_readcount and socket_buffer_remainder functions of http.c to return exactly the specified no. of lines and bytes and store the remaining data in the same buffer. Proposed implementation: The basic idea of this proposal is to add buffer to Nsock read functions without changing much in the usage of the Nsock API. A stateful buffer similar to socket_buffer in http.h, which should have state information like no of bytes read upto and the no of bytes remaining in the buffer etc; is proposed to use for Nsock read functions too. This is how Nsock read functions should work after adding the buffer to them. For example: * I use nsock_read_bytes(7) because I want exactly 7 bytes for SOCKS4. * The handler for read_bytes does a recv and gets whatever is on the socket, let's say it gets 36 bytes. * But it returns only 7 bytes and store the remaining 29 bytes in a buffer. (Currently, nsock_read_bytes will return all 36 bytes even though 7 is specified while calling.) * Next if nsock_read_bytes(15) is called, and instead of reading from the network, it would immediately return 15 bytes from the buffer. * If the buffer doesn't contain requested no. of bytes, then it would do as many as recvs needed to get a data of requested size. * nsock_read_lines() also works in a similar way as above, returning the requested no. of lines. Buffer:A new stateful buffer will be added in msiod structure. Note: For rest of the proposal,the usage of word "buffer" refers to "read_buf" added in msiod. New Additions: Data Structure Member 1. mspool int buffered_read_events No of read events whose buffer is not empty (means some data is buffered). 2. read_buf(new structure) char *start,*end char buffer[BUFSIZ] int nbytes /* No. of lines in the buffer */ int nlines /* No. of bytes in the buffer */ This is a new stateful buffer added to msiod. Nsock read functions stores the total data read from that iod in this buffer instead of filespace buffer present in msevent. Only requested no. of bytes or lines is moved to the filespace buffer of that event. For eg: We want 10 bytes of data and we used nsock_readbytes(10) function for this. But the no. of bytes read from the network/STDIN is 20, now this 20 bytes are stored in read_buf of msiod and out of this 20 bytes, the only requested 10 bytes are added to filespace buffer of that read event, which we can get using nse_readbuf() function. Finally to say, the filespace contains the data that is going to be returned to the user. The iod buffer contains the data that is being kept until the next read. 3. msiod struct read_buf *buf; If an read event whose buffer has some left over data,is going to be deleted, then it stores that remaining data in this buffer of its msiod,so that this data can be passed to subsequent read events created for that msiod. 4. event_lists gh_list buf_read_events New list which contains the read events with buffered data,so that we can handle them in this order: connect => buf_read =>read=>write=>timer Changes in Behaviour of functions: 1. handle_read_result(): Store the data in read_buf of msiod instead of filespace buffer of msevent and add specified no of bytes/lines to filespace buffer. 2. nsock_readbytes() and nsock_readlines(): After creating a new read event in these functions, we check if read_buf of msiod associated with this new event is not empty (has some data in it from previous read events of that iod.) and also look that buffer has enough no. of bytes/lines requested, by checking the value of read_buf->nbytes or read_buf->nlines. If yes, then no need of reading from the network/STDIN.So, * The event is added to buf_read_events instead of read_events, using gh_list_append. * Increment the count of buffered_read_events in mspool. 3. nsock_loop(): Check for any buffered_read_events in mspool before calling wait_for_events(). Process Flow: nsock_readbytes(nbytes)/nsock_readlines(nlines) | | msevent_new | | nsp_add_event | | gh_list_remove() from read_events and gh_list_append() to buf_read_events. (The last one happens only when the the read buffer of iod associated with that read event is not empty). The above process takes place in nsock_readbytes(nbytes) and nsock_readlines(nlines) functions. The below one is the process flow in the Nsock loop. nsock_loop() | | Check for any buffered_read_events of mspool | | |yes |No | | | wait_for_events(select) iterate_through_events<------------------------| | | handle_read_result | | do_actual_read Test Program: ....................................................................................................................................................... _______________________________________________ Sent through the nmap-dev mailing list http://cgi.insecure.org/mailman/listinfo/nmap-dev Archived at http://SecLists.Org
Current thread:
- Proposal for adding buffer to Nsock read functions. venkat sanaka (Aug 13)