Re: NSE HTTP Pipeline implementation

[Joao Correa <joao () livewire com br>]

On Wed, Aug 5, 2009 at 9:55 PM, <doug () hcsw org> wrote:
> On Tue, Aug 04, 2009 at 03:50:35AM -0300 or thereabouts, Joao Correa wrote:
> > The total timing
> > decreased from 11804 secs to 2090 secs when pipelining 10 requests
> > into a single connection and to only 542 when pipelining 40 requests.
>
> Very impressive, congratulations. This could be an excellent performance
> optimization for certain NSE scripts and I am looking forward to seeing how
> aggressively pipelined NSE scripts interact with different HTTP implementations.
Thanks!

> > In servers that don't have support to pipelined requests the response
> > to the first request is given and the connection is closed by the
> > server.
>
> This is what implementations *should* do but not necessarily what they
> *do* do. If this were guaranteed then web browsers could pipeline
> requests with impunity and know they would always find out right away
> whether they need to re-issue their requests.

Yes, I'm aware about that. Also, I've seen some servers where the
response header does not have information like Keep-Alive max
requests. I'm trying to make the function the most robust possible,
but certainly it will only be enough reliable with a lot of testing
around the internet.

> The problem is that not all HTTP implementations properly buffer
> keep-alive connections. Consider a client that pipelines 2 requests
> by sending the following:
>
> "GET /a HTTP/1.1\r\nHost: blah.com\r\n\r\nGET /b HTTP/1.1\r\nHost: blah.com\r\n\r\n"
>
> A correctly implemented HTTP/1.1 implementation can do 1 of 2 things:
>
> * Send the results of the first request with a connection: close header,
>  followed by closing the write direction of the socket [1].
> * Send two HTTP responses, optionally pipelined.
>
> But a broken HTTP server might read in the first string assuming the
> string it reads in will have only 1 HTTP request, process that request,
> and try to read from the socket again. In this case, the client that
> pipelined the first request will "hang" waiting for the second response
> that will never come. When the results of HTTP requests span read()
> calls, such servers are even more broken.

Actually I'm dealing with such situation dynamically handling the
number of simultaneous requests based on the responses received. If we
know the maximum number of simultaneous keep-alive requests, than this
value is used. If we notice that the number of responses is less than
expected, the script decreases the limit of simultaneous pipeline
requests. Also, if the field is not provided, the first attempt is
made with 40 pipelined requests (arbitrary value that can be changed)
and if these 40 responses are not received, than this value is
decreased.

I believe that this implementation makes the code a little bit robust,
and as far as I am testing this around the internet, I'm not having
much trouble. Anyway, I agree that some exotic implementations will
always show up.

The most important is that, if we send 40 requests, and only 1
response is received, the pipeline script will handle exactly like if
all the requests were done serially.
> Almost all HTTP servers properly maintain a buffer when handling
> keep-alive connections, but apparently this still isn't enough to
> justify enabling pipelining in general-purpose http clients. Of
> course NSE scripts are anything but general-purpose http clients
> so have fun.
Thanks Doug. I'm just testing the scripts a little bit more. I expect
that soon I'll "officially" share them. Anyway, the experimental
version is on my nmap-exp brach
(nmap-exp/joao/experimental/nselib/http.lua), just in case of anyone
wanting to take a look =).

João

> Doug
>
> [1] http://httpd.apache.org/docs/1.3/misc/fin_wait_2.html#appendix
>
>
> _______________________________________________
> Sent through the nmap-dev mailing list
> http://cgi.insecure.org/mailman/listinfo/nmap-dev
> Archived at http://SecLists.Org

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org