Re: NSE HTTP Pipeline implementation

[doug () hcsw org]

On Tue, Aug 04, 2009 at 03:50:35AM -0300 or thereabouts, Joao Correa wrote:
> The total timing
> decreased from 11804 secs to 2090 secs when pipelining 10 requests
> into a single connection and to only 542 when pipelining 40 requests.

Very impressive, congratulations. This could be an excellent performance
optimization for certain NSE scripts and I am looking forward to seeing how
aggressively pipelined NSE scripts interact with different HTTP implementations.

> In servers that don't have support to pipelined requests the response
> to the first request is given and the connection is closed by the
> server.

This is what implementations *should* do but not necessarily what they
*do* do. If this were guaranteed then web browsers could pipeline
requests with impunity and know they would always find out right away
whether they need to re-issue their requests.

The problem is that not all HTTP implementations properly buffer
keep-alive connections. Consider a client that pipelines 2 requests
by sending the following:

"GET /a HTTP/1.1\r\nHost: blah.com\r\n\r\nGET /b HTTP/1.1\r\nHost: blah.com\r\n\r\n"

A correctly implemented HTTP/1.1 implementation can do 1 of 2 things:

* Send the results of the first request with a connection: close header,
  followed by closing the write direction of the socket [1].
* Send two HTTP responses, optionally pipelined.

But a broken HTTP server might read in the first string assuming the
string it reads in will have only 1 HTTP request, process that request,
and try to read from the socket again. In this case, the client that
pipelined the first request will "hang" waiting for the second response
that will never come. When the results of HTTP requests span read()
calls, such servers are even more broken.

Almost all HTTP servers properly maintain a buffer when handling
keep-alive connections, but apparently this still isn't enough to
justify enabling pipelining in general-purpose http clients. Of
course NSE scripts are anything but general-purpose http clients
so have fun.

Doug

[1] http://httpd.apache.org/docs/1.3/misc/fin_wait_2.html#appendix

Attachment: _bin
Description:

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org