Nmap Development mailing list archives
Re: NSE HTTP Pipeline implementation
From: doug () hcsw org
Date: Thu, 6 Aug 2009 00:55:46 +0000
On Tue, Aug 04, 2009 at 03:50:35AM -0300 or thereabouts, Joao Correa wrote:
The total timing decreased from 11804 secs to 2090 secs when pipelining 10 requests into a single connection and to only 542 when pipelining 40 requests.
Very impressive, congratulations. This could be an excellent performance optimization for certain NSE scripts and I am looking forward to seeing how aggressively pipelined NSE scripts interact with different HTTP implementations.
In servers that don't have support to pipelined requests the response to the first request is given and the connection is closed by the server.
This is what implementations *should* do but not necessarily what they *do* do. If this were guaranteed then web browsers could pipeline requests with impunity and know they would always find out right away whether they need to re-issue their requests. The problem is that not all HTTP implementations properly buffer keep-alive connections. Consider a client that pipelines 2 requests by sending the following: "GET /a HTTP/1.1\r\nHost: blah.com\r\n\r\nGET /b HTTP/1.1\r\nHost: blah.com\r\n\r\n" A correctly implemented HTTP/1.1 implementation can do 1 of 2 things: * Send the results of the first request with a connection: close header, followed by closing the write direction of the socket [1]. * Send two HTTP responses, optionally pipelined. But a broken HTTP server might read in the first string assuming the string it reads in will have only 1 HTTP request, process that request, and try to read from the socket again. In this case, the client that pipelined the first request will "hang" waiting for the second response that will never come. When the results of HTTP requests span read() calls, such servers are even more broken. Almost all HTTP servers properly maintain a buffer when handling keep-alive connections, but apparently this still isn't enough to justify enabling pipelining in general-purpose http clients. Of course NSE scripts are anything but general-purpose http clients so have fun. Doug [1] http://httpd.apache.org/docs/1.3/misc/fin_wait_2.html#appendix
Attachment:
_bin
Description:
_______________________________________________ Sent through the nmap-dev mailing list http://cgi.insecure.org/mailman/listinfo/nmap-dev Archived at http://SecLists.Org
Current thread:
- NSE HTTP Pipeline implementation Joao Correa (Aug 03)
- Re: NSE HTTP Pipeline implementation David Fifield (Aug 04)
- Re: NSE HTTP Pipeline implementation Patrick Donnelly (Aug 04)
- Re: NSE HTTP Pipeline implementation doug (Aug 05)
- Re: NSE HTTP Pipeline implementation Joao Correa (Aug 05)
- Re: NSE HTTP Pipeline implementation Joao Correa (Aug 09)
- Re: NSE HTTP Pipeline implementation Joao Correa (Aug 05)
- Re: NSE HTTP Pipeline implementation David Fifield (Aug 04)