Nmap Development mailing list archives
Re: [nmap-svn] r28585 - nmap/todo
From: Patrick Donnelly <batrick () batbytes com>
Date: Wed, 16 May 2012 00:41:50 -0400
On Tue, May 15, 2012 at 5:16 PM, <commit-mailer () insecure org> wrote:
Author: fyodor Date: Tue May 15 14:16:49 2012 New Revision: 28585 Log: Add a todo entry about fixing NSE pipelining Modified: nmap/todo/nmap.txt Modified: nmap/todo/nmap.txt ============================================================================== --- nmap/todo/nmap.txt (original) +++ nmap/todo/nmap.txt Tue May 15 14:16:49 2012 @@ -123,6 +123,27 @@ o Review NSE-based port scanning and RST idle scan. http://seclists.org/nmap-dev/2011/q2/307. [Henri and Hani?] +o Investigate why http pipelining so often doesn't work in NSE + scripts, and often NSE ends up reverting to one request at a time. + Scripts may not be using it correctly, and also we wish it were more + transparent and there wasn't this big API divide between pipeline + and non-pipeline. We just want it send requests as fast as it can, + and get a callback when there's a response. Maybe the http library + buffers them, or pipelines them, or blocks the http.get call until + there's more room. It just seems to always degenerate to 1 request + at a time. For example: + sudo nmap --script=http-enum bamsoftware.com -p80 -d2 + quickly (within a few seconds) gives: + NSE: http-enum: Searching for entries under path '' (change with 'http-enum.basepath' argument) + NSE: Total number of pipelined requests: 2081 + NSE: Number of requests allowed by pipeline: 100 + NSE: Received only 41 of 100 expected responses. + Decreasing max pipelined requests to 41. + NSE: Received only 1 of 41 expected responses. + Decreasing max pipelined requests to 1. + 100 may a wildly high number of requests to attempt to pipeline. + And then something else probably goes wrong after it decides 41 is okay.
My earlier thoughts on the subject (for those reading nmap-svn and curious): http://seclists.org/nmap-dev/2011/q4/186 I think callbacks in general will be, from a programmatic perspective, inconsistent with the rest of our libraries. In particular, the nsock binding does not expose the callback mechanism to scripts. I would like to see a design that avoids callbacks. One possibility is to keep http.get as a "blocking" call but scripts can achieve http pipelining by having multiple workers. I don't think this is too unreasonable as we don't want more than ~5 pipelined requests anyway. So, the http library will automatically pipeline requests to the same host coming from multiple threads but one thread can only have one request pending at a time. -- - Patrick Donnelly _______________________________________________ Sent through the nmap-dev mailing list http://cgi.insecure.org/mailman/listinfo/nmap-dev Archived at http://seclists.org/nmap-dev/
Current thread:
- Re: [nmap-svn] r28585 - nmap/todo Patrick Donnelly (May 15)