NSE HTTP Pipeline implementation

[Joao Correa <joao () livewire com br>]

Hello Guys,

I've been playing with HTTP pipelines inside NSE these days. All the
files I've implemented are place in nmap-exp/joao/experimental, and
they are mainly nselib/http.lua, scripts/pipeline2-sql-injection.nse
and scripts/pipeline-http-enum.nse. Also the file scripts/test3.nse is
a simple example of how to use http pipeline inside scripts.

The first test I've made was using sql-injection.nse. What I can say
is that certainly pipeline helped in many cases. My "benchmark"
website provided around 1100 malicious queries to sql-injection, what
required a huge number of requests to check if any was vulnerable.
This website also supported pipeline requests. The total timing
decreased from 11804 secs to 2090 secs when pipelining 10 requests
into a single connection and to only 542 when pipelining 40 requests.

In servers that don't have support to pipelined requests the response
to the first request is given and the connection is closed by the
server. As the pipeline implementation is able to detect such
behaviour and to adapt the number of requests, there were no
performance differences for these tests, showing that pipeline
implementation does not introduce any prejudice.

The numbers seems amazing, but anyway, some problems were found.

My second experiment was applying pipeline to http-enum.nse, which
makes 41 requests. The first thing I could notice is that the original
http-enum.nse (without pipeline) performed better. The problem is that
pipelined requests rely on connections with keep-alive, what means
that the server will not close the connection once the data
transference is complete, it will wait for another request until the
connection times out. As we are performing only a small number of
requests, the defaults http.lua timeout values started being too
heavy.

I've decreased the timeout number manually, and so I could notice that
the scans started running much faster. With timeout set to 3 seconds,
all the tests I've performed with pipeline were, at least, equal in
timing to the test without pipelining.

I believe that decreasing the timeout might not be the best solution.
I'm thinking about writing a request function that is aware about the
number of pipelined requests made, and that is able to count the
number of requests received, being able to identify the when all the
responses were received and that it is possible to send new requests.
I believe that this approach will also make better use of the open
socket, not trying to use a new one when the pipeline limit is
reached.

Do someone have a better idea?

Thanks,
João

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org