[NSE] HTTP Library w/ Caching

[Patrick Donnelly <batrick () batbytes com>]

The HTTP Library now caches responses generated from http.get or
http.head (in revision 15243). Because many scripts tend to request
the same object, this helps to prevent sending duplicate requests that
consume network resources needlessly.

The cache is transparent to the script writer in that it will return a
cached result if present when http.get/http.head is called. How the
cache is used may be controlled by options table (described in [1])
passed to the http functions. Three new boolean values are present:
  o bypass_cache -- The contents of the cache is ignored for the
request (method == "GET" or "HEAD")
  o no_cache -- The result of the request is not saved in the cache
(method == "GET" or "HEAD")
  o no_cache_body -- The body of the request is not saved in the cache
(method == "GET" or "HEAD")

The size of the cache is by default 1 MB. The size can be changed
using the script arg (--script-args) http-max-cache-size (e.g.
--script-args http-max-cache-size=1e8).

In some recent simple benchmarks (over a number of different hostgroup
sizes) the cache had a large improvement on bytes saved over the
network. For a cache size of 1e5 (100 Kilobytes), the cache would
generally save about 50% the traffic that would normally be sent
during the scan. For larger host groups this number decreases. For a
cache size of 1e8 (100 Megabytes), the cache would generally save
about 200% as much traffic over the network (that is, the cache saved
twice as much traffic as was sent in the scan).

[1] http://nmap.org/nsedoc/lib/http.html#request

-- 
-Patrick Donnelly

"Let all men know thee, but no man know thee thoroughly: Men freely
ford that see the shallows."

- Benjamin Franklin

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org