Re: [NSE] http matching library

[David Fifield <david () bamsoftware com>]

On Sat, Apr 30, 2011 at 08:49:54AM +0200, Patrik Karlsson wrote:
> Given the recent increase in http/web scripts I thought I would put
> some work out on the list I did a while back. I started working on it
> right about the time Ron did his big overhaul of the http-enum script.
> [1] My idea was to implement what the http-enum script does today, but
> de-couple the probes and matches from each other. The response wasn't
> very positive at the time.
>
> Anyway, I did some more work on it and ended up creating a http-match
> library which pretty much does regexp matching based on rules created
> on-the-fly or loaded from a file. There's a script called http-fp that
> implements the decoupled probe and match approach on something similar
> to http-enum. It does so by loading all the rules (probes and matches)
> from a file (nselib/data/urls.txt). Once the probes have run matchers
> are used to process the response. Each matcher don't necessarily have
> to run for each probe as they can be restricted by url or category. In
> addition to regexp's a match can contain Lua code that will be
> executed on the http response received from the server.
>
> To be clear, I'm not suggesting we add this script, but seeing the
> increase of small scripts that do different types of matching lately,
> I think someone may find the library useful. Also, please consider the
> code for what it is (maybe something useful) as it is not as well
> documented as what I usually put out to the list, or as finished as I
> would like. In order to make some of the matches I needed I factored
> the cookie code out from http.lua into cookie.lua. I'm attaching this
> as well. To better understand how it all works, the http-fp script may
> be useful. In order to get it to run you need to drop the url.txt into
> nselib/data/urls.txt and copy both libraries (httpmatch.lua and
> cookie.lua) into nselib.

This is a really intriguing idea that may be worth revisiting. I hadn't
tried this script before, but I just did and here's some sample output.

PORT   STATE SERVICE REASON
80/tcp open  http    syn-ack
| http-fp:
|   path = /index.html
|     Additional information: SHA1 hash: 7B96DD715864C1E304D7C5DD2909F86F29D6C79A
|   path = /
|     Additional information: SHA1 hash: 7B96DD715864C1E304D7C5DD2909F86F29D6C79A
|   path = /images/
|_    Additional information: SHA1 hash: 32133E29EF45A645C20C200EA3D6144850200783
Nmap done: 1 IP address (1 host up) scanned in 3.58 seconds

Here's http-enum output for the same host:

80/tcp open  http    syn-ack
| http-enum:
|_  /wiki/: Wiki
Nmap done: 1 IP address (1 host up) scanned in 12.74 seconds

http-fp also had some URL-specific information in some cases:

PORT   STATE SERVICE REASON
80/tcp open  http    syn-ack
| http-fp:
|   path = /icons/
|     Authentication: The request was forbidden (403)
|     Additional information: Directory exists
|   path = /robots.txt
|     Additional information
|         SHA1 hash: DE2EFA4C8968A94BA466E9B52E68D9DE7786E7FD
|         Robots content
|           #

I think we may want to add the http-match library. Ideally it can use
the existing http-enum database, enhancing it with decoupled matches. It
could also then be used for other HTTP matching needs.

Making a separate cookie library is fine. It would be better to find a
way to share the parsing functions like read_token.

David Fifield
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/