Re: http-vhosts.nse ready for beta

[David Fifield <david () bamsoftware com>]

On Thu, Dec 02, 2010 at 08:46:49AM -0800, Carlos Pantelides wrote:
> David:
>
> Thank you for your patience! 
>
> I see the point in your critics, agree with all but number 1. Anyway I accepted the changes, because I understand 
> that is easier for nmap to deal with one "incorrect" file than deal with three "correct" files. I withdraw the 
> changes to unpwdb.lua. But please take a look some time at http://seclists.org/nmap-dev/2010/q4/445
>
>
> > 1. Embed the list of hostnames inside the script as a big
> > array. Remove
> >    hostnames.lst and the hostnamesdb script
> > argument.
>
> Done
>  
> > 2. Change the portrule to be
> >      portrule = shortport.http
> >    Remove the http-vhosts.service argument.
>
> Done
>
> > 3. Don't modify host.targetname. When I comment that out
> > and just use
> >    the Host header as you have already
> > implemented, it works fine.
>
> funny, a forgotten refactoring
>
> > 4. Change the name of the http-vhosts.resource script
> > argument to
> >    http-vhosts.path.
>
> Done

Thanks for taking care of this. I have committed your script. I want you
to make a few more improvements to take care of some things I noticed
while testing.

I had to make some changes to get it to work. First, I had to use the
bypass_cache option in http.head, otherwise the first response was
getting cached and no later requests were effective. I also changed the
output to show only the tested name and possibly a redirect. (The
original name, port, protocol, and IP address already appear in Nmap
output so they don't have to be repeated on each output line.) I made it
use host.targetname (through stdnse.get_hostname) to use the
user-provided name instead of the reverse DNS name when available.

The first thing I want you to change is that there is way too much
output. The few sites I tried in my tests all returned 200 for almost
all names, and all 393 lines of output appear to show the 200 code. But
even when a site returns 200 by default, you can find interesting stuff.
I found this:

| jemmy.mongodb.org: 200
| jfunc.mongodb.org: 200
| jira.mongodb.org: 302 -> http://jira.mongodb.org/secure/Dashboard.jspa
| jite.mongodb.org: 200
| jmeter.mongodb.org: 200
...
| webload.mongodb.org: 200
| webspoc.mongodb.org: 200
| wiki.mongodb.org: 302 -> http://wiki.mongodb.org/homepage.action
| windmill.mongodb.org: 200
| winrunner.mongodb.org: 200

Amid the flood of 200s, those 302s deserve more attention. So I'm
thinking it would be useful to collapse the common states into one
output line, like Nmap does with port states. Like this:

PORT   STATE SERVICE REASON
80/tcp open  http    syn-ack
| http-vhosts:
| jira.mongodb.org: 302 -> http://jira.mongodb.org/secure/Dashboard.jspa
| wiki.mongodb.org: 302 -> http://wiki.mongodb.org/homepage.action
|_391 other names had status 200.

The other thing I noticed is that the behavior is surprising when a name
without a "www" (or other) prefix is used. When scanning insecure.org
(with a shortened hostname list):

PORT   STATE SERVICE REASON
80/tcp open  http    syn-ack
| http-vhosts:
| org: 200
| www.org: 200
| docs.org: 200
|_images.org: 200

Now that I think about it, this particular case is probably a side
effect of my using host.targetname to guess the domain, but I think the
problem stands anyway. A good default behavior would be not to make a
name shorter than two components. (This will still have problems with
co.uk names for example.) If the user provides a name then you always
accept it.

Make sure you make your changes from the latest copy of the script in
Subversion.

David Fifield
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/