Re: Updates to http-enum.nse

[Ron <ron () skullsecurity net>]

Hi all,

I'm cross-posting this to nmap-dev and yokoso-devel, since it sort of 
affects both the products. I don't really know what etiquette is 
regarding cross-posting, so hopefully that isn't a no-no. :)

On 08/21/2009 02:49 AM, Fyodor wrote:
> On Thu, Aug 20, 2009 at 11:57:43AM -0500, Ron wrote:
> > Me and one of my minions at work (Andrew -- same guy who I did the iis
> > unicode script with) have put a lot of work into improving http-enum.nse
> > (in case that wasn't obvious from all the http.lua errors I've been
> > posting). Rob's script was a great start, but we made a ton of improvements:
> > - Cleaned up the code, put a bunch of it into functions
> > - Support for many more HTTP status codes
> > - Improved detection for 404 pages (especially those that return 200) --
> > we still have some more work to do on this, but it's getting there
> > - More intelligent usage of HEAD vs. GET requests
> > - Ability to parse external fingerprint file (attached)
>
> Thanks Ron and Andrew!  Those sound like exciting changes!  It worked
> well in my testing, though the results against scanme.nmap.org are
> basically false positives (we might wan to consider only showing 200
> results by default--I'm not sure).
I agree that the script should only show 200 by default, but give an 
option to show others. That makes sense. You're going to miss hidden 
folders, by by default Apache hides certain things so that kinda makes 
sense.

For the average person, just giving 200 makes sense, and for the 
non-average who understand the extra errors, they can enable a script-arg.

> The format of the fingerprint file is a bit questionable.  Comments
> lines starting with '#' are parsed and then printed in the script
> output when paths given later in the file are discovered.  I realize
> you didn't invent this format, but it is so simple that it could
> easily be improved.  For example, each path could include the
> description on the same line.  Or there could be a keyword introducing
> the description on one line, followed by the paths on their own lines
> as they are now.  Then we could use comments for notes which we don't
> want parsed and printed by http-enum.  In deciding on the format, it
> may be worth thinking about how it could be extended if we want to
> include more information later (just as an idea off the top of my
> head, we might want to later indicate what status code we look for to
> address issues such as http-enum reporting that scanme.nmap.org has
> "TeraStation PRO RAID 0/1/5 Network Attached Storage" just because
> /cgi-bin/image/shikaku2.png shows forbidden).
The format is a bit odd, and I'd talked to Kevin a bit about changing it 
to have the ability to AND things together (so if a host contains these 
4 files that have common names, but are unique as a combo, it'll check 
for all of them). At the same time, maybe we can consider other format 
changes. Maybe I'll think about it and propose something, see where that 
goes.

Regarding including the status code, that is sort of an issue with the 
different purposes of Yokoso vs Nmap. Yokoso doesn't care what the 
status code is, but rather a binary: the user has been to the page, or 
the user hasn't. Nmap, on the other hand, cares of the page exists. I'm 
sure there must be a way to get the best of both worlds, of course -- 
maybe optional fields or something?

> Another serious issue involves inclusion of the Yokoso DB.  You say:
>
> > That last point is the interesting one, to me -- we use the same file
> > format as the Yokoso project (by Kevin Johnson and others, from Intel
> > Guardians). This lets us leverage their fingerprints as well (and
> > they've given me permission to include a copy of their fingerprints
> > file, too,
>
> That was nice of them, but it is important to get more clarification
> and more explicit permission whenever we include 3rd party code or
> data into Nmap.  I hate dealing with copyright stuff as much as the
> next guy, but we really need to be very careful about this sort of
> thing.  When they say we can include the DB with Nmap, what does that
> really mean.  Remember that Nmap is open source, so people can
> incorporate parts into other projects or fork Nmap under a different
> name.  A strict reading of "you may include this file within Nmap"
> would not allow such things, which would mean that part of Nmap is not
> open source.  Also, a strict reading might mean that we can only
> include the file and not modify it (create derivative works).  In
> general, we can put third party code/data in with Nmap if it is given
> to us under one of the following licenses (either via special
> permission or because the code is already under such a license):
>
> o Public domain -- that means people can do whatever they want with it.
>
> o BSD-style (includes MIT license, Lua license, etc.) - preferably
>    2-clause variant.  If it has the advertising clause, we need to
>    mention it explicitly in the man page
>    (http://nmap.org/book/man-legal.html#third-party-soft) and
>    potentially other places.
>
> o Nmap license - if they're OK with us distributing it under the terms
>    of the Nmap license (http://nmap.org/data/COPYING), that is OK too.
>
> So if they let us use the data under one of these licenses, inclusion
> with Nmap is OK.  In any case, the license permission granted needs to
> be included (described) at the top of the file.  We only need license
> rights to the list of URL paths and descriptions, and not the rest of
> Yokoso.
>
> Note that even when a data file isn't licensed appropriately for
> distribution with Nmap, we can generally point to it in the
> documentation (e.g. if it is a URL somewhere) so users can download
> and use it.
I'll leave that question for the Yokoso guys to answer.

> > One thought I had -- http-enum.nse and Yokoso sort of have different
> > points. http-enum.nse is designed for finding common locations, like
> > /icons, /scripts, /test, etc, and Yokoso is designed for fingerprinting
> > common web apps. So, for that reason, it might make sense to put it in a
> > different script that the user can run separately. Or maybe not. I'm
> > happy with going either way.
>
> I'm not sure, but my gut reaction is that with a good file format
> (which doesn't have to be too complex), we could probably combine the
> Yokoso DB with the existing enum DB.  There is also a DB by HD Moore
> (a NASL script he wrote) which I hope to request permission for us to
> use.
>
> If we end up with more URLs than we want to scan by default, we could
> look at either splitting it up into multiple scripts or introducing
> NSE arguments to select categories of paths to try.
You're probably right about combining the databases. My concern is that 
it'll make it more difficult to update when Yokoso is updated -- Yokoso 
promises to be an active project, and if they're updating the 
fingerprints and we've combined the fingerprints, we might run into 
versioning problems.

I think the best bet is to have multiple fingerprint files of the same 
format. Yokoso, defaults, extended, etc. Then the script can load 
whatever it sees fit.

I'd definitely like to use HD's database if it's for the same purpose!

> > I plan to move the hardcoded tests from http-enum.nse into their own
> > file, too, once I'm happy with how it's working.
>
> Or maybe one combined file.
We'll see. :)

> Thanks again for all your efforts on this!
No problem!

Is it ok if I commit what I have in http-enum.nse? I'll leave out the 
yokoso file for now, and provide instructions to download it. I strongly 
suspect that the current http-enum.nse works at least as well as the 
previous version.

> Cheers,
> -F
Thanks!
Ron

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org