Re: Web App Scanner - GSoC 2009

[Fyodor <fyodor () insecure org>]

On Sat, Mar 28, 2009 at 02:15:17AM -0300, João wrote:

> The idea is developing a Web app scanner. Before scanning a host and
> finding a web server running on it, it would be very interesting that
> you could have a way to discover which applications are running in
> this web server.

Hi João!  I agree that could be very useful!  In fact, I wrote this in
my Nmap book:

 "Nmap may also grow in its ability to handle web scanning. When Nmap
  was first developed, different services were often provided as
  separate daemons identified by the port number they listen on. Now,
  many new services simply run over HTTP and are identified by a URL
  path name rather than port number. Scanning for known URL paths is
  similar in many ways to port scanning (and to the SunRPC scanning
  which Nmap has also done for many years). Nmap already does some web
  scanning using the Nmap Scripting Engine (see Chapter 9, Nmap
  Scripting Engine), but it would be faster and more efficient if
  basic support was built into Nmap itself."
  --http://nmap.org/book/history-future.html

> I mean, we could scan for installations of wordpress,
> php-myadmin, wikis, web-repos, webmin, OSSIM server, webmail services,
> and many other applications.

Yep!

> I really would appreciate some feedback. 

I think it is a very promising idea.  If possible, it should be done
as an NSE module which scripts can call.  If desired performance
levels can't be reached in LUA for some reason, it could be made into
an NSE C module.  And if even that isn't performant enough, I suppose
it could be built into Nmap (like RPC scan and the like).  But the
first two options are preferred for maintainability reasons.

Also, there may be many NSE scripts which want to see the results of
spidering a web site.  There should probably be a way for each script
to either see all the pages as they are downloaded, or perhaps the web
tree should be saved somewhere (in the filesystem or a database or
maybe a limited tree in RAM) so that scripts can look through them.

Some things which need to be considered and/or discussed:

o Performance is obviously critical.  Think about how you can make the
  best use of web technology such as pipelining, keepalive, head
  requests when you don't need page content, etc.  At the same time,
  it should be configurable so you can say things like "don't request
  more than 1 URI per second" in order to avoid floodign a web server.

o Some scripts only check for URLs (such as app names), whereas others
  such as SQL injection detection scripts need the page content.  You
  need to think about how to handle that.  Maybe the library can have
  a list of interested scripts and pages to each of them when they are
  retrieved.  Or maybe you need to store the whole web tree (with some
  limits so you don't download gigabytes by default or get stuck in
  infinite loops with dynamically generated pages)

o You sent your ideas to the Umit list too.  Note that if it is done
  in Nmap/NSE, Umit can make use of the functionality too.

o We already have some web code available (the url module).  It
  probably makes sense to enhance this rather than start over.

o High speed web authentication brute force cracking might be welcome
  too.

o I love the idea of fingerprinting web applications, though you'll
  need to think about how to collect the DB of URLs, how to deal with
  URLs which can be relocated (in a different directory, say), how to
  deal with determining the right name for the "host" header, etc.

o I'm wondering if it might make sense to apply as some sort of NSE
  Web maven.  Then you could have a list of concrete web-related NSE
  goals.  These might take the form of improvements to the NSE
  infrastructure, module updates, and/or new scripts.

Cheers,
-F

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org