Re: [RFC] Improve NSE HTTP architecture.

[Djalal Harouni <tixxdz () opendz org>]

On Thu, Jun 16, 2011 at 08:06:30AM +0200, Patrik Karlsson wrote:
> I think there some great ideas in here and I'm of course happy that the http matcher wasn't a waste and that it may 
> come in use!
Yes the http matcher library extends the regex capabilities.

> A few things came to mind when first reading this:
>
> 1. In my experience it's kind of difficult to write a good spider/crawler. 
> Today it's a lot more complex than using regexp to discover all <a href tags or stuff that looks like an url due to 
> javascript, flash, silverlight, etc ...
> That said, I think a decent spider/crawler could still be written for NSE.
> What I also think could be a good idea is to allow the user to "import" a file containing the URLs to process.
> This way you could manually cover most parts of a site using a local proxy, extract the urls and feed them to NSE.
This is another good idea.

> 2. I think the http-brute dependency to http-enum is kind of neat and would definitively be a nice to have.
> It would of course still be very useful to be able to point it at a specific path without running http-enum.
> For that use case, I think we should consider how we should handle those app/product specific brute scripts.
> In my opinion, in most cases, it's a matter of running the correct usernames and passwords and pointing it at the 
> correct URL.
> In order to handle this now, the whole script (http-brute or http-form-brute) must essentially be duplicated into 
> another script making these adjustments.
> I don't think this is an ideal way of handling it. Maybe both brute scripts should be changed to libraries instead 
> allowing this kind of product/app specific scripts to contain only the path and dictionaries?
Well, in this situation we just make http-brute register matches dynamically
and depend on http-enum, and we do not change the path argument. If
http-enum is running then the http-brute can have more paths, otherwise
it will just run as usual, check arguments, etc.

> 3. When working on the http-brute script I remember finding a bug in the http library preventing more than one Nmap 
> thread running.
> I'm not even sure whether this bug is still there or not but it's definitively something that needs looking in to.
Libraries code should be reentrant and thread-safe.
Perhaps this bug was corrected, I remember that David or Paulino said
on an IRC discussion that it was corrected, but I'm not sure.

> That's what I have comment-wise so far.
Thanks for the comments.

> //Patrik
> --
> Patrik Karlsson
> http://www.cqure.net
> http://www.twitter.com/nevdull77
>
> _______________________________________________
> Sent through the nmap-dev mailing list
> http://cgi.insecure.org/mailman/listinfo/nmap-dev
> Archived at http://seclists.org/nmap-dev/

-- 
tixxdz
http://opendz.org
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/