Re: [NSE] http Library Enhancements/Strategy

[David Fifield <david () bamsoftware com>]

On Mon, May 03, 2010 at 02:04:37PM +0200, Marc Ruef wrote:
> Hello,
>
> I am currently developing a nmap nse port of my httprecon project  
> (requested in [1], first announced at [2], actual screenshot at [3]). I  
> will publish an article series about nse hacking at [4]. After finishing  
> the series I will release httprecon-nse. Feature requests and feedback  
> is always welcome!
>
> During the development I did rely on the http library very heavily. It  
> was a surprise to see that the whole 3xx redirects are not handled  
> within the library. All the http scripts need to implement the feature  
> themselves:
>
> * At the moment html-title.nse is providing a neat implementation.  
> Re-directs to other hosts/ports are detected properly (which is a  
> requirement in professional testing due to limitation of confirmed 
> targets).
> * On the other hand http-malware-host.nse uses a quick hack which may  
> lead to false-positives: Only the status code is compared. Bbecause no  
> further requests are initiated, this is no "danger" for the target site.

In the case of http-malware-host the quick check is enough, because the
mere presence of the redirect indicates an infection.

> Wouldn't it make sense to provide a redirect identification within the  
> http library? I would prefer a function that returns the new url as  
> string. Additional optional arguments could prevent access outside the  
> target host/port.

I think it's a good idea. Are you interested in writing this function
for the http library?

I had to write something like this for the http-favicon survey script. I
didn't implement external host checking. I attached the script in case
you want to use it for inspiration. The relevant functions are dirname,
parse_url_relative, and http_get_redirected. It is used like this:

        root_host, root_port, root_path, body =
                http_get_redirected(host, port, "/", name, ip, REDIRECT_LIMIT)

> Furthermore, I would appreciate a generic function for generating and  
> sending http requests. The definition of the target host, target port,  
> method, resource, protocol and additional headers would be great.

We have this now, with http.generic_request.
        http://nmap.org/nsedoc/lib/http#generic_request
The build_request that generates the request string is local to the http
library, but it could be exposed externally if needed.

David Fifield
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/