Nmap Development mailing list archives
Re: [RFC] Improve NSE HTTP architecture.
From: Djalal Harouni <tixxdz () opendz org>
Date: Sat, 18 Jun 2011 09:47:31 +0100
On Thu, Jun 16, 2011 at 06:02:02PM -0500, Ron wrote:
On Thu, 16 Jun 2011 08:06:30 +0200 Patrik Karlsson <patrik () cqure net> wrote:1. In my experience it's kind of difficult to write a good spider/crawler. Today it's a lot more complex than using regexp to discover all <a href tags or stuff that looks like an url due to javascript, flash, silverlight, etc ... That said, I think a decent spider/crawler could still be written for NSE. What I also think could be a good idea is to allow the user to "import" a file containing the URLs to process. This way you could manually cover most parts of a site using a local proxy, extract the urls and feed them to NSE.Something else to keep in mind - http-fingerprints.nse can seed http-spider.nse. So in addition to the URLs to process, we can use a whole pile of known URLs. Djalal - Great work on this writeup! I replied to a thread about this from Patrik yesterday, but it seems like you were wayyy ahead of me. In nmap-exp/ron, I have a mostly working (but not really tested) http-spider.nse script. If you're planning on working on a spider, that might be a decent starting point (or not.. :) )
I'll check that thread. I didn't go deep in the study of the spider, since it will be Paulino's job, and he already has a proposal draft in his nmap-exp/ branch. And yes we must check your http-spider script, it really can be a good starting point. Our proposal is based on your http-enum and http-fingerprints work, and also Patrik's work. Just to let you know that currently I'm not working on implementing the proposal ideas, feel free to experiment them. Thanks Ron. -- tixxdz http://opendz.org _______________________________________________ Sent through the nmap-dev mailing list http://cgi.insecure.org/mailman/listinfo/nmap-dev Archived at http://seclists.org/nmap-dev/
Current thread:
- [RFC] Improve NSE HTTP architecture. Djalal Harouni (Jun 14)
- Re: [RFC] Improve NSE HTTP architecture. Patrik Karlsson (Jun 15)
- Re: [RFC] Improve NSE HTTP architecture. Ron (Jun 16)
- Re: [RFC] Improve NSE HTTP architecture. Djalal Harouni (Jun 18)
- Re: [RFC] Improve NSE HTTP architecture. Djalal Harouni (Jun 18)
- Re: [RFC] Improve NSE HTTP architecture. Ron (Jun 16)
- Re: [RFC] Improve NSE HTTP architecture. Fyodor (Jun 16)
- Re: [RFC] Improve NSE HTTP architecture. Djalal Harouni (Jun 19)
- Re: [RFC] Improve NSE HTTP architecture. Patrick Donnelly (Jun 20)
- Re: [RFC] Improve NSE HTTP architecture. Djalal Harouni (Jun 20)
- Re: [RFC] Improve NSE HTTP architecture. Djalal Harouni (Jun 19)
- Re: [RFC] Improve NSE HTTP architecture. Patrik Karlsson (Jun 15)