Nmap Development mailing list archives
Re: Web crawling library proposal
From: Paulino Calderon <paulino () calderonpale com>
Date: Mon, 07 Nov 2011 11:16:47 -0800
Hi list,Thank you all for the feedback. You've pointed out some very important issues. I've started working on the following changes:
* OO design. * crawl() must return a iterator. * Allow multiple crawlers to run at the same time.* Remove anchors from links by default instead of having to set the variable httpspider.ignoreParams.
* Queue will be removed. * Implement a better uri filter. Cheers! -- Paulino Calderón Pale Web: http://calderonpale.com Twitter: http://www.twitter.com/paulinocaIderon _______________________________________________ Sent through the nmap-dev mailing list http://cgi.insecure.org/mailman/listinfo/nmap-dev Archived at http://seclists.org/nmap-dev/
Current thread:
- Re: Web crawling library proposal Paulino Calderon (Oct 18)
- Re: Web crawling library proposal Patrick Donnelly (Oct 19)
- Re: Web crawling library proposal Paulino Calderon (Oct 19)
- Re: Web crawling library proposal Patrick Donnelly (Oct 19)
- Re: Web crawling library proposal Paulino Calderon (Oct 19)
- Re: Web crawling library proposal Paulino Calderon (Oct 19)
- Re: Web crawling library proposal Patrick Donnelly (Oct 19)
- Re: Web crawling library proposal Patrik Karlsson (Oct 19)
- Re: Web crawling library proposal Fyodor (Nov 01)
- Re: Web crawling library proposal David Fifield (Nov 05)
- Re: Web crawling library proposal Paulino Calderon (Nov 07)
- Re: Web crawling library proposal Patrik Karlsson (Nov 30)