Re: Web crawling library proposal

[Paulino Calderon <paulino () calderonpale com>]

Hi list,

I'm attaching my working copies of the web crawling library and  a few 
scripts that use it. It would be great if I can get some feedback.

All the documentation is here:
https://secwiki.org/w/Nmap/Spidering_Library

I'm including 3 scripts using the library:
* http-sitemap - Returns a list of URIs found. (Useful for target enum)
* http-phpselfxss-scan - Returns a list of PHP files vulnerable to Cross 
Site Scripting via infecting the variable $_SERVER["PHP_SELF"].
* http-email-harvest - Returns a list of the email accounts found in the 
web server.

NSE scripts would start a crawling process and then get a list of URIs 
to be processed as the programmer wishes. For example if we wanted to 
write a script to look for backup files we could simply do:

  httpspider.crawl(host, port)
  local uris = httpspider.get_sitemap()
  for _, uri in pairs(uris) do
    local obj = http.get(uri .. ".bak")
    if page_exists(obj and other params...) then
        results[#results+1] = uri
  end

There is still work to be done since spidering can be as complex as we 
want but I wanted to get an idea of what are the most important things 
to add to my TODO list for the following days.

I've also setup a vulnerable application that you are free to scan:

http://calder0n.com/sillyapp/
nmap -p80 --script http-sitemap,http-email-harvest,http-phpselfxss-scan 
--script-args httpspider.path=/sillyapp/ calder0n.com

nmap -p80 --script http-phpselfxss-scan,http-email-harvest,http-sitemap 
--script-args httpspider.path=/sillyapp/ calder0n.com

Starting Nmap 5.59BETA1 ( http://nmap.org ) at 2011-10-19 00:13 PDT
Nmap scan report for calder0n.com (173.45.233.210)
Host is up (0.14s latency).
PORT   STATE SERVICE
80/tcp open  http
| http-email-harvest: info () domain com
|_nmap-dev () insecure org
| http-sitemap: URIs found:
| http://calder0n.com/sillyapp/secret/2.php
| http://calder0n.com/sillyapp/index.php
| http://calder0n.com/sillyapp/
| http://calder0n.com/sillyapp/secret/1.php?hola=1
| http://calder0n.com/sillyapp/one.php
| http://calder0n.com/sillyapp/1.php
| http://calder0n.com/sillyapp/two.php
|_http://calder0n.com/sillyapp/three.php
| http-phpselfxss-scan: Vulnerable files:
| 
http://calder0n.com/sillyapp/secret/2.php/%27%22/%3E%3Cscript%3Ealert(1)%3C/script%3E
| 
http://calder0n.com/sillyapp/1.php/%27%22/%3E%3Cscript%3Ealert(1)%3C/script%3E
|_http://calder0n.com/sillyapp/three.php/%27%22/%3E%3Cscript%3Ealert(1)%3C/script%3E


Cheers!

--
Paulino Calderón Pale
Web: http://calderonpale.com
Twitter: http://www.twitter.com/paulinocaIderon

Attachment: httpspider.lua
Description:

Attachment: http-email-harvest.nse
Description:

Attachment: http-phpselfxss-scan.nse
Description:

Attachment: http-sitemap.nse
Description:

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/