[NSE] robots

["Eddie Bell" <ejlbell () gmail com>]

Hey all, committed a new script that downloads a http server's
robots.txt file and displays all disallowed entries

eddie@tesla ~/nmap> ./nmap -sT -p80 --script=./scripts/robots.nse nytimes.com

Interesting ports on 199.239.136.245:
PORT   STATE SERVICE
80/tcp open  http
|   robots.txt: /pages/college/ /library/ /aponline/ /cnet/
|_ /archives/ /thestreet/ /financialtimes/

cheers
 - eddie

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org