Re: robots.txt NSE script

["Eddie Bell" <ejlbell () gmail com>]

Doug,

I haven't seen this before and it is interesting! The robot script is
a little weak atm :| but this gave me a couple of extra feature ideas.

I'm going to commit a new version after the next stable release

thanks
 - eddie

On 23/09/2007, doug () hcsw org <doug () hcsw org> wrote:
> Hi Eddie!
>
> Just thought I'd send over this article about a
> large-scale robots.txt survey in case you hadn't
> already seen it:
>
> http://www.nextthing.org/archives/2007/03/12/robotstxt-adventure
>
> Lots of interesting tidbits, as well as some ideas
> for fingerprinting web frameworks and add-on packages
> by commonly excluded path-names.
>
> Also there's apparently a very common file called
> info.txt that Alexa uses for site info:
>
> http://www.varisearch.com/Webmasters/PromotingYourSite/TipSheets/Alexa.htm
>
> Best,
>
> Doug
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.4 (GNU/Linux)
>
> iD8DBQFG9shC3LTjmOMguVMRAuiPAJ9cV+SrtoeDyHTlBHm+0bHTSL4+egCgkdvN
> 9qrlgP6+BaE63oXT4pfe5oo=
> =xW0K
> -----END PGP SIGNATURE-----

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org