Vulnerability Development mailing list archives

Re: Google lists vulnerable sites.


From: Skot <skot () canada com>
Date: 8 Jul 2002 15:55:00 -0000

In-Reply-To: <5.1.0.14.0.20020706205941.00ba75c8 () email telnor net>

From: Octavio / Super <alvarezp () telnor net>

Google (or any other robot/search engine) might as well ignore robots.txt. I think not to trust these.


Keep an eye on your logs: I've always noticed Google droping the hunt after the first hit it got was the 
robots.txt file.  There are other search robots that are not quite as polite though, they get blocked.

Skot.


Current thread: