Full Disclosure mailing list archives
Re: Google's robots.txt handling
From: Denis McMahon <denismfmcmahon () gmail com>
Date: Mon, 10 Dec 2012 20:36:15 +0000
On 10/12/12 19:25, Hurgel Bumpf wrote:
I tried to contact google, but as they didn't answer my email, I do forward this to FD.
This shouldn't be a discussion about bad practice but the google feature itself.
I seem to recall that the robots.txt exclusion standard was fairly common before I ever heard of google, I'm thinking of the mid to late 1990s when alta vista (which at the time was I think a technology demonstrator for VAX Alpha RISC hardware?) was the goto of search engines. Rgds Denis McMahon _______________________________________________ Full-Disclosure - We believe in it. Charter: http://lists.grok.org.uk/full-disclosure-charter.html Hosted and sponsored by Secunia - http://secunia.com/
Current thread:
- Google's robots.txt handling Hurgel Bumpf (Dec 10)
- Re: Google's robots.txt handling James Lay (Dec 10)
- Re: Google's robots.txt handling Gynvael Coldwind (Dec 10)
- Re: Google's robots.txt handling Benji (Dec 11)
- Re: Google's robots.txt handling Swair Mehta (Dec 11)
- Re: Google's robots.txt handling Stefan Edwards (Dec 11)
- Re: Google's robots.txt handling Gildseth, Tommy (Dec 11)
- Re: Google's robots.txt handling Gynvael Coldwind (Dec 10)
- Re: Google's robots.txt handling Philip Whitehouse (Dec 11)
- Re: Google's robots.txt handling Denis McMahon (Dec 11)
- Re: Google's robots.txt handling Lehman, Jim (Dec 12)
- Re: Google's robots.txt handling Christoph Gruber (Dec 12)
- Re: Google's robots.txt handling Patrick Webster (Dec 12)
- Re: Google's robots.txt handling Mario Vilas (Dec 13)
- Re: Google's robots.txt handling Philip Whitehouse (Dec 13)
- Re: Google's robots.txt handling Jeffrey Walton (Dec 13)
- Re: Google's robots.txt handling Julius Kivimäki (Dec 14)
- Re: Google's robots.txt handling Christoph Gruber (Dec 12)
- Re: Google's robots.txt handling James Lay (Dec 10)
- Re: Google's robots.txt handling Lehman, Jim (Dec 13)
- <Possible follow-ups>
- Re: Google's robots.txt handling Scott Ferguson (Dec 11)
- Re: Google's robots.txt handling Ulisses Montenegro (Dec 11)