Full Disclosure mailing list archives
Re: Google's robots.txt handling
From: Scott Ferguson <scott.ferguson.it.consulting () gmail com>
Date: Tue, 11 Dec 2012 09:19:19 +1100
/From/: Hurgel Bumpf <l0rd_lunatic () yahoo com> /Date/: Mon, 10 Dec 2012 19:25:39 +0000 (GMT) ------------------------------------------------------------------------ Hi list, i tried to contact google, but as they didn't answer my email, i do forward this to FD. This "security" feature is not cleary a google vulnerability, but exposes websites informations that are not really intended to be public. (Additionally i have to say that i advocate robots.txt files without sensitive content and working security mechanisms.) Here is an example: An admin has a public webservice running with folders containing sensitive informations. Enter these folders in his robots.txt and "protect" them from the indexing process of spiders. As he doesn't want the /admin/ gui to appear in the search results he also puts his /admin in the robots text and finaly makes a backup to the folder /backup. <snipped> This shouldn't be a discussion about bad practice but the google feature itself. Indexing a file which is used to prevent indexing.. isn't that just paradox and hypocrite? Thanks, Conan the bavarian
Your point eludes me - Google is indexing something which is publicly available. eg.:- curl http://somesite.tld/robots.txt So it seems the solution to the "question" your raise is, um, nonsensical. If you don't want something exposed on your web server *don't publish references to it*. The solution, which should be blindingly obvious, is don't create the problem in the first place. Password sensitive directories (htpasswd) - then they don't have to be excluded from search engines (because listing the inaccessible in robots.txt is redundant). You must of missed the first day of web school. Kind regards.
Attachment:
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Full-Disclosure - We believe in it. Charter: http://lists.grok.org.uk/full-disclosure-charter.html Hosted and sponsored by Secunia - http://secunia.com/
Current thread:
- Re: Google's robots.txt handling, (continued)
- Re: Google's robots.txt handling Philip Whitehouse (Dec 11)
- Re: Google's robots.txt handling Denis McMahon (Dec 11)
- Re: Google's robots.txt handling Lehman, Jim (Dec 12)
- Re: Google's robots.txt handling Christoph Gruber (Dec 12)
- Re: Google's robots.txt handling Patrick Webster (Dec 12)
- Re: Google's robots.txt handling Mario Vilas (Dec 13)
- Re: Google's robots.txt handling Philip Whitehouse (Dec 13)
- Re: Google's robots.txt handling Jeffrey Walton (Dec 13)
- Re: Google's robots.txt handling Julius Kivimäki (Dec 14)
- Re: Google's robots.txt handling Christoph Gruber (Dec 12)
- Re: Google's robots.txt handling Lehman, Jim (Dec 13)
- Re: Google's robots.txt handling Ulisses Montenegro (Dec 11)
- Re: Google's robots.txt handling Philip Whitehouse (Dec 11)