Full Disclosure mailing list archives
Re: Google's robots.txt handling
From: "Lehman, Jim" <jim.lehman () interactivedata com>
Date: Tue, 11 Dec 2012 23:23:12 +0000
It is possible to use white listing for robots.txt. Allow what you want google to index and deny everything else. That way google doesn't make you a goole dork target and someone browsing to your robots.txt file doesn't glean any sensitive files or folders. But this will not stop directory bruting to discover your publicly exposed sensitive data, that probably should not be exposed to the web in the first place. I would rather have some one pound on my server to find something, I might have more time to respond, rather than having mr. bad googleing for the weakness in the web site and only making one request to get what they are after. http://www.sans.org/reading_room/whitepapers/awareness/robotstxt_33955 Its not a great paper, but it might have some value for those that have not looked into how this file works. -----Original Message----- From: full-disclosure-bounces () lists grok org uk [mailto:full-disclosure-bounces () lists grok org uk] On Behalf Of Hurgel Bumpf Sent: Monday, December 10, 2012 11:26 AM To: full-disclosure () lists grok org uk Subject: [Full-disclosure] Google's robots.txt handling Hi list, i tried to contact google, but as they didn't answer my email, i do forward this to FD. This "security" feature is not cleary a google vulnerability, but exposes websites informations that are not really intended to be public. (Additionally i have to say that i advocate robots.txt files without sensitive content and working security mechanisms.) Here is an example: An admin has a public webservice running with folders containing sensitive informations. Enter these folders in his robots.txt and "protect" them from the indexing process of spiders. As he doesn't want the /admin/ gui to appear in the search results he also puts his /admin in the robots text and finaly makes a backup to the folder /backup. Nevertheless these folders arent browsable but they might contain f(a)iles with easy to guess namestructures, non-encrypted authentications (simple AUTH) , you name it... Without a robots.txt nobody would know about the existance of these folders, but as some folders might be linked somewhere, these folders might appear in search results when not defined in the robots.txt The admin finds himself in a catch-22 situation where he seems to prefer the robots.txt file. Long story short. Although google accepts and respects the directives of the robots.txt file, google INDEXES these files. This my concern. http://www.google.com/search?q=inurl:robots.txt+filetype%3Atxt+Disallow%3A+%2Fadmin http://www.google.com/search?q=inurl:robots.txt+filetype%3Atxt+Disallow%3A+%2Fbackup http://www.google.com/search?q=inurl:robots.txt+filetype%3Atxt+Disallow%3A+%2Fpassword As these searches can be used less for targeted attacks, they more can be used to find victims. http://www.google.com/search?q=inurl:robots.txt+filetype%3Atxt+%2FDisallow%3A+wp-admin http://www.google.com/search?q=inurl:robots.txt+filetype%3Atxt+%2FDisallow%3A+typo3 <Just be creative> This shouldn't be a discussion about bad practice but the google feature itself. Indexing a file which is used to prevent indexing.. isn't that just paradox and hypocrite? Thanks, Conan the bavarian _______________________________________________ Full-Disclosure - We believe in it. Charter: http://lists.grok.org.uk/full-disclosure-charter.html Hosted and sponsored by Secunia - http://secunia.com/ ******************************************************* This message (including any files transmitted with it) may contain confidential and/or proprietary information, is the property of Interactive Data Corporation and/or its subsidiaries, and is directed only to the addressee(s). If you are not the designated recipient or have reason to believe you received this message in error, please delete this message from your system and notify the sender immediately. An unintended recipient's disclosure, copying, distribution, or use of this message or any attachments is prohibited and may be unlawful. ******************************************************* _______________________________________________ Full-Disclosure - We believe in it. Charter: http://lists.grok.org.uk/full-disclosure-charter.html Hosted and sponsored by Secunia - http://secunia.com/
Current thread:
- Google's robots.txt handling Hurgel Bumpf (Dec 10)
- Re: Google's robots.txt handling James Lay (Dec 10)
- Re: Google's robots.txt handling Gynvael Coldwind (Dec 10)
- Re: Google's robots.txt handling Benji (Dec 11)
- Re: Google's robots.txt handling Swair Mehta (Dec 11)
- Re: Google's robots.txt handling Stefan Edwards (Dec 11)
- Re: Google's robots.txt handling Gildseth, Tommy (Dec 11)
- Re: Google's robots.txt handling Gynvael Coldwind (Dec 10)
- Re: Google's robots.txt handling Philip Whitehouse (Dec 11)
- Re: Google's robots.txt handling Denis McMahon (Dec 11)
- Re: Google's robots.txt handling Lehman, Jim (Dec 12)
- Re: Google's robots.txt handling Christoph Gruber (Dec 12)
- Re: Google's robots.txt handling Patrick Webster (Dec 12)
- Re: Google's robots.txt handling Mario Vilas (Dec 13)
- Re: Google's robots.txt handling Philip Whitehouse (Dec 13)
- Re: Google's robots.txt handling Jeffrey Walton (Dec 13)
- Re: Google's robots.txt handling Julius Kivimäki (Dec 14)
- Re: Google's robots.txt handling Christoph Gruber (Dec 12)
- Re: Google's robots.txt handling James Lay (Dec 10)
- Re: Google's robots.txt handling Lehman, Jim (Dec 13)
- <Possible follow-ups>
- Re: Google's robots.txt handling Scott Ferguson (Dec 11)
- Re: Google's robots.txt handling Ulisses Montenegro (Dec 11)
- Re: Google's robots.txt handling Philip Whitehouse (Dec 11)
- Re: Google's robots.txt handling Ulisses Montenegro (Dec 11)