Full Disclosure mailing list archives
Re: Google's robots.txt handling
From: Swair Mehta <swairmehta () gmail com>
Date: Mon, 10 Dec 2012 19:12:53 -0800
Coldwind is right, u r talking about security through obscurity. If u tell a pentester that u r using joomla and php together, he/she will try <yourwebsite>.com/administrator Since if u r ignorant and havent blocked access to it, your joomla access page will show up and hydra/brutus will be able to take over from there. On 10-Dec-2012, at 3:35 PM, Gynvael Coldwind <gynvael () coldwind pl> wrote:
Hey,Here is an example: An admin has a public webservice running with folders containing sensitive informations. Enter these folders in his robots.txt and "protect" them from the indexing process of spiders. As he doesn't want the /admin/ gui to appear in the search results he also puts his /admin in the robots text and finaly makes a backup to the folder /backup.If no one would know about a folder, why would one add it to robots.txt in the first place? But that's missing the point anyway - robots.txt is not a security mechanism. If someone uses robots.txt as the only and last line of defense he plainly doesn't understand what he's doing (especially that it's one of the first files both pentesters & attackers look at). If someone has an /admin/ site (which is a really easily guessable name, checked by every web directory scanner out there) he cannot rely on concealment*, but on proper user authentication using mechanisms designed for such purpose (e.g. requiring a password). (* for historical reasons there is a Polish IT term for such attempts - "deep hiding", there's even a wiki page on that - http://pl.wikipedia.org/wiki/G%C5%82%C4%99bokie_ukrycie)I'm wondering if, in perhaps .htaccess, one could allow ONLY site crawlers access to the robots.txt file. Then add robots.txt to robots.txt...would this mitigate some of the risk?1. It's still missing the point. 2. No, it wouldn't work in case of scanners that try to impersonate robots. -- gynvael.coldwind//vx _______________________________________________ Full-Disclosure - We believe in it. Charter: http://lists.grok.org.uk/full-disclosure-charter.html Hosted and sponsored by Secunia - http://secunia.com/
_______________________________________________ Full-Disclosure - We believe in it. Charter: http://lists.grok.org.uk/full-disclosure-charter.html Hosted and sponsored by Secunia - http://secunia.com/
Current thread:
- Google's robots.txt handling Hurgel Bumpf (Dec 10)
- Re: Google's robots.txt handling James Lay (Dec 10)
- Re: Google's robots.txt handling Gynvael Coldwind (Dec 10)
- Re: Google's robots.txt handling Benji (Dec 11)
- Re: Google's robots.txt handling Swair Mehta (Dec 11)
- Re: Google's robots.txt handling Stefan Edwards (Dec 11)
- Re: Google's robots.txt handling Gildseth, Tommy (Dec 11)
- Re: Google's robots.txt handling Gynvael Coldwind (Dec 10)
- Re: Google's robots.txt handling Philip Whitehouse (Dec 11)
- Re: Google's robots.txt handling Denis McMahon (Dec 11)
- Re: Google's robots.txt handling Lehman, Jim (Dec 12)
- Re: Google's robots.txt handling Christoph Gruber (Dec 12)
- Re: Google's robots.txt handling Patrick Webster (Dec 12)
- Re: Google's robots.txt handling Mario Vilas (Dec 13)
- Re: Google's robots.txt handling Philip Whitehouse (Dec 13)
- Re: Google's robots.txt handling Jeffrey Walton (Dec 13)
- Re: Google's robots.txt handling Christoph Gruber (Dec 12)
- Re: Google's robots.txt handling James Lay (Dec 10)