Re: Google's robot.txt handling

[Mario Vilas <mvilas () gmail com>]

I think we can all agree this is not a vulnerability. Still, I have yet to
see an argument saying why what the OP is proposing is a bad idea. It may
be a good idea to stop indexing robots.txt to mitigate the faults of lazy
or incompetent admins (Google already does this for many specific search
queries) and there's not much point in indexing the robots.txt file for
legitimate uses anyway.

On Tue, Dec 11, 2012 at 2:01 PM, Scott Ferguson <
scott.ferguson.it.consulting () gmail com> wrote:

> > If I understand the OP correctly, he is not stating that listing
> something
> > in robots.txt would make it inaccessible, but rather that Google indexes
> > the robots.txt files themselves,
>
> <snipped>
>
>
> Well, um, yeah - I got that.
>
> So you are what, proposing that moving an open door back a few
> centimetres solves the (non) problem?
>
> Take your proposal to it's logical extension and stop all search engines
> (especially the ones that don't respect robots.txt) from indexing
> robots.txt. Now what do you do about Nutch or even some perl script that
> anyone can whip up in 2 minutes?
>
> Security through obscurity is fine when couple with actual security -
> but relying on it alone is just daft.
>
> Expecting to world to change so bad habits have no consequence is
> dangerously naive.
>
> I suspect you're looking to hard at finding fault with Google - who are
> complying with the robots.txt. Read the spec. - it's about not following
> the listed directories, not about not listing the robots.txt.  Next
> you'll want laws against bad weather and furniture with sharp corners.
>
> Don't put things you don't want seen to see in places that can be seen.
>
> > On Mon, Dec 10, 2012 at 8:19 PM, Scott Ferguson <
> > scott.ferguson.it.consulting () gmail com> wrote:
> >
> >
> >     /From/: Hurgel Bumpf <l0rd_lunatic () yahoo com>
> >     /Date/: Mon, 10 Dec 2012 19:25:39 +0000 (GMT)
> ------------------------------------------------------------------------
> >     Hi list,
> >
> >
> >     i tried to contact google, but as they didn't answer my email,  i do
> >
> > forward this to FD.
> >
> >     This "security" feature is not cleary a google vulnerability, but
> >
> > exposes websites informations that are not really
> >
> >     intended to be public.
> >
> >     Conan the bavarian
> >
> > Your point eludes me - Google is indexing something which is publicly
> > available. eg.:- curl http://somesite.tld/robots.txt
> > So it seems the solution to the "question" your raise is, um,
> nonsensical.
> > If you don't want something exposed on your web server *don't publish
> > references to it*.
> >
> > The solution, which should be blindingly obvious,  is don't create the
> > problem in the first place. Password sensitive directories (htpasswd) -
> > then they don't have to be excluded from search engines (because listing
> > the inaccessible in robots.txt is redundant).  You must of missed the
> > first day of web school.
> >
> > Kind regards.
>
> _______________________________________________
> Full-Disclosure - We believe in it.
> Charter: http://lists.grok.org.uk/full-disclosure-charter.html
> Hosted and sponsored by Secunia - http://secunia.com/



-- 
“There's a reason we separate military and the police: one fights the enemy
of the state, the other serves and protects the people. When the military
becomes both, then the enemies of the state tend to become the people.”

_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/