Re: robots.txt access rules

["James Fields" <jvfields () tds net>]

Checked out my IDS alerts on this today - it's actually kind of nice to have
turned on.  I don't get hit with anything like an overwhelming number, and
when I did reverse DNS lookups on the source IPs I found a cable modem user
poking around amongst the web crawlers...

----- Original Message -----
From: "Seymour, Keith E." <KESeymour () magellanhealth com>
To: <focus-ids () securityfocus com>
Cc: "Federico Petronio" <petrus () activesec biz>
Sent: Thursday, January 22, 2004 9:12 AM
Subject: RE: robots.txt access rules


Federico,

Depending on how attentive you are to the traffic, the rule will alert you
to anyone requesting the file. Obviously, robots should request the file and
avoid the listed directories. An attacker or curious visitor will go
directly to the listed directories. If your interested, check your web log
after you see this rule fire and you will have some insight into that
visitor.

This is one of the 'it would be nice to know' rules than 'the sky is
falling' rule.

Keith

-----Original Message-----
From: Federico Petronio [mailto:petrus () activesec biz]
Sent: Wednesday, January 21, 2004 9:15 AM
To: focus-ids () securityfocus com
Subject: robots.txt access rules


Hi all...

I have installed snort-inline and I am customizing rulesets.

My cuestion is about the rule sid:1852 which match accesses to
/robots.txt files. The goal of this rule is to not let access to
information about sensitive areas of the webserver (which can be use to
achive knowledge about restricted areas, etc), but if they are not
present Google, etc. would intent to index those areas... So... what
shoud I do? Is it better to have that rule active or inactive? The
restriccted areas should be RESTRICTED and not just "hidden" so... the
rule make no sence?

I would like to hear you opions about this... Thanks a lot.
--
                                         Federico Petronio
                                         petrus () activesec biz


---------------------------------------------------------------------------
---------------------------------------------------------------------------


---------------------------------------------------------------------------
---------------------------------------------------------------------------


---------------------------------------------------------------------------
---------------------------------------------------------------------------