Re: how to alert if web http crawls are taking place

[David Glosser <david_glosser () yahoo com>]

How about if you place a hidden link on a web page (say a 1 pixel gif image)
that you deny in robots.txt.  Then you can 1) create a rule based on
matching the hidden web page or 2) have that web page launch an email
alerting you that the page was accessed.   You'll get a lot of "false
positives" with search engine spiders not following robots.txt. But you'll
get plenty of hits from users running page-suckers or spam-harvesters. You
can really get fancy/dangerous and have the launched script deny access to
that user by writing the ip address to an .htaccess file (if you are running
apache)

----- Original Message ----- 
From: "Bob Konigsberg" <bobkberg () networkeval com>
To: "'Jason Truong'" <Jason.Truong () plumtree com>; "'Snort-Users'"
<snort-users () lists sourceforge net>
Sent: Wednesday, December 08, 2004 5:58 PM
Subject: RE: [Snort-users] how to alert if web http crawls are taking place


> You might write a rule looking for an http get of robots.txt, since that
> should be the "official" starting point for any spider activity, but even
> that is no guarantee, since someone could spider your site without that.
>
> Bob
>
>
>
> -----Original Message-----
> From: snort-users-admin () lists sourceforge net
> [mailto:snort-users-admin () lists sourceforge net] On Behalf Of Jason Truong
> Sent: Wednesday, December 08, 2004 2:50 PM
> To: Snort-Users
> Subject: [Snort-users] how to alert if web http crawls are taking place
>
>
> Just wondering if anyone in the Snort community have a rule in place that
> alerts them when a web (http) crawl takes place.
>
> I'm interested in being alerted when a engineer decides to write his own
> application (or downloads one) that basically points to a website and
starts
> to crawl/download all the content by following all the links within the
> code.
>
> I think this has happened before in my environment but since its http, it
> usually looks like normal traffic to snort.
>
> Thanks,
>
> Jason T.
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide Read honest & candid
reviews
> on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Snort-users mailing list
> Snort-users () lists sourceforge net
> Go to this URL to change user options or unsubscribe:
> https://lists.sourceforge.net/lists/listinfo/snort-users
> Snort-users list archive:
> http://www.geocrawler.com/redir-sf.php3?list=ort-users
>
>
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Snort-users mailing list
> Snort-users () lists sourceforge net
> Go to this URL to change user options or unsubscribe:
> https://lists.sourceforge.net/lists/listinfo/snort-users
> Snort-users list archive:
> http://www.geocrawler.com/redir-sf.php3?list=snort-users



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users