RE: how to alert if web http crawls are taking place

["Bob Konigsberg" <bobkberg () networkeval com>]

You might write a rule looking for an http get of robots.txt, since that
should be the "official" starting point for any spider activity, but even
that is no guarantee, since someone could spider your site without that.

Bob

 

-----Original Message-----
From: snort-users-admin () lists sourceforge net
[mailto:snort-users-admin () lists sourceforge net] On Behalf Of Jason Truong
Sent: Wednesday, December 08, 2004 2:50 PM
To: Snort-Users
Subject: [Snort-users] how to alert if web http crawls are taking place


Just wondering if anyone in the Snort community have a rule in place that
alerts them when a web (http) crawl takes place.

I'm interested in being alerted when a engineer decides to write his own
application (or downloads one) that basically points to a website and starts
to crawl/download all the content by following all the links within the
code.  

I think this has happened before in my environment but since its http, it
usually looks like normal traffic to snort.

Thanks,

Jason T.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide Read honest & candid reviews
on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=ort-users





-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users