RE: HTTP robot detection?

["Sheahan, Paul (PCLN-NW)" <Paul.Sheahan () priceline com>]

Actually I mean a robot that you download....like a utility you can use to
download an entire website to your local drive so you can browse it later
etc. These type of robot tools grab all URL's at a very fast pace from a web
site. I'm wondering if this can be detected somehow......thanks

-----Original Message-----
From: Hyoung-Kee Choi [mailto:hkchoi () lancope com]
Sent: Thursday, January 24, 2002 4:42 PM
To: Sheahan, Paul (PCLN-NW)
Subject: RE: [Snort-users] HTTP robot detection?


If one does not want a robot to copy its site one can specify that order in
"robot.txt". Hence, robots are supposed to look for "robot.txt".

> -----Original Message-----
> From: snort-users-admin () lists sourceforge net
> [mailto:snort-users-admin () lists sourceforge net]On Behalf Of
> Sheahan, Paul (PCLN-NW)
> Sent: Thursday, January 24, 2002 3:55 PM
> To: Snort List (E-mail)
> Subject: [Snort-users] HTTP robot detection?
>
>
>
> Anyone have any ideas on this one?
>
> I was wondering if there was a way to make Snort detect someone running an
> automated script or robot against a website the way it checks for
> portscans?
> For example, Snort flags traffic as a portscan when there are
> connections to
> a certain number of ports on one host within a certain time
> period. Is there
> way to do this with URLs? For example, so many URLs accessed at one IP
> address within a certain time period would be flagged as some sort of
> automated tool or robot scanning a site?
>
> Thanks!
>
>
> _______________________________________________
> Snort-users mailing list
> Snort-users () lists sourceforge net
> Go to this URL to change user options or unsubscribe:
> https://lists.sourceforge.net/lists/listinfo/snort-users
> Snort-users list archive:
> http://www.geocrawler.com/redir-sf.php3?list=snort-users

_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users