RE: Web bots - blocking the bad

["Chad" <ncm () xs4all nl>]

Howdy Bob/List,

> Hello...In reviewing our IIS logs we are seeing many occurrences of web
> robots scanning our site.  Like all things, I am seeing there are good
> robots and bad robots. I'd like to know your thoughts on spending time
> implementing measures to block the bad robots from scanning and
> indexing the
> web-site. Your opinions with a short explanation are appreciated.

I've been faced with the same problem before and my solution was simply to
build an asp script that would redirect the robot to a "friendly 404" page
using a list of bad bots and the request's User Agent info. This solution is
working well for me, I update the bad bots list as I catch them in my logs.

Personally I'm interested in taking this one step higher, to the point where
IIS itself would do the intercepting via a custom written ISAPI
dll/component, using the data from a central "bad robot/crawler  database"
to scan the visitor's user agent info.

I'll write the dll/component myself, but am wondering if anyone else would
be interested in making use of such service where a central database of "bad
bots" is maintained.

I have really not had the opportunity to google the web and see if such a
service/component is already available, and if it's free.

Chad



---------------------------------------------------------------------------
----------------------------------------------------------------------------