RE: Web bots - blocking the bad

["Chad" <ncm () xs4all nl>]

> From: Ben Carr
> Sent: 19 August 2003 19:34

> The place to start is your robots.txt file. This file does/should

> From what I see in my logs, compared to my robots.txt file

User-agent: *
Disallow: /

I still get robots that crawl the site multiple times a day. Seems that not
all crawlers/robots follow this rule. Plus, I think the "bad
crawlers/robots" out there will not be looking at adhering to the robots.txt
standard anyway.

Chad



---------------------------------------------------------------------------
----------------------------------------------------------------------------