web crawler DDOS

[David Pirolo <webmaster () WARNERPACIFIC EDU>]

We were hit by a DDOS yesterday.  After some investigation it was
discovered that it appears to have been caused by the web crawler
80legs.  Things started to quiet down about 5-10 minutes after I blocked
them in our robots.txt however, I'm not 100% positive if this was what
stopped them as others have reported seeing this agent ignore that file.

As you may know, the experience isn't very pleasant.  I've been doing a
bit of googleing and see a number of home-brewed solutions.  I've also
been looking in our AWStats and it appears that most of the spiders come
through as "unknown robot"; which isn't very helpful.

On top of reevaluating some of the firewall rules and tweaking our
system monitors, I was curious if any of the rest of you know of any
other abusive crawlers that we should be blocking access. Your best
practice advice is welcome as well.

Thanks,

David Pirolo
Warner Pacific College