Educause Security Discussion mailing list archives
web crawler DDOS
From: David Pirolo <webmaster () WARNERPACIFIC EDU>
Date: Tue, 27 Nov 2012 10:54:55 -0800
We were hit by a DDOS yesterday. After some investigation it was discovered that it appears to have been caused by the web crawler 80legs. Things started to quiet down about 5-10 minutes after I blocked them in our robots.txt however, I'm not 100% positive if this was what stopped them as others have reported seeing this agent ignore that file. As you may know, the experience isn't very pleasant. I've been doing a bit of googleing and see a number of home-brewed solutions. I've also been looking in our AWStats and it appears that most of the spiders come through as "unknown robot"; which isn't very helpful. On top of reevaluating some of the firewall rules and tweaking our system monitors, I was curious if any of the rest of you know of any other abusive crawlers that we should be blocking access. Your best practice advice is welcome as well. Thanks, David Pirolo Warner Pacific College
Current thread:
- web crawler DDOS David Pirolo (Nov 27)