diagnosing possible DOS

[Alexander Kurt Keller <alkeller () SFSU EDU>]

Hello Folks,

Requesting suggestions to diagnose an Apache issue (Ubuntu server 8.04 LTS with Apache 2.2.8 serving custom PHP 
application with MySQL backend).  Server runs normally for a few hours and then Apache locks up, logs entries simply 
halt. Strace on the Apache processes all look like this:

Process 31281 attached - interrupt to quit
     0.000000 restart_syscall(<... resuming interrupted call ...>) = 0
     0.538531 poll([{fd=15, events=POLLIN|POLLPRI}], 1, 0) = 0
     0.000100 gettimeofday({1324400675, 745453}, NULL) = 0
     0.000046 gettimeofday({1324400675, 745492}, NULL) = 0
     0.000032 gettimeofday({1324400675, 745524}, NULL) = 0

We identified that a website blog function had allowed for significant commercial blog spam to be posted on the site 
(nonsensical text with lots of links to commercial sites: “Ugg boots clearance”, “Denim jackets cheap”, etc.), those 
posts have been deleted and the blog mechanism has been secured. Reviewing the Apache logs and Wireshark captures, we 
see that we have a LOT of traffic trying to get to those unauthorized (and now unresolvable) blog entries. Many of the 
requesting IPs are reverse proxies and search engine bots who seem to be crawling those spam URLs very aggressively.

We have concluded that our site was leveraged for a search engine “optimization” campaign, but now it appears we are 
suffering from a denial of service condition that may not have been intentional (If we were selling Ugg boots, we would 
be rich by now). We have some leads on mitigation: blocking aggressive hosts, mod_security, etc., but on a more 
fundamental level we are hoping to use this opportunity to educate ourselves on what to look for (and how to look for 
it) when experiencing these sort of events.

Any hints on Wireshark log parsing options for diagnosing DOS? Any thoughts on this behavior and the underpinnings of 
unscrupulous SEO campaigns?

I’ll take this opportunity to thank everyone for their contributions to the list in 2011 and offer a toast to an 
equally productive 2012!

Cheers,
alex


Alex Keller
Systems Administrator
Academic Technology, San Francisco State University
☛Burk Hall 155 ☎ (415)338-6117 ✉alkeller () sfsu edu<mailto:alkeller () sfsu edu>