Re: diagnosing possible DOS

[Alexander Kurt Keller <alkeller () SFSU EDU>]

Thank you Steven, Randall, Kevin, and Kay for your thoughtful responses to my query about our DOS condition. We are 
continuing to experiment but have started to bring the situation under control by clearing the errant content, using 
mod_security to restrict aggressive hosts, and implementing a PHP timeout on stalled requests. The Google webmaster 
tools proved useful for understanding what content was being searched for by visitors directed to our site as a result 
of the SEO tomfoolery. 

I'll post back to the list as merited, but wanted to say thank you for the assistance!

Best,
alex

Alex Keller
Systems Administrator
Academic Technology, San Francisco State University
☛Burk Hall 155 ☎ (415)338-6117 ✉alkeller () sfsu edu


-----Original Message-----
From: The EDUCAUSE Security Constituent Group Listserv [mailto:SECURITY () LISTSERV EDUCAUSE EDU] On Behalf Of Steven 
Alexander
Sent: Thursday, January 05, 2012 5:44 PM
To: SECURITY () LISTSERV EDUCAUSE EDU
Subject: Re: [SECURITY] diagnosing possible DOS

Wireshark and other sniffers are useful if you want to take a look at some individual connections to see what is 
happening, but not for detecting DoS or other malicious activity in the first place.  Your better bet would be to 
record session data using Argus or Cisco  Netflows.  You can also deploy an IDS, but I'd focus on getting session data 
first. The session data will give you a short record of every session with a timestamp, source/destination IP, 
source/destination port, in/out packet count and/or in/out bytes transferred.  If you identify a potential issue from 
session logs or web server logs (based on timing, volume, etc), you can grab a full packet sniff with 
Wireshark/tcpdump/tethereal to tell you exactly what is happening.

If you can log full content all the time, that will be very helpful in following up on any suspicious activity but 
you'll never want to use it as your first source of information.

You may also want to develop some scripts for scanning your web server logs.  If you develop scripts to extract all 
entries corresponding to a specific IP, error code, or time frame it will make it much easier to review logs in future 
investigations.

My $.02,

Steven

________________________________
From: The EDUCAUSE Security Constituent Group Listserv [SECURITY () LISTSERV EDUCAUSE EDU] on behalf of Alexander Kurt 
Keller [alkeller () SFSU EDU]
Sent: Thursday, January 05, 2012 11:56 AM
To: SECURITY () LISTSERV EDUCAUSE EDU
Subject: [SECURITY] diagnosing possible DOS

Hello Folks,

Requesting suggestions to diagnose an Apache issue (Ubuntu server 8.04 LTS with Apache 2.2.8 serving custom PHP 
application with MySQL backend).  Server runs normally for a few hours and then Apache locks up, logs entries simply 
halt. Strace on the Apache processes all look like this:

Process 31281 attached - interrupt to quit
     0.000000 restart_syscall(<... resuming interrupted call ...>) = 0
     0.538531 poll([{fd=15, events=POLLIN|POLLPRI}], 1, 0) = 0
     0.000100 gettimeofday({1324400675, 745453}, NULL) = 0
     0.000046 gettimeofday({1324400675, 745492}, NULL) = 0
     0.000032 gettimeofday({1324400675, 745524}, NULL) = 0

We identified that a website blog function had allowed for significant commercial blog spam to be posted on the site 
(nonsensical text with lots of links to commercial sites: “Ugg boots clearance”, “Denim jackets cheap”, etc.), those 
posts have been deleted and the blog mechanism has been secured. Reviewing the Apache logs and Wireshark captures, we 
see that we have a LOT of traffic trying to get to those unauthorized (and now unresolvable) blog entries. Many of the 
requesting IPs are reverse proxies and search engine bots who seem to be crawling those spam URLs very aggressively.

We have concluded that our site was leveraged for a search engine “optimization” campaign, but now it appears we are 
suffering from a denial of service condition that may not have been intentional (If we were selling Ugg boots, we would 
be rich by now). We have some leads on mitigation: blocking aggressive hosts, mod_security, etc., but on a more 
fundamental level we are hoping to use this opportunity to educate ourselves on what to look for (and how to look for 
it) when experiencing these sort of events.

Any hints on Wireshark log parsing options for diagnosing DOS? Any thoughts on this behavior and the underpinnings of 
unscrupulous SEO campaigns?

I’ll take this opportunity to thank everyone for their contributions to the list in 2011 and offer a toast to an 
equally productive 2012!

Cheers,
alex


Alex Keller
Systems Administrator
Academic Technology, San Francisco State University ☛Burk Hall 155 ☎ (415)338-6117 ✉alkeller () sfsu 
edu<mailto:alkeller () sfsu edu>


This email has been scanned by a Spam/Virus Firewall. If your email has been classified as Spam please contact the 
HelpDesk at (209) 384-6180.