interesting-people.org DDOS'ed

[Dave Farber <dave () farber net>]

Delivered-To: dfarber+ () ux13 sp cs cmu edu
Date: Tue, 21 Oct 2003 14:58:49 -0400 (EDT)
From: James M Galvin <galvin () eListX com>
Subject: interesting-people.org DDOS'ed
To: Dave Farber <dave () farber net>


As you requested, here is the summary of what happened to the
interesting-people.org domain for IP.


The topic for discussion, what to do when your domain is DDOS'ed?


At 12:17pm Eastern time on 20 October (yesterday), a spam attack started
for the interesting-people.org domain.  A dictionary attack began in
which one or more parties were sending messages to random and
semi-random user names in the domain interesting-people.org.

This attack is notable in part because there are no email addresses in
that domain.  In fact, the only purpose for the domain is for the
hostname "www.intereesting-people.org" where the archives for the IP
list are located.

This attack is ironic because the spam was selling an anti-spam product!

Strictly speaking, I don't host that domain, I only host the
www.interesting-people.org domain.  So my server will not relay messages
for that domain and thus the default action of the affected server was
simply to reject the messages.  So, for over 12 hours I simply
"weathered the storm."  There were 12,000-15,000 messages per hour
coming in for the 12 hour period.

I should point out that "weathering the storm" is problematic.  Although
no email was lost as far as I know, there were some obvious delays.
Perhaps more annoying is the fact that for some valid messages, once
they arrived on the server they would be delivered to their destinations
multiple times.  The affected server would do this because it was
generally operating at a reduced rate and at "peak" times various
timeouts would occur.  The result was that messages would be re-enqueued
for delivery because protocol acknowledgements were "lost" when in fact
the acknowledgements were just delayed.  Dave Farber suffered directly
here because I host his "farber.net" domain.  As if he doesn't get
enough email he was getting multiple copies of messages for a while.

In general, there's not much you can do about this kind of DDOS attack.
In this case, over the course of the attack there were over 61,000
distinct IP addresses dumping messages at the server.  I do operate a
real-time blacklist that I derive and manage myself.  The algorithm is a
bit more complicated than this but basically any IP address that
attempts to deliver to multiple "bad" email addresses is automatically
blacklisted (and later appropriately unblacklisted).  At its peak I had
just over 13,000 IP entries in the blacklist just because of the spam
coming to this domain.  The real point is even my ISP couldn't help me
at their ingress because there was no single source to block.

And then it got worse.

For a 3 hour period last night the incoming volume doubled to over
30,000 messages per hour.  I suspect it would have been higher but the
spammer or spammers had now hit a limit on the affected server:
specifically they had reached the maximum number of simultaneous
incoming connections.  They couldn't dump any more messages on me but
neither could any real messages get through.  Now I had to do something
more aggressive.

The solution was to more aggressively identify IP addresses for
blocking.  This kind of blocking is very efficient on my system because
it drops the connection at the earliest possible moment: as soon as the
source IP address is passed to the application it stops dead after one
lookup.  By gradually tweaking the criteria I was able to block more
addresses sooner and stay ahead of the incoming volume.  However, this
would not work long-term.  After about 6 hours I was fast approaching
the effective limit of this technique.

It was now time for a drastic move.  I could have done this as soon as
the issue surfaced but frankly we never figured the attack would last
this long.  We've weathered these attacks before and the longest they've
lasted is 2-3 hours.  We probably should have given up on dealing with
this much sooner but it was actually instructive to be able to work with
blacklist application a bit.

Fortunately, I was lucky.  I had a final solution.  In general, most
people won't.

The final solution in this case was to remove the address record from
the DNS for the domain interesting-people.org.  The record should never
have been there in the first place but it's never been an issue so there
was no reason to notice.  Lucky for us the time-to-live on the record
was an hour.  So, after spending a bit too long trying to contact the
service provider for the domain by going through channels, I was able to
contact Meng Weng directly who took care of removing the record
immediately.  Most of you will know him from ListBox as the host for the
IP mailing list.  Thanks to Dave Farber for Meng's direct phone number.

After removing the address record from the DNS the incoming volume
plummetted, and now 2 hours later it is gone.  However, removing an
address record or even changing your IP address is not a preferred
option for most people.  It is a technical solution but it's far from a
practical one.


All totaled, the attack lasted just shy of 24 hours.  Except for the
first 2 hours and the 3 hours last night, the sustained incoming stream
was 12,000-15,000 per hour.  There were over 61,000 distinct IP address
involved and eyeball math says only 250,000 distinct email addresses
attempted.  Multiple deliveries were attempted to some number of the
email addresses (from different IP addresses of course).

I have not tried to get stats on the number of networks involved but
even a cursory inspection makes it clear I could not have blocked at my
ISP's ingress.


So, what's in your server?


Jim Galvin
eList eXpress LLC

-------------------------------------
You are subscribed as interesting-people () lists elistx com
To manage your subscription, go to
 http://v2.listbox.com/member/?listname=ip

Archives at: http://www.interesting-people.org/archives/interesting-people/