Re: An unusual denial of service attack

[David Farber <dave () farber net>]

Begin forwarded message:

From: Benjamin Black <b () b3k us>
Date: May 5, 2009 5:56:53 AM EDT
To: dave () farber net
Subject: Re: [IP] An unusual denial of service attack

Dave,

This has been a concern to many folks, inside and outside Microsoft,  
for several years now.  Solving it is non-trivial as slowing down or  
otherwise pacing the delivery of updates can mean critical security  
vulnerabilities remain unpatched for days longer.  One attractive  
solution is the use of peer protocols.  Though I failed to make much  
headway in that direction, I remain optimistic something more  
efficient than the current thundering herd will come along.  That will  
present a new set of trade-offs as many folks, Brett included, dislike  
the impact of peer protocols on their networks, as well.

As a former member of the team responsible for building Microsoft's  
CDN, I suggest Brett take a closer look at the traffic.  He might  
notice that the traffic is being served from up to 4 different CDNs.   
He might also notice that the vast majority of Akamai traffic comes  
from a very small number of data centers, as is the case for any  
modern, efficient CDN.  Lots of tiny caches everywhere makes for good  
marketing but bad business.  The typical approach is for ISPs to peer  
with the CDNs directly.

Brett mentions having his own caches, but being unable to make use of  
them because of cache-control headers from the CDNs.  Has he  
considered simply ignoring those headers?  I can't recommend mucking  
about with Windows Update, but it might provide some relief.


Ben

On Mon, May 4, 2009 at 8:56 AM, David Farber <dave () farber net> wrote:


Begin forwarded message:

From: Brett Glass <brett () lariat net>
Date: May 4, 2009 11:07:50 AM EDT
To: dave () farber net, "Ip ip" <ip () v2 listbox com>
Subject: An unusual denial of service attack

Dave, and everyone:

This weekend, my ISP suffered an unusual sort of denial of service  
attack.

Starting on Saturday morning, users were reporting that their Web  
browsing had slowed to a crawl, though other services were working  
properly. I investigated, and saw that our upstream connection to the  
Internet backbone was being saturated -- but not by any one customer.  
So, I looked at the statistics on our Web cache (an activity, by the  
way, which I'm sure that certain privacy advocates would find  
tantamount to "snooping," even though it was for the purpose of  
managing the network). After awhile, I was able to figure out what was  
wrong.

We were facing a distributed denial of service attack from the world's  
largest "botnet:" Microsoft's "Windows Update."

Virtually every Windows machine on our network -- and most of our  
customers's machines are running Windows XP or Windows Vista -- was  
individually downloading many large updates. (See

http://www.computerworld.com/action/article.do?command=printArticleBasic&taxonomyName=Security&articleId=9131573&taxonomyId=17

for a list of some of the many security holes that were being patched.)

Fixing holes in Windows is a good thing, but to command more than 90%  
of all of the computers around the globe to "phone home" at the same  
time is, obviously, not. It's doubly bad when the updates are  
explicitly marked as non-cacheable, making our Web cache of no use to  
stem the flood.

What's worse -- at least for our small ISP -- is that the updates are  
distributed for Microsoft by a company called Akamai. Akamai, as many  
of you know, places caches at the hubs of many ISPs' networks -- but,  
alas, only those of larger ones. Our smaller ISP, which has never been  
able to convince Akamai to place a cache at our location despite many  
years of requests, therefore must use backbone bandwidth to service  
all of these redundant requests. When I checked -- and it was not at  
the peak -- the traffic was consuming about half of our main DS-3 line  
to the Internet, leaving only half of its capacity available to carry  
all other traffic (including VoIP and bandwidth-intensive streaming  
video). Our cache's CPU utilization was above 95%, slowing response  
times still further.

I solved the problem by telling the cache to throttle traffic to and  
from Akamai's upstream caches, which were serving up the updates.  
Instantly, the load dropped off and normal service was restored.

As Spider-Man creator Stan Lee once noted, "with great power comes  
great responsibility." Microsoft, by virtue of its control over  
Windows-based PCs, has the ability to shut down the entire Internet at  
will -- and must be careful not to do it, inadvertently, by turning  
90% of the world's PCs into a "zombie army."

Furthermore, content delivery networks such as Akamai, which  
distributes Microsoft's updates, must not be allowed to discriminate  
against smaller providers by making updates uncacheable (at least by a  
standards-conforming Web cache) and then denying smaller ISPs access  
to a cache that WILL cache them. (Google, too, is also placing caches  
at the hubs of larger ISPs, thus giving them an edge when it comes to  
delivering Google services and video.) Small and competitive ISPs  
already have a tough row to hoe when competing with the telcos and  
cable companies. If they are further disadvantaged by prejudicial  
business practices of content providers and content delivery networks,  
Internet service will -- devastatingly for consumers -- become a  
duopoly.

--Brett Glass





-------------------------------------------
Archives: https://www.listbox.com/member/archive/247/=now
RSS Feed: https://www.listbox.com/member/archive/rss/247/
Powered by Listbox: http://www.listbox.com





-------------------------------------------
Archives: https://www.listbox.com/member/archive/247/=now
RSS Feed: https://www.listbox.com/member/archive/rss/247/
Powered by Listbox: http://www.listbox.com