An unusual denial of service attack

[David Farber <dave () farber net>]

Begin forwarded message:

From: Brett Glass <brett () lariat net>
Date: May 4, 2009 11:07:50 AM EDT
To: dave () farber net, "Ip ip" <ip () v2 listbox com>
Subject: An unusual denial of service attack

Dave, and everyone:

This weekend, my ISP suffered an unusual sort of denial of service  
attack.

Starting on Saturday morning, users were reporting that their Web  
browsing had slowed to a crawl, though other services were working  
properly. I investigated, and saw that our upstream connection to the  
Internet backbone was being saturated -- but not by any one customer.  
So, I looked at the statistics on our Web cache (an activity, by the  
way, which I'm sure that certain privacy advocates would find  
tantamount to "snooping," even though it was for the purpose of  
managing the network). After awhile, I was able to figure out what was  
wrong.

We were facing a distributed denial of service attack from the world's  
largest "botnet:" Microsoft's "Windows Update."

Virtually every Windows machine on our network -- and most of our  
customers's machines are running Windows XP or Windows Vista -- was  
individually downloading many large updates. (See

http://www.computerworld.com/action/article.do?command=printArticleBasic&taxonomyName=Security&articleId=9131573&taxonomyId=17

for a list of some of the many security holes that were being patched.)

Fixing holes in Windows is a good thing, but to command more than 90%  
of all of the computers around the globe to "phone home" at the same  
time is, obviously, not. It's doubly bad when the updates are  
explicitly marked as non-cacheable, making our Web cache of no use to  
stem the flood.

What's worse -- at least for our small ISP -- is that the updates are  
distributed for Microsoft by a company called Akamai. Akamai, as many  
of you know, places caches at the hubs of many ISPs' networks -- but,  
alas, only those of larger ones. Our smaller ISP, which has never been  
able to convince Akamai to place a cache at our location despite many  
years of requests, therefore must use backbone bandwidth to service  
all of these redundant requests. When I checked -- and it was not at  
the peak -- the traffic was consuming about half of our main DS-3 line  
to the Internet, leaving only half of its capacity available to carry  
all other traffic (including VoIP and bandwidth-intensive streaming  
video). Our cache's CPU utilization was above 95%, slowing response  
times still further.

I solved the problem by telling the cache to throttle traffic to and  
from Akamai's upstream caches, which were serving up the updates.  
Instantly, the load dropped off and normal service was restored.

As Spider-Man creator Stan Lee once noted, "with great power comes  
great responsibility." Microsoft, by virtue of its control over  
Windows-based PCs, has the ability to shut down the entire Internet at  
will -- and must be careful not to do it, inadvertently, by turning  
90% of the world's PCs into a "zombie army."

Furthermore, content delivery networks such as Akamai, which  
distributes Microsoft's updates, must not be allowed to discriminate  
against smaller providers by making updates uncacheable (at least by a  
standards-conforming Web cache) and then denying smaller ISPs access  
to a cache that WILL cache them. (Google, too, is also placing caches  
at the hubs of larger ISPs, thus giving them an edge when it comes to  
delivering Google services and video.) Small and competitive ISPs  
already have a tough row to hoe when competing with the telcos and  
cable companies. If they are further disadvantaged by prejudicial  
business practices of content providers and content delivery networks,  
Internet service will -- devastatingly for consumers -- become a  
duopoly.

--Brett Glass





-------------------------------------------
Archives: https://www.listbox.com/member/archive/247/=now
RSS Feed: https://www.listbox.com/member/archive/rss/247/
Powered by Listbox: http://www.listbox.com