IP: on AOL outage - risks of scaling inappropriately

[Dave Farber <farber () central cis upenn edu>]

Posted-Date: Thu, 8 Aug 1996 12:05:08 -0400
Date: Thu, 08 Aug 1996 08:33:30 -0700 (MST)
From: "Joel M Snyder, Now Overwhelmed Again" <Joel_M_Snyder () Opus1 COM>




>     NEW YORK, Aug 7 (Reuter) - America Online Inc. (NASDAQ:AMER)
> suffered a service outage that left its more than 6 million
> subscribers without access to the world's largest computer
> online service for most of Wednesday, according to a
> spokeswoman.


The repercussions of this were felt a long way away from AOL.  


One of my clients is a public organization which sends out tens of
thousands of messages a day advising people on the current status of
pending legislation.  People are encouraged to subscribe to this free
service to follow their favorite bills.


Yesterday morning, I got a call because their mail system was backing
up heavily.  It took a while to discover the cause, but it turned
out to be AOL.  Because AOL's incoming mail from the Internet runs on
relatively slow systems, and because they receive hundreds of thousands of
Internet messages a day, they have 30 systems to receive incoming mail, all
pointed at from the AOL.COM name.  That means that any mail system trying
to send mail to AOL would have to individually try all 30 addresses before
giving up.  Translate that to a 60 second (typical) wait for a connection
timeout, and you've got a 30 minute time-in-queue for an AOL message.


Well, this client was using multi-threaded mailers, but because of AOL's
massive presence on the Internet & the large number of messages addressed
to them, AOL messages ended up clogging up all of the outgoing queue spots. 
Hundreds of them.  


The solution, unfortunately, is to treat AOL mail separately from other
Internet mail.  It now gets its own queue and is specifically segregated
away from other mail so that this doesn't impact other users.  The downside
of this is that ALL AOL mail is now operated (implicitly) at a lower
priority than other Internet mail, which means that AOL users have
effectively become second-class citizens.  


The risk?  While growing so large (and having all your eggs in one basket)
offers tremendous economies of scale, it has other exogenous effects.  We
are already familiar with the implicit social reaction that AOL's marketing
strategy has brought on all their users from some corners of the 
Internet: "oh you're from AOL, you must be clueless," a sort of
techno-racism aimed at the profile of the typical AOL user.  Now, AOL's size
may cause their users other discrimination of a more technical nature.




jms


Joel M Snyder, 1404 East Lind Road, Tucson, AZ, 85719
Phone: +1 520 324 0494 (voice)  +1 520 324 0495 (FAX)  
jms () Opus1 COM    http://www.opus1.com/jms    Opus One