Interesting People mailing list archives
IP: interesting additions to Welcome to the Telecom Dark Ages
From: Dave Farber <dave () farber net>
Date: Wed, 10 Jul 2002 06:20:23 -0400
------ Forwarded Message
From: Dave Burstein <dave () dslprime com>
Date: Tue, 09 Jul 2002 21:04:43 -0400
To: Andrew Odlyzko <odlyzko () dtc umn edu>, david () priorintelligence com
Subject: Re: IP: Welcome to the Telecom Dark Ages
Andrew
I ran your data by Worldcom at the time, who simply referred me
back to Sidgmore's comments as the original source, which they stood by. So
even if he's not a crook, his judgment is so questionable I wouldn't want
him running Worldcom. I also asked Jim Crowe of Level 3 a direct question
with your data, and he simply said "We know what the numbers are, and it
doubling every 120 days."
When I first understood your data two years ago, I was so
incredulous I checked it intensely with carriers, several of whom (not for
attribution) confirmed it. That included a 10-25% actual increase per year
in bit demand per subscriber from two of the largest consumer ISPs. Add that
to new users and news apps (gas pumps), and your 100-150% number is probably
generous.
I refer to you as the $10B man, a modest estimate of what you
would have saved AT&T alone if Armstrong had listened.
db
At 08:32 PM 7/9/02, Andrew Odlyzko wrote:
David,
In your letter to UK Times (posted on Dave Farber's IP list), you make some
very good points. However, I would like to take issue with a few. For
example, you wrote:
... Rather than being an expensive asset
which remains unused, such cable actually represents an economy of scale in
deployment in respect of the costs of installing such cable.
That seems very questionable. Yes, if you are building a network (one
network)
it makes little difference in cost whether you lay down 24 fibers or 384, so
you might as well put down 384, in the hope that something unexpected will
happen to create demand for them. However, that argument fails to explain why
anyone thought it made sense to create more than a dozen new fiber networks,
each
one with lots of unlit fiber. Most were bound to go bust.
You also wrote:
... Michael O'Dell, previously of UUNET,
has empirical evidence of this point. From UUNET's experience, a 100% per
annum increase in network demand from the access environment requires a
100%
increase in capacity at the network core every 3-4 months.
You are probably referring to O'Dell November 30, 2000 message to Dave
Farber's
IP list (available at
<http://www.interesting-people.org/archives/interesting-people/>).
I was very dubious about his claim then, and am even more dubious now. In
fact, I hope that the ongoing investigations of WorldCom will unearth some
solid
data about UUNET to finally clarify some major mysteries about that network.
Over the years, every time I tried to trace the rumors of "Internet traffic
doubling
every three or four months" to their source, I always was pointed at folks
from
WorldCom, typically Ebbers or Sidgmore. Thus they seemed to be more
responsible
for inflating the Internet bubble than anyone (until it collapsed on them).
(In fact, just as the telecom industry was about to fall off a cliff, on March
7, 2000,
Ebbers was quoted in the Boston Globe as predicting that network upgrades were
going to require $100 billion.) Lots of other people (including Mike
Armstrong
and other high AT&T officials, as well as Jim Crowe of Level 3) were talking
of
Internet traffic doubling every three or four months, but they always did it
in
terms of overall Internet traffic. They never talked of THEIR network traffic
growing at these rates. Often they backed up their statements with references
to UUNET.
Now here are some of the mysteries in this area:
(I) The UUNET/WorldCom folks never (as far as I know) talked of traffic, just
of network capacity. That was true of Sidgmore's paper at Vortex 98, for
example,
with its clear claim of an annual growth rate of 1,000% over several years, as
well as of the Boston Globe article cited above, of the claims on the WorldCom
Web site, and the fascinating Fiber Optics On-line interview with Jack Wimmer
(VP of network and technology planning for MCI WorldCom) that I will refer to
later, available at
<http://www.fiberopticsonline.com/content/news/article.asp?DocID={31D66203-095
B-11D4-8C32-009027DE0829}>.
Yet somehow the story cited in the press, in personal discussions, and in
dot-com
business plans was always of Internet traffic doubling every three or four
months
(corresponding to growth rates of 700% to 1,500% per year). Many times,
inside
AT&T, when I would bring up the studies Kerry Coffman and I had done (starting
in
late 1997), showing that Internet traffic was only about doubling each year
(more
precisely, growing at something between 70% and 150% per year), this work
would be
dismissed with remarks of the type
"But Mike Armstrong was at XXX last month, and heard Bernie Ebbers say
explicitly
that UUNET traffic is doubling every 100 days. We just have to try harder
to match
those growth rates and catch up with WorldCom."
So mystery #1 is, how did claims about astronomical growth rates of capacity
become transformed into a nearly universal belief about astronomical growth
rates of traffic?
(II) Mystery #2 is what was going on with the UUNET network, and in
particular
how they measure capacity. The UUNET/WorldCom folks did consistently (but
also
somewhat inconsistently, to be explained below) talk of astronomical growth
rates
in the capacity of their IP network, typically on the order of 1,000% per
year.
Now as I pointed out in the short note, "Internet growth: Myth and reality,
use
and abuse," in iMP magazine in November 2000, available at
<http://www.cisp.org/imp/november_2000/odlyzko/11_00odlyzko.htm>, to claim
that
such growth rates could have held for many years betrays astonishing
innumeracy,
a lack of appreciation for the power of compound interest. If you grow 900%
per
year (10x) between year-end 1996 and year-end 2001 (at these growth rates, you
have to be careful to get even months right, else you can be off by huge
factors),
your network will grow by a cumulative factor of 100,000x. Now by year-end
1996,
UUNET had a nationwide T3 network, so if it grew by 100,000x to the end of
last
year, it should have had lots of routes with multi-terabit capacity (if we use
any reasonable measure of capacity). Yet we don't see anything like it (and
the technology to handle such links does not exist, and the revenues of
supplier
companies, like Juniper, don't show the spending this would require).
The Sidgmore paper at Vortex 98, the Bernie Ebbers speech reported in the
March 7, 2000 Boston Globe, as well as the Jack Wimmer interview in Fiber
Optics
On-line in 2000 (as well as what seemed to be a very similar presentation
by Shoa-Kai of WorldCom at OFC 2000), and some statements on the UUNET web
page
(boasting of the size and growth rate of the network) all talked of high
growth
rates in capacity, but did not define it. The clearest statement seemed to be
the one by Mike O'Dell in his November 28, 2000 posting to the IP list. He
wrote:
I see people still don't really understand the difference
between offered load (measured as gigabits injected into
the edge of the network) and network capacity (measured in
gigabit-route-miles of trunking).
This is indeed somewhat subtle and possibly counter-intuitive.
For offered load to double every year, network capacity
must double every 4 months or so, at least in our network
(UUNET). It is slowing down a some, but that's still pretty fast.
This is actually a pretty simple result from graph theory,
once one gets the picture right (as are most results
from graph theory - grin).
This excerpt from his message appears to imply that traffic on the UUNET
network is growing about 2x each year, while capacity (measured in
gigabit-route-miles) is growing about 8x each year. This is supposed
to come from a theorem in graph theory. Now here are some of the reasons
I am dubious about it, at least under the standard interpretation of the
terms used:
1. Neither I nor anybody else I have talked to about this has been able
to reproduce this "pretty simple result from graph theory," (and I do
know at least some graph theory). Moreover, no other ISP has reported
anything like this in their network. Now 19 months ago (when O'Dell
sent his message), UUNET was larger than any other network, so one could
imagine it had run into phenomena nobody else had encountered yet. Today
it may still be the largest, but several others are about as large as
UUNET was then, and they all seem to be able to grow their capacity
at about the same rate as their traffic.
2. Growth rates of 8x per year, compounded over a few years, yield
absurd estimates for the current size of the network, as mentioned
above.
3. If your traffic grows 2x in a year, and your capacity (as measured
in gigabit-route-miles under the usual definition) grows 8x, then the
average utilization of your trunks drops by 4x (assuming that the
average distance traveled per bit stays constant, and while there is
some dispute about locality of Internet traffic, the general consensus
appears to be that distances have been long from the beginning, and
are if anything decreasing, with caching, content distribution networks,
and more dense local meshes of links). So even if the UUNET network
was running at 100% of capacity at year-end 1996, if we take O'Dell's
statement at its most natural interpretation, we conclude that UUNET
must have been running at at most 25% of capacity at year-end 1997,
at most 6.25% of capacity at year-end 1998, ...., and at at most
0.1% of capacity at year-end 2001. (The power of compound interest,
again!)
Now I aroused considerable controversy back in 1998 when I claimed
that Internet backbones were running at (long-term) average utilizations
of 10-15%, and were likely to stay at that level. (This seemed to be
inconsistent with the congestion many users experienced, and also
contradicted the dogma that the Internet would lead to more efficient
use of transport than the voice network. Yet that estimate still
seems to be valid, and there is much more evidence for it now.) However,
if we are to believe O'Dell, the real operating rate (at least on UUNET's
network) must be a far harder to believe 0.1%.
4. If WorldCom had a "pretty simple result from graph theory" that
implied capacity had to grow 8x for 2x growth in traffic, then it
was deceving its investors (and everybody else) by keeping it quiet
and not getting out of the Internet backbone business. After all,
traffic is still growing at about 2x per year (this is not just my
estimate, but others are coming up with it, for example RHK, and
Larry Roberts is claiming slightly higher rates), so this
implies that ISPs have to grow their capacity by at least 8x each year.
Yet if we look at even the most optimistic forecasts of technological
progress, it does not allow for prices to decline by more than
50% per year in the long term. Thus total revenues would have to grow at
least 4x per year, or 2x per bit carried. That would say that IP is a
disfunctional technology, suffering from diseconomies of scale. Any
honest management faced with such a business proposition would surely run
away from it and invest its shareholders' money in something more
promising, like garbage disposal, or even bank CDs.
5. In the 19 months since O'Dell's message, WorldCom has slashed
its capital spending, and its suppliers (such as Juniper) are reeling.
Yet according to various reports from other ISPs (of things such
as peering traffic with UUNET), UUNET traffic is still growing at
close to the industry's 100% annual growth rate. If O'Dell's
8x capacity for 2x traffic estimate were to hold, UUNET's capacity
would have had to grow 27x in the last 17 months. Where is the
evidence for that? Or have they perhaps found another "simple result
from graph theory" that contradicted the earlier one and enabled them
to sidestep the problem?
In summary, either O'Dell is wrong, or my interpretation of what he
wrote is wrong. Now to muddle the picture some more, let us consider
the Jack Wimmer interview mentioned above. He says:
To meet the needs of its share of 200 million Internet users, MCI
Worldcom's
UUNet division has expanded its backbone 200 times since year-end 1995,
Wimmer
reports.
Now if we assume he was talking of year-end 1999 (the interview was in
April 2000, and the anomalies are even greater if he is talking of that
date), we have 200x growth in 4 years, for annual growth rate of 3.8x.
If we assume that growth was 10x in 1996 (which is what Kerry Coffman
and I estimated for the annual growth rate of Internet traffic during
what we feel were the anomalous years 1995 and 1996 of abnormally fast
growth from a small base) then we get a growth rate of 20x over 3 years,
which comes to 2.7x per year. Either growth rate is a far cry from the
8x, 11x, or 16x that have been claimed at various times for UUNET by
various of its spokespeople.
Getting back to Wimmer, the interview then reads
The MCI WorldCom network as of 1998 took 15 years to build. Using that as
a
benchmark and overlaying the build time with the projected rate of network
performance growth, Wimmer calculates that the number of days it will take
to build the existing network starts at 242 this year, drops to 31 days
next
year, and shrinks to 0.1 by 2004.
Now if what you have today takes 242 days to build this year, a year from now
your network will be 2.8 times as large as it is today. Thus we are talking
of
a fairly reasonable 180% per year growth rate, not all that remarkable.
However,
let us consider the rest of the Wimmer quote, taking it literally. Suppose you
are
at y now, a year from now you will be at 2.8y, and then, according to Wimmer,
it will
take you 31 days to add y, which is a growth rate of 39x per year ( ( 1 +
1/2.8 )^12 ).
Now that IS astronomical!
Further, if we consider a rough estimate of going from 31 days in 2001 to 0.1
days
in 2004, we have a growth rate of 310 in 3 years, which is annual growth by
6.8x.
Which is these various growth rates is correct (if any)?
Now it could be that UUNET is using some strange measure of network capacity.
Creative accounting (using techniques such as adding up the capacities of
the PVCs in an ATM or Frame Relay network, without taking into account
overbooking) can come up with just about any number one wishes. That is
a major reason I always preferred looking at traffic as measured in bytes.
There are far fewer ambiguities to exploit there (although that is also
possible).
The bottom line is that UUNET's claims of astronomical growth rates are
inconsistent and very murky. Let us hope that some real data emerges from
the current (and likely future) investigations of WorldCom to clarify
what had really gone on in the UUNET network.
Best regards,
Andrew
-----Please note new address-----
Andrew Odlyzko
University of Minnesota
Digital Technology Center
499 Walter Library
117 Pleasant St. SE
Minneapolis, MN 55455
odlyzko () umn edu email
612-624-9510 voice phone
612-625-2002 fax
http://www.dtc.umn.edu/~odlyzko <http://www.dtc.umn.edu/%7Eodlyzko>
----------------------------------------------------------------------
Date: Sat, 29 Jun 2002 06:48:36 -0400
From: Dave Farber <dave () farber net>
Subject: IP: Welcome to the Telecom Dark Ages
He has it right. I have preached this as have many . Djf
- ------ Forwarded Message
From: "David Prior" <david () priorintelligence com>
Reply-To: <david () priorintelligence com>
Date: Sat, 29 Jun 2002 11:46:34 +0100
To: <dave () farber net>
Subject: FW: Welcome to the Telecom Dark Ages
Dave,
Further to Dana Blankenhorn's piece:
A copy of a letter I sent to the UK Times in response to an article
(http://www.timesonline.co.uk/newspaper/0,,173-339450,00.html -
subscription, sadly, required!) on the telecoms circumstance. Please feel
free to distribute via IP.
Have fun,
David.
- -----Original Message-----
From: David Prior [mailto:david () priorintelligence com]
Sent: 28 June 2002 11:51
To: letters () thetimes co uk
Subject: Welcome to the Telecom Dark Ages
Importance: High
Sir,
Your article of June 27th (Spending spree that crippled the giants of
telecoms, Nic Hopkins) only perpetuates the misconception that the current
climate for the telecommunications sector is a result of massive
overspending resulting in huge overcapacity. Your correspondent writes that
"with a limited number of people there was always going to be a confined
marketplace. Many of those thousands of miles of cable lie unused."
Those of us who have worked with the telecommunications sector recognise
that much of the fibre-optic cable installed remains unlit, awaiting demand
before being brought into service. Rather than being an expensive asset
which remains unused, such cable actually represents an economy of scale in
deployment in respect of the costs of installing such cable. As previous
commentators have remarked, to say that there is a glut of
telecommunications/network capacity because there is a lot of cable is like
saying there is a glut of microchips because there is a lot sand!
The problem lies in demand. Rather than a limited number of people to make
use of network capacity, by far the biggest problem is that this is
compounded by the limitations inherent in the access network environment. >
As
long as network access is restricted to traditional telephony infrastructures - the famous 'last mile' or 'local loop' - aggregated demand in the network core will remain low. Michael O'Dell, previously of UUNET, has empirical evidence of this point. From UUNET's experience, a 100% per annum increase in network demand from the access environment requires a 100% increase in capacity at the network core every 3-4 months. Indeed, my own research has indicated that current access network technology penetration rates (for dial-up, ISDN, xDSL, fixed-wireless, cable modem and fibre-optics) will limit around 85% of latent demand for network capacity >
at
the end of this year. It is unfortunate that this circumstance, in the UK and elsewhere, is itself perpetuated by an industry focus on the roll-out >
of
xDSL technology. xDSL makes no sense in the medium term and,. according to
a
contact at one major equipment manufacturer, is loved only because it 'turns copper into gold'. xDSL is a short-term technology solution that delivers unexpected revenues to local exchange carriers. xDSL is not, by any stretch of the imagination, 'broadband'; nor is it capable of liberating latent demand for converged network services. In early 2001, a discussion forum was launched under the title 'The Telecom Dark Ages'. This forum, participated in by peers with extensive telecommunications expertise, recognised the limitations of existing access technology and the knock-on effects of such limitations on core network providers and equipment suppliers. It seems that it is only >
now
that the rest of the industry, and those who supported its rise, are catching on to the problem. The fact of the matter is that existing approaches to 'broadband' are short-term only. Without a move towards real broadband, and by real I mean
a
network infrastructure providing at least 10Mbps of converged (voice, Internet, TV, audio, video, etc.) services, demand will remain fettered; deployed cable will remain unlit, and the telecommunications sector will continue to decline. I would suggest that, rather than focus on short-term revenues from an unsustainable technology, telecommunications providers and their investors need to wake up to the idea that now is the time to invest in a real solution based on optical fibre to the building. Although initial costs are relatively high, the potential returns for customers and suppliers alike >
are
immense. Furthermore, should existing communications players fail to grasp this opportunity, Government and private enterprise should begin to address the situation at community level with a view to providing a municipal fibre solution, operated under a wholesale access model. With such models underway in Canada, Sweden, and the United States, as well as some localities in the United Kingdom, telecommunications as we have known it is no longer a viable proposition. Yours Faithfully, David Prior p r i o r i n t e l l i g e n c e Glen Lodge Buckland Brewer Bideford Devon EX39 5LY United Kingdom Telephone: +44 (0)1805 622239 Facsimile: +44 (0)870 139 1215 Cellular: +44 (0)7811 359792 Email: david () priorintelligence com http://www.priorintelligence.com <http://www.priorintelligence.com/> Collaborative Member - VAWSS: http://www.vawss.org/panel.aspx Member - Broadband Content Coalition:
http://www.broadbandcc.org <http://www.broadbandcc.org/> ------ End of Forwarded Message
Current thread:
- IP: interesting additions to Welcome to the Telecom Dark Ages Dave Farber (Jul 10)