Changes to the Internet Going on Now (telecom digest)

[Dave Farber <farber () central cis upenn edu>]

Date: Fri, 31 Mar 95 16:32:46 CST
From: telecom () delta eecs nwu edu (TELECOM Digest (Patrick Townson))


Some thoughts on the massive changes the Internet has undergone over the
past year or so and what lies ahead in the near future. I offer this
special report submitted to the Digest for your consideration this
weekend.




PAT


  Date: Wed, 29 Mar 1995 21:05:04 -0600
  From: breit () MR Net (Kelly Breit)
  Subject: FYI> Hang onto your Packets: The Information Superhighway 
                Heads to Valleyfair




Thought you might find this article interesting. It was written by a
regional ISP.


                 ------------------------


Hang onto your Packets: The Information Superhighway heads to Valleyfair


                                 or


Building a high performance computer system without reading the instructions.


March 14, 1995


Preface:


We are now in the midst of an immense transition on the Internet: The
implementation of a new structure that has been in the making for several
years. It was publicly mapped out two years ago with the release of a
solicitation for proposals by the National Science Foundation (NSF
solicitation 93-52) for four separate components of a new American national
backbone architecture. The effects of this transition will be felt over the
next few months and repercussions of it will evolve over the next few years.
There is a potential for profound effects on the immediate needs and
requirements of all MRNet members and subscribers:  namely, stable and
reliable Internet access. Because there may be difficulties and disruptions
in these services and many periods of wild ups and downs (not all that
different from a roller coaster ride at the nearby Valleyfair amusement
park), I am writing to give you some background of how we arrived at this
point and what might be expected in the near term.


This description will be part history and part personal observation. It is
not intended to be a comprehensive scholarly description, but rather my
observations based on watching things over the past year or so and some
recent events that have confirmed some of my fears and also my enthusiasms.


Ancient and Recent History:


The Internet, as we know it up to this point, has a long history (in
technology industry terms), going back to research with the ARPAnet and
introduction of the first early backbone to connect the NSF sponsored
supercomputer centers in the mid 1980s. The core of the national Internet we
are all familiar with had its start with the awarding of a 5-year
cooperative agreement in 1987 to Merit, the Michigan state networking
organization, and its partners MCI and IBM. This was to provide a national
backbone network service with a bandwidth of 1.5Mbit/sec, and several access
points around the country. Merit provided the expertise in managing a
network service, including routing, IBM provided the core backbone routing
equipment and MCI facilities were used for the trunk lines.


This system was originally research and education based and mostly higher
education institutions and research corporations were the users. A system of
regional and state networks (like MRNet) grew up around this core and
provided access to the user organizations in their areas. Most all of these
regional networks were nonprofit public-service based organizations that had
grown out of major research universities.


The popularity of the system grew at a steady pace, primarily with financial
help from the National Science Foundation, championed by the director of the
NSF Division of Network Communications and Research Infrastructure (DNCRI)
Steve Wolff. DNCRI provided funding to higher education and regional
networks to help with connectivity. The system evolved into an orderly
structure of a high-speed backbone (the NSFNET backbone service), mid-level
networks and campus and corporate networks where the end-users were
attached. Routing and management was centered at Merit and policies and
procedures evolved for the smooth operation and growth of the system.


As the popularity of such a system increased, some people saw it would be
worthwhile to provide access to this system as a business opportunity.
Entrepreneurs, with experience gained from regional network operation or
similar networking services provision, established new businesses to get in
on a new opportunity. The first generation of these were Performance Systems
International (PSI) which was spawned out of NYSERNet in New York, and
Alternet, a service of UUNet Technologies in Virginia, which had
considerable experience in UUCP network services among others.


In 1991, the Merit/IBM/MCI partnership was reformed with the creation of a
new nonprofit corporation, Advanced Networks and Services (ANS) in which the
founding partners entrusted the operation of the NSFNET backbone service.
ANS formed a for-profit subsidiary, ANS CO+RE (Commercial +
Research&Education) to offer full commercial traffic on its backbone. This
was somewhat controversial, since ANS had the advantage of the NSF subsidy
of about $10 million/year to operate the NSFNET backbone service. The
concept of NSF-sponsored research/educational-only traffic and commercial
traffic running on the same wires was a difficult concept for many to accept
and ANS was considered to have an unfair competitive advantage over PSI,
Alternet and now Sprint (who was also entering into the commercial backbone
service). This resulted in much discussion on several mailing lists,
self-appointed crusaders for justice, an Inspector General report and a
congressional hearing. None of this had much of an effect on anything in the
end, however.


Plans were underway at that time to upgrade the NSFNET backbone from its
T1-based (1.5Mbit/sec) bandwidth to a T3-based (45Mbit/sec) service. This
required new designs in routing equipment and interfaces and the transition
was a somewhat lengthy one, stretching over several months with some degree
of technical difficulty, since setting up a T3-based high-performance
backbone service with high levels of production traffic was, at that time,
on the leading edge of technology. There were a number of small disruptions
in service as the network stabilized and Merit and ANS learned how to deal
with these new technologies and rapidly growing levels of participants and
traffic.


However, there was a critical advantage to that transition, in that it was
being designed and engineered by people who had considerable experience in
operating such a system, and the new service would be provided by those same
people. There was no handoff to any new organizations. The same group
operated both the T1 and T3 backbones during the transition and the T1
backbone was always there as a fallback (though its ability to actually
handle the traffic load was lacking at that point).


As the five year cooperative agreement period advanced to its conclusion,
the NSF engaged a small team to come up with a new agreement or set of
agreements to bring the national backbone system into a new structure. The
NSF observed that its sponsorship of the backbone service, once considered
an area of advanced technology and research, was now operating as a
commodity service with several commercial networks in place. The NSF was set
to move on to other advanced network technology projects and worked to come
up with a way to withdraw from the established networking services and allow
the commercial free market to carry on. However, NSF did have a
responsibility to the educational and research activity it had been
sponsoring for so long, so it did not intend to just walk away. Rather, it
worked to come up with a scheme to facilitate an orderly transition to the
new system.


This was a lengthy process and there was considerable public input
solicited, especially from the mid-level network community. This resulted in
the publishing of a solicitation for proposals. It requested the proposals
for four areas that would comprise the new national Internet structure:


1. NAPs - Network Access Points.


The NSF proposed to sponsor a number of exchange points where national
backbone providers (also called Network Service Providers or NSPs) could
meet and exchange traffic. The NAPs would fulfill this function, and were
intended to be a level 2 service; i.e. they would operate at the data link
layer and carry the network layer (TCP/IP) traffic that was managed by the
backbone operators who connected there. The idea behind this structure was
to establish a limited number of interconnect points for the commercial
backbones. NSF's stake was to guarantee full connectivity for the research
and education community. Without the sponsorship of a core set of meet
points, backbone providers would likely set up a hodgepodge of bilateral
connect points, potentially resulting in routing chaos. The NAP operator is
to provide the exchange facility. It is up to the individual NSPs that
connect to the NAP to work out bilateral exchange agreements with the other
NAP connectees.


2. Routing Arbiter.


This would be an independent organization that would operate route servers
at each of the NAPs. These servers would contain the database of routes to
ease the transfer of traffic among the backbone providers that met at the
NAPs.


3. vBNS - Very-High-Speed Backbone Network Service


This would be the one new backbone network that the NSF would sponsor. It
was intended to be a leading-edge research network operating at a minimum
bandwidth of 155Mbit/sec with later upgrade to 622Mbit/sec. There would be a
strict acceptable use policy on this network: It could only be used for
meritorious high-bandwidth research activities. There could be no production
traffic such as general file transfers, remote logins, Web browsing or
email. That was to travel on the commercial backbones.


4. Inter-regional connectivity


These would be a series of awards made to the academic regional networks
(the group that built the original Internet). Since access to the
ANS-operated NSFNET had no gateway access charge, there would be a large
expense shock to the regional nets and their clients when they now had to
pay hefty access fees for 45Mbit or above gateways onto the commercial
backbones. The NSF proposed to award the regional nets a subsidy, declining
to zero over a four year period, to ease the transition to higher fee levels
or growth that would support the costs of commercial backbone access.


Many organizations spent the summer of 1993 responding to this solicitation
and by August of that year, they were all in. Independent review panels made
recommendations to NSF staff and in February of 1994, the first series of
awards were made:


1. NAPs


The NSF awarded 3 priority NAPs. One in the New York area, one in Chicago
and one in the San Francisco Bay area (California NAP)


a. The NY NAP was awarded to Sprint, who proposed an interim FDDI ring with
   routers, to be substituted by an ATM switch when they felt the
   technology ready.
b. The Chicago NAP was awarded to Ameritech and Bellcore. Ameritech is the
   Regional Bell Operating Company in the Great Lakes area. Bellcore is the
   research arm of the Regional phone companies. Ameritech proposed to
   install an ATM switch system using an AT&T Globeview-2000 switch.
c. The California NAP was awarded to Pacbell and Bellcore. Pacbell also
   proposed the immediate installation of an ATM switch system using 2
   Newbridge 36150 units.
d. A fourth semi-offical NAP, called the DC NAP also is being put inplaced.
   It is built and operated by Metropolitan Fiber Systems (MFS) in the
   Washington DC area. MFS is evolving its facility from its current
   Ethernet meet point, to an FDDI system.


2. Routing Arbiter


This award went to  a joint team of Merit (the routing manager of the
current NSFNET) and the Information Sciences Institute (ISI) of the
University of California. ISI will do most of the work on routing management
systems and Merit will implement the route servers and route server
database.


3. vBNS


This award went to MCI, who is implementing this service now as a 155Mbit/s
ATM service. Connections to the five NSF-sponsored supercomputer centers and
the NSF priority NAPs are under way and service is expected to be available
by April 1, 1995.


4. Inter-regional connectivity


A series of awards was made to several regional networks who chose Network
Service Providers (NSPs) to carry their traffic to the NAPs and other
backbones. Most (7-8) of the regionals chose InternetMCI. Two or three chose
SprintLink and one chose ANS.


Now What:


This brings us to where we are today: Smack in the middle of the transition
from the old NSFNET to the new structure. You will notice that this moves
the national structure from a primary R&E T3 backbone with several growing
parallel commercial backbones to a more complex system of multiple
commercial backbones with major exchange points. Previously, the ANS/NSFNET
was really the center of the national Internet. There will no longer be a
single national central backbone; indeed, traffic on the current T3
ANS/NSFNET would soon be reaching the point of saturation. There is no
current production technology that can provide a single replacement network
backbone to carry the required traffic.


What, Me Worry?


You may notice that this newer complex scheme has no single authority
overseeing the implementation. The success of the construction depends upon
the mutual cooperation of multiple phone companies, (both regional (Regional
Bell Operating Companies or RBOCs) and national long-distance (IntereXchange
Carriers or IXCs), Academic and commercial research institutions, equipment
manufacturers and regional network providers. The National Science
Foundation does not see it as its role to manage the new national Internet
structure. Merit is responsible, via the existing cooperative agreement, for
the smooth running and handoff of the existing system, but not the
management of the building of the new structure.


This new system is a massive complex assembly of components and subsystems
that must be put in place by multiple independent organizations, most of
them fierce competitors of each other, on a strict schedule (funding for the
existing NSFNET terminates irrevocably on April 1, 1995) and there is no
Project Manager.


There are also potential sources of problems in the implementation of the
components by the major players. The greatest risk comes at the NAPs. A NAP
is really a high-performance computing system, with multiple I/O channels,
sophisticated hardware and software, and a need for operational procedures
that are well planned and understood. I used to be involved in the design
and development of high-performance computing systems and there were a
couple of fundamental axioms you followed if you didn't want to fall flat on
your face:


1. Don't change technology and architecture at the same time. The
   developmental risks of trying to do two major things at once is too
   great.
2. Build a prototype that you plan to discard. This is required of any major
   systems development project, either hardware or software. The rule is,
   plan to build a prototype, because if you don't, you will build it
   anyway; it will be called version 1.0 or model 100 and will be a lot
   more expensive.


Half of the NAP operators violated these axioms. There is a major
architectural change in these new systems. It will take the NAP operators
time to figure out how to manage a major switching system with multiple
high-bandwidth backbone operators depending upon it and pouring traffic into
it. It will also take a while to work this into the grand new national
system. This is the architectural change. The NAP operators may potentially
have to deal with new technologies to implement the high-bandwidth needs of
managing the exchange of such large volumes of IP packets. This is the
technology change. There is also the issue of inexperience. With the
exception of MFS, none of the NAP operators has ever done this kind of thing
before.


MFS, in DC, has some experience at managing a meet point. It has operated
the Metropolitan Area Ethernet in the East (MAE-East) for a year or two.
This is an informal exchange point where most of the NSPs meet to exchange
traffic. They have chosen to build on this experience and expand the
technology from Ethernet to FDDI, not a great technological leap. This is a
fairly safe approach.


Sprint, in NY, has never operated a proto-NAP, but will learn how. It has
chosen to implement the new architecture, but build a prototype out of
current technology. They will then migrate it to ATM after experience is
gained at both NAP operation and when ATM technology is more proven. They
are building a prototype.


Ameritech Advanced Data Services in Chicago had determined that it doesn't
need a prototype and was going to use ATM from the start. Ditto with Pacbell
in California.


Well, it turns out that there are problems with both the ATM technology in a
high-bandwidth high-volume production application and with some of the
interface equipment required to be used with the ATM equipment. (Surprise,
surprise - a new unproven technology has some kinks yet.) Ameritech, at the
last minute in January decided to build a prototype after all and ordered an
FDDI ring and some Cisco 7000 routers to build a limited prototype.


Pacbell ran into the same problems, but planned to blast on with its initial
plans, convinced that they would make it work in time.


OK, Everybody Jump:


Beginning in Dec '94 and Jan-Feb '95, the regional networks began to move
their traffic onto their new NSP backbones. The problem was, the NAPs
weren't fully ready. MAE-East, a 10Mbit ethernet, for a while became the
center of the US Internet, since it was the only working meet-point for a
large number of NSPs. Fortunately, MAE-East evolved into MAE-East+, an FDDI
version, more capable of handling the traffic volumes.


As of this time (mid March) the Chicago NAP is still not operational. The
Pacbell NAP had already  been written off as a reliable exchange point by
most. The Sprint NAP is up, but not everyone is connected there yet. The key
players, MCI, Sprint and ANS are, however and traffic between the
MCI-connected regionals and Sprint customers is tranversing the Sprint NAP.
Traffic load through it is high because of the lack of operational status of
the other two NAPs.


Since, in January, Pacbell was still planning to go ahead with the ATM
system which many were convinced would not work, and not build a prototype,
the manager of the NASA Science Internet decided to build one for them. NASA
is installing a DEC Gigaswitch at its Ames facility and inviting any NSP or
other entity to  wire into it. It will be called MAE-West. This will provide
a critical exchange point for the west coast. Most of the western regional
networks are sending traffic to the east coast for exchange onto other
backbones. Packets from one Seattle company to another can travel to
Washington and back to reach their destination; not an optimal situation.


Pressure was brought to bear on Pacbell, and I believe they have agreed to
install FDDI equipment to finally construct a prototype. How this now fits
in with the new MAE-West is unclear.


So, When Does the Ride Start? Are we gonna crash?:


The transition is underway now. All MRNet traffic now travels via CICNet
infrastructure onto the InternetMCI national T3 backbone. Within a month or
two, our traffic will travel directly via T3 link to InternetMCI. Traffic to
SprintLink, PSI, Alternet or other customers transits the exchange points,
which at the present are either MAE-East+ or the NY NAP. Traffic to
SprintLink customers goes via MAE-East+ because Sprint does not yet have
enough bandwidth in place to connect to its NY NAP, so I've been told.


Over the next month or so, we are likely to see the Chicago NAP come up and
the MAE-West facility come up. The Pacbell NAP may also be working within a
short time also. These new exchange points will take a lot of pressure
off the few exchange points now working.


The primary reason for this long background and explanation is to let you
know that there are likely to be disruptions in Internet service, either
regional or national, that will affect your operations. It is important that
you realize this and be prepared to cope with it. There are things beyond
MRNet's control that will occur as these systems come together. These
problems will be either partial or total as the systems come online and
start carrying heavier traffic loads. There are likely also to be
disruptions on the major NSP backbones as they become accustomed to carrying
such large production loads. None of the equipment now in place has ever
been put in large-scale use under such heavy traffic loads. The people who
are operating these facilities, in some cases, are new at the job. The
transition is not complete and a lot is yet to come as the NSPs, NAPs,
Routing Arbiter system, etc. gets put into place and shaken out.


As you can see, there is an immense potential for disasters all over the
place, with no project manager, NAP operators who are inexperienced, NAP
technology that is not ready for prime-time, NSPs who have varying levels of
experience, network routers that are being stressed into new performance
territory and several other trip points. However, all is not dark. This is,
after all, the Internet - a system that grew up on lack of central authority
and management. Indeed, the fact that we have traveled this far into the
transition with only minor derailments is very encouraging and we will
probably come through in the end with only minor scrapes and bruises.


There are a lot of new people, but a lot of the people who built the
Internet as we now know it are still in action and providing expert
guidance. All players from the smallest state networks to the largest phone
companies have a lot at stake in success. The Internet is a critically
important resource and it will be made to work because so many need and want
it to work. The mutually beneficial unmanaged cooperative culture of the
Internet is strong (Use the Force, Luke.) and even the new players are
moving according to its informal ethereal guidelines.


Designers and operators have broken the rules about project management and
system development, and the technology is quirky and unpredictable. But it
is somehow coming together. It will likely be a rough and bumpy ride, hence
the whimsy of the title, and we should be prepared for that. It might take
six months to a year to feel a new sense of stability and reliability, but
it will be worth it. This new structure will provide a new basis for the
growth of the Internet that we may not be able to imagine yet. If you are
willing to hang on with us, we'll do our best to get you through.


If you have any questions or would like additional background details,
please write or call me directly or any of the MRNet engineering staff.


Dennis Fazio, executive director
Minnesota Regional Network
(612) 342-2570
dfazio () MR Net


                     ------------------------




Forwarded to TELECOM Digest by:


Kelly Breit       President and CEO
ITE/Netalliance, Inc.
6009 Wayzata Blvd., Suite 103
Minneapolis, MN 55416-1623
612-542-9440    612-542-9341 Fax