Peering BOF VII Meeting Minutes (NANOG 30 Miami)

["William B. Norton" <wbn () equinix com>]

Hi all -

For those of you who could not attend the BOF, here are my notes from the 
Peering BOF. Comments welcome -

                        Peering BOF VII - NANOG 30 - Miami
                                2/10/2004 7PM
                        Moderator: William B. Norton

We were at capacity in the room and started right at 7PM.

The Great Debate
--------------------------
The Peering BOF started with a refereed debate on
"Restrictive Peering Policy: Makes Business Sense vs. Counter Productive."

The debate at the Peering BOF was a first miniature stab at a Internet 
Operations debate focusing on an emotional charged topic, one area 
perceived as being concealed by NDAs, hidden agendas, greedy corporate 
interests, ill will: Restrictive Peering Policies.  People attending the 
BOF heard in about 20 minutes some pretty good arguments why such a policy 
made business sense, and why it was also counter productive.

I opened sharing some of the controversy, sharing the definitions of 
peering, transit, restrictive, selective and open peerign policies, etc. 
These should be on-line at http://www.nanog.org/mtg-0402/norton.html

Vijay Gill (AOL) agreed to present the Affirmative Case, that Restrictive 
Peering Policies make business sense. He argued that Peering Policies are 
not emotional, not personal, but black & white decisions based on 
economics, a calculation on a spreadsheet. Peering is not free and there 
are scaling issues and associated operational costs that must be justified. 
Peering is also somewhat of a threat to revenue production as well, since 
peering between customers bypasses the ISP.

Avi Freedman (Akamai) agreed to present the Counter Case, that Restrictive 
Peering Policies are counter-productive. He pointed to the overhead 
associated with peering as being a rare exception; peering sessions tends 
to stay up. Avi pointed to the counter productive aspects of a Restrictive 
Peering Policy:
1) It inspires the wrong thing. Sprint will never peer with anyone that 
ever bought transit from them, and that in fact drives the "End-run, peer 
with Sprint's customers" behavior that lowers revenue for Sprint.
2) Many large networks lose revenue by not peering more openly, as it turns 
off customers, and
3) Performance improvements from peering widely *increase* revenue. Avi 
eluded to his AboveNet experience where he witnessed
a) customer behavior: customers spending more time on line because of a 
better experience,
b) lower latency and lower packet loss lead to the TCP window opening up 
faster means more data is exchanged, thus leading to more $ for those that 
charge on a per-Mbps basis. Better performance drives more revenue.

Vijay countered with the "reducto ad adsurdium" argument; the case of 
peering with each of 200 million laptops computers running zebra. The 
overhead of peering with 200M laptops would certainly fail the cost benefit 
analysis. As far as the performance argument, he claimed that the top 
visited sites from AOL show little performance difference ("imperceptible") 
between paths reached across peering versus transit links. Finally, he 
dismissed the selection of an ISP based on Peering Policy, claiming it is 
today all driven by price. The peering decision is business and 
mathematical, black & white, a pure economic decision that does what is 
best for the company.

Avi finished up by dismissing the peer-with-all-laptops as not reasonable, 
not what anyone is advocating, not what is being debated. He argued that 
the debate is really speaking to the reasonable middle ground case, where 
both parties have infrastructure deployed. Around 50% of customer traffic 
will go around you if you do not peer more openly. Restrictive Peering 
Policies analysis must include the opportunity cost of lost business and 
lost revenue, a more difficult calculation to make but one that ultimately 
shows Restrictive Peering policies as counter productive.

VERDICT: The audience voted on "which side presented the more compelling 
case". The winner: Restrictive Peering Policies are Counter Productive 
(35-43). (Editor's note: this was much closer than I think most people 
expected; the audience at the Peering BOFs are generally open or selective 
peers, and have or expect to be stubbed if trying to peer with a 
Restrictive Peering Policy peer. Both sides presented a good case.)

Cable & Wireless: A Tier 1 Peering Policy evolution & C&W migration to Savvis
------------------------------------------------------------------------------------------------------------------
After the debate, Peter Jansen (Peering Coordinator for C&W) volunteered to 
share with the audience a bit about the evolution of the C&W Peering Policy 
and what led to their current Peering Policy. He positioned a restrictive 
policy as a natural outcome of commercial interests (peering costs money) 
and but said that they will in fact peer with those that meet the peering 
criteria. There is a committee that evaluates and verifies the requests and 
there are no personalities involved here. There was a good exchange in the 
group with a handful of Q&A. Peter did a great job - he put a face to what 
was otherwise seen as cold corporate interests. He also spoke about the 
transition C&W is going through, migrating its network to Savvis.

The audience was definitely engaged and interactive during these two 
sessions. The correlation was brought up between Restrictive Peering 
Policies and Bankruptcy (Wcom, C&W, Genuity, TeleGlobe, ...). Someone made 
the point that many of the ISPs with Restrictive Peering Policies that had 
argued for making Peering decisions based on economic analysis had in fact 
gone bankrupt... How good can those spreadsheets be?

The debate generated a good sized crowd at the Peering BOF. Everyone likes 
a good fight I suppose, and the Peering BOF is one of the few places where 
people understand and care about this stuff.

Peering Personals
---------------------------
We spent the second half of the Peering BOF with Peering Personals, a 
chance for Peering Coordinators to introduce themselves to each other in a 
2 minute talk. On the screen behind them were Contact details, AS#, Peering 
Policies, and a US Map with Peering locations mapped out. They spoke about 
their network, their peering policy, what they look for in a peer and why 
others should want to peer with them.

The following people participated in the Peering Personals:
(I note below a couple remarkable points brought up during their 2 minutes...)

Company AS Number       First Name      Last Name
--------------  -----------------       ---------------- 
----------------
Akamai  12222   Patrick Gilmore
Broadwing       6395    Allison Feese
Carpathia Hosting, Inc. 29748   Scott   Bethke
CENIC   2150++  Dave    Reese
CET Networks    22934   Dennis  Nugent
        - 4Gbps of traffic, little peering (yet)
FLAG Telecom, Ltd.      15412   Nigel   Titley
Global Crossing 3549    Maurice         Dean
Japan Telecom America   4725    Amir    Arif
        - 3.5Gbps to the U.S.
JENS Co.        2915, 4682      Daizo   Tomigahara
New Edge Networks       19029   Cathy   Chen
        - DSL in many tier 2 cities
Packet Clearing House   3856, 42        Steve   Gibbard
ServePath       26228   Steve   Gibbard
SingAREN        7610    Steve   Gibbard
TDS Telecom     4181    Chris   Malayter
TeleGlobe       6453, 8297      Sylvie  LaPerriere
        - very strict peering policy, 300Mbps, US, Asia, Europe, 24x7 NOC, 
<2:1 ratio
UPC     6830    Frank   Hellemink
        - 10 countries, 14Gbps traffic, 65% peering today
Los Nettos      226     Celeste Anderson
Walmart.com     17374   Will    Campbell
Yahoo   5779, 10310     Brokaw  Price
        - Brokaw contested the operational load argument from the debate;
        - they have over 600 peering sessions and the load is mostly virtual
        - they are saving "in the seven figures" annually by peering
Microsoft       8068, 8075      Doug    Wilson
        - Doug had a good sense of humor (I flashed the BSOD 25 seconds in 
;-) )
Aleron  4200    Ryan    McClune
        - requires multiple locations across the U.S.
AARNet  7575+   Mark    Prior
China Telecom   4134    xiaoyi  liu
        - 11 Gbps between US & China
        - 80% traffic in China is China Telecom
Cable & Wireless        1273    Christian       Kaufmann
        - (PLC) in U.S. will have a more open peering policy
Adelphia Communications Corporation     18756, 19548    Dean    Deback
        - 10 Gbps in, 6Gbps out
        - ~ 40% peer2peer (see slides)
SoftbankBB      17676   Masato  YAMANISHI
        - not peering in U.S. yet, 48Gbps ingress in Japan, 24Gbps outbound
        - 10G peering
Google, Inc.    15169   Paul    Nguyen
UltraDNS        12008   Rodney  Joffe
        - 12Gbps of traffic

There were a lot of people on this list with "Pulse Peering Policies"; if 
you have a pulse, we will peer with you. (The previous name for this policy 
was replaced.) I produced a matrix showing those with open policies that 
were colocated at the same IXes, and there were a lot of green squares 
indicating that peering could/should/maybe already does occur between these 
companies.

All in all I thought it was a useful and interesting Peering BOF. We ran 
over by about 20 minutes due to the late addition of Peter's C&W talk, and 
a couple Peering Personals that were past the cut off date but too 
important for this group to pass up. Both of these however represent 
significant changes in the U.S. Peering Ecosystem that they were worth the 
time.

Comments welcome -

Bill

//
//  William B. Norton <wbn () equinix com>         +1.650.315.8635
//  Co-Founder and Chief Technical Liaison              Equinix, Inc.
//