RE: Long hops on international paths

[Adam Thompson <athompson () merlin mb ca>]

Peering connection, I think, can explain this.
With some notable exceptions (all of whom participate here, I think), carriers still don’t throw around 100G+ peering 
routers around like sprinkles on a donut.  (Even those big networks don’t, it just looks like they do because they’re 
freaking huge.)
I suspect what you may be seeing is NOT “international carriers all concentrate on a single router” at all, but rather 
a) “large carriers tend to interconnect at only a few points” and b) “the best path from North America to Europe will 
therefore tend to always go through the same small set of inter-AS peering routers on this side”.
What you’ve described, purely in the public-visible layer 3 internet, is normal in my experience.  I would fully expect 
upwards of 90% of my daily traffic crosses one of 3 peering routers in my network, no surprise there, but ALSO I 
estimate at least 80% of it crosses one of no more than 5 or 6 routers even as I get 5, 6, 7, 8 hops away from me.  The 
internet isn’t as diversely-pathed as it once was.
In the MPLS carrier case, I’m aware of a couple that use MPLS to tunnel peering traffic from their edge back to a 
centralized “core” router that speaks BGP.  Not sure how common this is, I have a very small sample set.  But in this 
example, no matter how diverse the carrier is at L1/L2, your L3 investigation will always hit the same much-smaller set 
of routers.

To directly answer your question, the cost/benefit is driven by the fact that running BGP is (relatively) complicated, 
error-prone and expensive, compared to not running BGP.  And those routers running BGP are (broadly speaking) the 
routers controlling inter-carrier traffic.  So the “chokepoints” naturally occur as an emergent property of each 
carrier controlling their own operational and financial risk.  Very much depends on the carrier and their operational 
philosophy.  I know nearly nothing about Telia, but Zayo doesn’t (didn’t? did they get bought?) have a lot of peering 
routers – they were historically more of an L2 and/or Private-network operator, as I understand it.  (I was never a 
customer, so that’s hearsay.)
-Adam

Adam Thompson
Consultant, Infrastructure Services
[MERLIN]
100 - 135 Innovation Drive
Winnipeg, MB, R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
athompson () merlin mb ca<mailto:athompson () merlin mb ca>
www.merlin.mb.ca<http://www.merlin.mb.ca/>

From: NANOG <nanog-bounces+athompson=merlin.mb.ca () nanog org> On Behalf Of PAUL R BARFORD
Sent: Tuesday, January 18, 2022 8:49 AM
To: davidbass570 () gmail com
Cc: nanog () nanog org
Subject: Re: Long hops on international paths

Hello David,

Understanding the physical topology of the network is not​ our objective.  What we're trying to understand is the 
logical topology revealed by traceroute (we are well-aware of traceroute limitations) and why a relatively small set of 
routers in different countries tend to have the majority of the international connections.  Our expectation was that 
layer 3 connectivity revealed in traceroute to be relatively evenly spread out along coasts and near submarine landing 
points.  We're not seeing that.  So, the question is what is the cost/benefit to providers to configure/maintain routes 
(that include long MPLS tunnels) that tend to concentrate international connectivity at a relatively small number of 
routers?

Regards, PB
________________________________
From: davidbass570 () gmail com<mailto:davidbass570 () gmail com> <davidbass570 () gmail com<mailto:davidbass570 () 
gmail com>>
Sent: Tuesday, January 18, 2022 8:22 AM
To: PAUL R BARFORD <pb () cs wisc edu<mailto:pb () cs wisc edu>>
Cc: morrowc.lists () gmail com<mailto:morrowc.lists () gmail com> <morrowc.lists () gmail com<mailto:morrowc.lists () 
gmail com>>; nanog () nanog org<mailto:nanog () nanog org> <nanog () nanog org<mailto:nanog () nanog org>>
Subject: Re: Long hops on international paths

I think a large part of your problem is that you’re using trace route to try and determine the full topology of a large 
complex network.  It won’t show the full topology.

On Mon, Jan 17, 2022 at 7:43 PM PAUL R BARFORD <pb () cs wisc edu<mailto:pb () cs wisc edu>> wrote:
What we're considering specifically are consecutive (layer 3) hops as identified by traceroute.  Thus, TTL is 
decremented by 1 and no more than 1 (i.e., we have to get full information (not *****) from consecutive hops to 
consider the link).  I have asked my colleague to put together a set of examples.  We assume that there are multiple 
layer 1 and 2 links, and possibly layer 3 hops masked from traceroute by MPLS.  But what we're seeing in terms of hops 
exposed by traceroute make it look like a single (TTL decremented by 1) hop.

I'll post the examples when I get them.

PB
________________________________
From: morrowc.lists () gmail com<mailto:morrowc.lists () gmail com> <morrowc.lists () gmail com<mailto:morrowc.lists () 
gmail com>>
Sent: Monday, January 17, 2022 5:13 PM
To: PAUL R BARFORD <pb () cs wisc edu<mailto:pb () cs wisc edu>>
Cc: Pengxiong Zhu <pzhu011 () ucr edu<mailto:pzhu011 () ucr edu>>; nanog () nanog org<mailto:nanog () nanog org> <nanog 
() nanog org<mailto:nanog () nanog org>>

Subject: Re: Long hops on international paths



On Mon, Jan 17, 2022 at 5:31 PM PAUL R BARFORD <pb () cs wisc edu<mailto:pb () cs wisc edu>> wrote:
Dear Pengxiong,

Thanks for your questions:


  1.  We are using CAIDA’s Internet Topology Data Kit (ITDK) that uses the MIDAR alias resolution method to infer IP 
addresses assigned to the same router.
  2.  We understand the concerns about IP geolocation.  Interfaces of the router in question are assigned similar 
domain names e.g., “chi-b2-link.ip.twelve99.net<http://chi-b2-link.ip.twelve99.net>” (62.115.50.61). We also used 
CAIDA’s ITDK, which provides geolocation information, and indicates that this router is located in Chicago.  We 
cross-reference with Maxmind where possible.  In this particular case, there is the telltale in the use of "chi" in the 
domain name.
  3.

I think nick's point about ttl expiry and missing some context on topology still stands.
I'd be that the paths between 2 continents do not actually land in chicago... that you're seeing (or not seeing) 
missing hops between the coast(s) and chicago inside 1299's network in the US.


  1.
Hope that helps.

Regards, PB
________________________________
From: Pengxiong Zhu <pzhu011 () ucr edu<mailto:pzhu011 () ucr edu>>
Sent: Monday, January 17, 2022 3:23 PM
To: PAUL R BARFORD <pb () cs wisc edu<mailto:pb () cs wisc edu>>
Cc: nanog () nanog org<mailto:nanog () nanog org> <nanog () nanog org<mailto:nanog () nanog org>>
Subject: Re: Long hops on international paths

Hi Paul,

Just curious. How do you determine they are the same routers? Is it based on IP address or MAC addresses? Or using 
CAIDA’s router alias database?

Also how do you draw the conclusion that the AS1299 router is indeed in Chicago? IP-geolocation based on rDNS is not 
always accurate though.


Pengxiong

On Mon, Jan 17, 2022 at 10:03 AM PAUL R BARFORD <pb () cs wisc edu<mailto:pb () cs wisc edu>> wrote:
Hello,

I am a researcher at the University of Wisconsin.  My colleagues at Northwestern University and I are studying 
international Internet connectivity and would appreciate your perspective on a recent finding.

We're using traceroute data from CAIDA's Ark project for our work.  We've observed that many international links (i.e., 
a single hop on an end-to-end path that connects two countries where end points on the hop are identified via rDNS) 
tend to originate/terminate at the same routers.  Said another way, we are observing a relatively small set of routers 
in different countries tend to have a majority of the international connections - this is especially the case for hops 
that terminate in the US.  For example, there is a router operated by Telia (AS1299) in Chicago that has a high 
concentration of such links.  We were a bit surprised by this finding since even though it makes sense that the set of 
providers is relatively small (i.e., those that offer global connectivity), we assumed that the set of routers that 
used for international connectivity within any one country would tend to be more widely distributed (at least with 
respect to how they appear in traceroute data - MPLS notwithstanding).

We're interested in whether or not this is indeed standard practice and if so, the cost/benefit for configuring 
international connectivity in this way?

Any thoughts or insights you might have would be greatly appreciated - off-list responses are welcome.

Thank you.

Regards, PB

Paul Barford
University of Wisconsin - Madison

--

Regards,
Pengxiong Zhu
Department of Computer Science and Engineering
University of California, Riverside