nanog mailing list archives
Re: Long hops on international paths
From: Saku Ytti <saku () ytti fi>
Date: Tue, 18 Jan 2022 16:28:14 +0200
Hey Paul,
Thank you for the summary. We're clear about the fact that what we're seeing are MLPS paths - that was not in question. What we are not clear about and the reason for the post is why the provider - zayo.telia in this case - would decide to configure MPLS paths between Chicago and distant international locations. We assumed we would see hops in traceroute between Chicago and coastal locations and then hops that transited submarine infrastructure followed by hops to large population centers.
MPLS is the default, not exception. And like any other form of tunnelling, MPLS decouples underlay and overlay. This means, the key value proposal of tunneling is that the devices between tunnel end points do not know the original sender or final receiver. This means, when TTL expires in transit, the P device may not know how to return packet to sender. There are three cases here 1) MPLS-TTL does not expire in transit => easy 2) MPLS-TTL expires in transit 2a) generate TTL exceeded and put it back to tunnel, sending it to egressPE, which is guaranteed to know how to return to sender 2b) randomly assume that you know how to reach the sender and try to send the TTL exceeded directly with 2a) all P hops display egressPE latency, but it works. With 2b) it might not work, as P might not know how to return. Some devices, like Cisco, allows you heuristically to decide if to tunnel ICMP or not, based on stack depth, but this does not work. As default table during repair is as deep as vrf without repair, so we cannot really discriminate. So the best solution is to not expire in transit (the norm in tunneling), i.e. set MPLS-TTL to 255. 2nd best is to tunnel, but the RTT will confuse uneducated, or as it may be, hjghly educated, users. We could implement something like https://ytti.github.io/icmp-eo-timestamp/draft-ytti-intarea-icmp-eo-timestamp.html to offer correct forward latencies to P/LSR in 2a scenario.
Regards, PB ________________________________ From: Saku Ytti <saku () ytti fi> Sent: Tuesday, January 18, 2022 12:50 AM To: PAUL R BARFORD <pb () cs wisc edu> Cc: Lukas Tribus <lukas () ltri eu>; Esteban Carisimo <esteban.carisimo () northwestern edu>; nanog () nanog org <nanog () nanog org>; Fabian E. Bustamante <fabianb () cs northwestern edu> Subject: Re: Long hops on international paths 1) all (meaning all hitting the zayo.telia) your traceroutes originate from University in Chicago 2) the zayo.telia device is physically close to the university 3) we should expect physically close-by backbone device to be present in disproportionate amount of traceroutes 4) almost certainly zayo.telia is imposing the MPLS label of TTL 255, _NOT_ copying IP TTL, therefore until MPLS label is popped, TTL is not expiring. I.e. you are seeing ingressPE and egress PE ot Telia, you are not seeing any P routers. This is not esoteric knowledge, but a fairly basic Internet concept. I am worried you are missing too much context to produce actionable output from your work. It might be interesting to see your curriculum, why this confusion arose, why it seems logical that the reason must be that almost all waves are terminated there, because it would not seem logical for people practising in the field who have even cursory understanding, this implies problems in the curriculum. On Tue, 18 Jan 2022 at 07:21, PAUL R BARFORD <pb () cs wisc edu> wrote:Please find the examples for the case of Telia below. FROM jfk-us (jfk-us.team-probing.c008820.20201002.warts.gz) traceroute from 216.66.30.102 (Ark probe hosted in New York City, NY, US. No AS info found) to 223.114.235.32 (MAXMIXD: Turpan, CN) 1 216.66.30.101 0.365 ms 2 62.115.49.173 3.182 ms 3 * 4 62.115.137.59 17.453 ms [x] (chi-b23-link.ip.twelve99.net., CAIDA-GEOLOC -> Chicago, IL, US) 5 62.115.117.48 59.921 ms [x] (sea-b2-link.ip.twelve99.net., RIPE-IPMAP -> Seattle, WA, US) 6 62.115.171.221 69.993 ms 7 223.120.6.53 69.378 ms 8 223.120.12.34 226.225 ms 9 221.183.55.110 237.475 ms 10 221.183.25.201 238.697 ms 11 221.176.16.213 242.296 ms 12 221.183.36.62 352.695 ms 13 221.183.39.2 300.166 ms 14 117.191.8.118 316.270 ms 15 * 16 * 17 * 18 * 19 * FROM ord-us (ord-us.team-probing.c008820.20201002.warts.gz) traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 109.25.215.237 (237.215.25.109.rev.sfr.net., MAXMIXD: La Crau, FR) 1 140.192.218.129 0.795 ms 2 140.192.9.124 0.603 ms 3 64.124.44.158 1.099 ms 4 64.125.31.172 3.047 ms 5 * 6 64.125.15.65 1.895 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US) 7 62.115.118.59 99.242 ms [x] (prs-b3-link.ip.twelve99.net., CAIDA-GEOLOC -> Paris, FR) 8 62.115.154.23 105.214 ms 9 77.136.10.6 119.021 ms 10 77.136.10.6 118.830 ms 11 80.118.89.202 118.690 ms 12 80.118.89.234 118.986 ms 13 109.24.108.66 119.159 ms 14 109.25.215.237 126.085 ms traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 84.249.89.93 (dsl-tkubng12-54f959-93.dhcp.inet.fi., MAXMIXD: Turku, FI) 1 140.192.218.129 0.243 ms 2 140.192.9.124 0.326 ms 3 64.124.44.158 0.600 ms 4 * 5 * 6 64.125.15.65 1.792 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US) 7 62.115.123.27 121.199 ms [x] (hls-b4-link.ip.twelve99.net., CAIDA-GEOLOC -> Helsinki, FI) 8 * 9 141.208.193.190 127.723 ms 10 84.249.89.93 139.051 ms traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US) to 193.28.231.50 (MAXMIXD: None, HU) 1 140.192.218.129 0.240 ms 2 140.192.9.124 0.333 ms 3 64.124.44.158 0.648 ms 4 * 5 64.125.25.75 0.752 ms 6 64.125.15.65 1.877 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US) 7 62.115.119.39 123.952 ms [x] (bpt-b2-link.ip.twelve99.net., **I suspect it is in Budapest, HU**) 8 62.115.39.122 117.171 ms 9 88.151.96.148 117.202 ms 10 88.151.96.213 124.787 ms 11 * 12 * 13 * 14 * 15 * traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 152.195.4.11 (MAXMIXD: Los Angeles, CA, US) 1 140.192.218.129 0.224 ms 2 140.192.9.124 0.545 ms 3 64.124.44.158 0.640 ms 4 * 5 * 6 64.125.15.65 1.786 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US) 7 62.115.118.247 54.597 ms [x] (las-b22-link.ip.twelve99.net., CAIDA-GEOLOC -> Los Angeles, CA, US) 8 62.115.11.129 55.979 ms 9 * 10 * 11 * 12 * 13 * traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 47.31.143.217 (MAXMIXD: Delhi, IN) 1 140.192.218.129 2.277 ms 2 140.192.9.124 0.449 ms 3 64.124.44.158 0.576 ms 4 * 5 * 6 64.125.15.65 1.814 ms [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US) 7 62.115.114.41 210.056 ms [x] (snge-b5-link.ip.twelve99.net.,) 8 62.115.177.11 200.840 ms 9 103.198.140.16 233.636 ms 10 103.198.140.16 232.871 ms 11 103.198.140.171 232.648 ms 12 * 13 * 14 * 15 * 16 * ________________________________ From: Lukas Tribus <lukas () ltri eu> Sent: Monday, January 17, 2022 1:52 PM To: PAUL R BARFORD <pb () cs wisc edu> Cc: Nick Hilliard <nick () foobar org>; nanog () nanog org <nanog () nanog org>; Esteban Carisimo <esteban.carisimo () northwestern edu>; Fabian E. Bustamante <fabianb () cs northwestern edu> Subject: Re: Long hops on international paths On Mon, 17 Jan 2022 at 20:00, PAUL R BARFORD <pb () cs wisc edu> wrote:What we're curious about is why we're seeing a concentration of hops at a small number of routers that appear on international paths.I suggest you share a few actual examples (IP addresses, traceroutes). I don't think discussing your conclusion based on data we don't have makes sense. Lukas-- ++ytti
-- ++ytti
Current thread:
- Long hops on international paths PAUL R BARFORD (Jan 17)
- Re: Long hops on international paths Nick Hilliard (Jan 17)
- Re: Long hops on international paths PAUL R BARFORD (Jan 17)
- Re: Long hops on international paths Lukas Tribus (Jan 17)
- Re: Long hops on international paths PAUL R BARFORD (Jan 17)
- Re: Long hops on international paths Christopher Morrow (Jan 17)
- Re: Long hops on international paths Saku Ytti (Jan 17)
- Re: Long hops on international paths PAUL R BARFORD (Jan 18)
- Re: Long hops on international paths Saku Ytti (Jan 18)
- Re: Long hops on international paths Mark Tinka (Jan 18)
- Re: Long hops on international paths Saku Ytti (Jan 18)
- Re: Long hops on international paths Mark Tinka (Jan 18)
- Re: Long hops on international paths PAUL R BARFORD (Jan 17)
- RE: Long hops on international paths Michael Hare via NANOG (Jan 18)
- Re: Long hops on international paths Saku Ytti (Jan 18)
- Re: Long hops on international paths PAUL R BARFORD (Jan 18)
- Re: Long hops on international paths Nick Hilliard (Jan 17)
- Re: Long hops on international paths Mark Tinka (Jan 18)
- Re: Long hops on international paths Mike Hammett (Jan 18)
- Re: Long hops on international paths Lady Benjamin Cannon of Glencoe, ASCE (Jan 17)