nanog mailing list archives

RE: Verizon: Extremely Strange CPE Routing in NYC/NJ Area

From: Matthew Huff <mhuff () ox com>
Date: Fri, 30 Nov 2018 13:17:17 +0000

Windows traceroute uses ICMP by default, Unix based systems use UDP. For example from my Mac on Fios in Westchester, NY 
(and yes, they are doing something  with ICMP traceroutes, possibly within their gateway box):


acpro:Documents mhuff$ sudo traceroute www.yahoo.com
traceroute: Warning: www.yahoo.com has multiple addresses; using 72.30.35.9
traceroute to atsv2-fp-shed.wg1.b.yahoo.com (72.30.35.9), 64 hops max, 52 byte packets
 1  firewall (10.1.1.1)  0.916 ms  0.419 ms  0.394 ms
 2  * * *
 3  b3369.nycmny-lcr-22.verizon-gni.net (130.81.16.102)  3.939 ms  10.172 ms
    b3369.nycmny-lcr-21.verizon-gni.net (130.81.16.100)  5.192 ms
 4  * * *
 5  0.ae4.xl2.buf4.alter.net (140.222.228.143)  13.500 ms *  12.970 ms
 6  0.xe-5-3-0.gw1.buf4.alter.net (152.63.26.62)  12.390 ms  12.122 ms
    0.ae4.xl2.buf4.alter.net (140.222.228.143)  12.704 ms
 7  0.xe-4-2-0.gw1.buf4.alter.net (152.63.26.46)  14.326 ms  13.069 ms
    verizon.com.customer.alter.net (204.148.47.162)  14.436 ms
 8  et-19-0-0.pat1.bfz.yahoo.com (216.115.97.103)  13.990 ms
    unknown-72-30-223-x.yahoo.com (72.30.223.3)  14.220 ms
    et-19-0-0.pat1.bfz.yahoo.com (216.115.97.103)  13.711 ms
 9  et-0-0-0.msr2.bf1.yahoo.com (74.6.227.137)  12.098 ms
    et-0-0-0.msr1.bf1.yahoo.com (74.6.227.129)  13.388 ms
    et-18-0-1.msr1.bf1.yahoo.com (74.6.227.131)  14.457 ms
10  et-19-0-0.clr2-a-gdc.bf1.yahoo.com (74.6.122.37)  14.554 ms
    et-0-0-0.msr1.bf1.yahoo.com (74.6.227.129)  14.810 ms
    et-1-1-1.msr1.bf1.yahoo.com (74.6.227.135)  13.502 ms
11  eth-18-3.bas2-1-flk.bf1.yahoo.com (98.139.128.75)  14.607 ms  16.608 ms
    eth-18-3-bas1-1-flk.bf1.yahoo.com (98.139.128.73)  15.161 ms
12  media-router-fp1.prod1.media.vip.bf1.yahoo.com (72.30.35.9)  13.470 ms
    eth-17-3.bas2-1-flk.bf1.yahoo.com (98.139.128.71)  13.582 ms
    media-router-fp1.prod1.media.vip.bf1.yahoo.com (72.30.35.9)  12.952 ms

macpro:Documents mhuff$ sudo traceroute -I www.yahoo.com
traceroute: Warning: www.yahoo.com has multiple addresses; using 72.30.35.10
traceroute to atsv2-fp-shed.wg1.b.yahoo.com (72.30.35.10), 64 hops max, 72 byte packets
 1  firewall (10.1.1.1)  0.675 ms  0.347 ms  0.322 ms
 2  media-router-fp2.prod1.media.vip.bf1.yahoo.com (72.30.35.10)  2.456 ms  21.139 ms  12.834 ms

----
Matthew Huff             | 1 Manhattanville Rd 
Director of Operations   | Purchase, NY 10577
OTA Management LLC       | Phone: 914-460-4039


-----Original Message-----
From: NANOG [mailto:nanog-bounces () nanog org] On Behalf Of Lee
Sent: Thursday, November 29, 2018 5:42 PM
To: Nick Zurku <nzurku () teraswitch com>
Cc: Brendan Mannella <bmannella () teraswitch com>; nanog () nanog org; Todd Kammerer <todd () libsyn com>
Subject: Re: Verizon: Extremely Strange CPE Routing in NYC/NJ Area

On 11/29/18, Nick Zurku <nzurku () teraswitch com> wrote:

Can anyone from Verizon take a look at this behavior for us?

We’re having multiple Verizon FiOS users in the NYC/NJ area appear to 
teleport from their FiOS router to our IP in the Pittsburgh region.


Verizon is doing something seriously weird to windows traceroute:
C:\Users\Lee>tracert www.yahoo.com

Tracing route to atsv2-fp-shed.wg1.b.yahoo.com [98.138.219.232] over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  fw.home.net
  2     1 ms    <1 ms    <1 ms  vbz-router.home.net [192.168.1.1]
  3     8 ms     3 ms     6 ms
media-router-fp2.prod1.media.vip.ne1.yahoo.com [98.138.219.232]

Trace complete.

C:\Users\Lee>ping -i 12 www.yahoo.com.

Pinging atsv2-fp-shed.wg1.b.yahoo.com [98.138.219.232] with 32 bytes of data:
Reply from 98.138.0.87: TTL expired in transit.
Reply from 98.138.0.87: TTL expired in transit.
Reply from 98.138.0.87: TTL expired in transit.
Reply from 98.138.0.87: TTL expired in transit.

Ping statistics for 98.138.219.232:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

C:\Users\Lee>ping -i 13 www.yahoo.com.

Pinging atsv2-fp-shed.wg1.b.yahoo.com [98.138.219.232] with 32 bytes of data:
Reply from 98.138.219.232: bytes=32 time=32ms TTL=54 Reply from 98.138.219.232: bytes=32 time=31ms TTL=54 Reply from 
98.138.219.232: bytes=32 time=33ms TTL=54 Reply from 98.138.219.232: bytes=32 time=33ms TTL=54

Ping statistics for 98.138.219.232:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds:
    Minimum = 31ms, Maximum = 33ms, Average = 32ms

C:\Users\Lee>

traceroute from a linux box works as expected..

Lee

Users
are seeing extreme slowness with TCP traffic, but ping times seem 
reasonable.

User 1:
 1  fios_quantum_gateway (192.168.1.1)  1.575 ms  2.426 ms  3.193 ms
 2  204.16.244.8 (204.16.244.8)  2.269 ms  3.055 ms  2.727 ms

User 2:
 1  fios_quantum_gateway (192.168.1.1)  1.565 ms  1.048 ms  0.947 ms
 2  204.16.244.8 (204.16.244.8)  2.162 ms  3.588 ms  3.048 ms

I can provide end-user NYC/NJ IPs off-list if desirable.


Here's a normal looking trace from an FiOS line locally in the 
Pittsburgh
region:


IP:  108.39.229.34
Tracing route to four.libsyn.com [204.16.244.8] over a maximum of 30 
hops:

 1    <1 ms    <1 ms    <1 ms  192.168.1.1
 2     5 ms     2 ms     7 ms  lo0-100.PITBPA-VFTTP-301.verizon-gni.net
<http://lo0-100.pitbpa-vfttp-301.verizon-gni.net/> [108.39.229.1]
 3     5 ms     6 ms     6 ms  B3301.PITBPA-LCR-22.verizon-gni.net
<http://b3301.pitbpa-lcr-22.verizon-gni.net/> [100.41.223.244]
 4     *        *        *     Request timed out.
 5     *        *        *     Request timed out.
 6    13 ms    12 ms    13 ms  0.et-7-1-5.BR1.IAD8.ALTER.NET
<http://0.et-7-1-5.br1.iad8.alter.net/> [140.222.226.17]
 7    10 ms    10 ms    10 ms  verizon.com.customer.alter.net
 [152.179.50.110]
 8    12 ms    12 ms    13 ms  be3084.ccr42.dca01.atlas.cogentco.com
 [154.54.30.65]
 9    22 ms    22 ms    22 ms  be2820.rcr21.pit02.atlas.cogentco.com
 [154.54.83.54]
10    22 ms    22 ms    21 ms  38.104.120.90
11    26 ms    21 ms    19 ms  204.16.241.133
12     *        *        *     Request timed out.
13    21 ms    21 ms    21 ms  204.16.244.8

Is this a possible traffic engineering blip? I can’t say we’ve ever 
seen trace routes return such sparse results and actually make it to 
the destination.

--
Nick Zurku
Systems Engineer
TeraSwitch, Inc.
Cell: 412-953-0481
Office: 412-945-7048
nzurku () teraswitch com

Current thread:

Verizon: Extremely Strange CPE Routing in NYC/NJ Area Nick Zurku (Nov 29)
- Re: Verizon: Extremely Strange CPE Routing in NYC/NJ Area Lee (Nov 29)
  - RE: Verizon: Extremely Strange CPE Routing in NYC/NJ Area Matthew Huff (Nov 30)