Re: Verizon Fios, WashDC gateway problem

[Paul Paukstelis <shocksofmighty () gmail com>]

I avoided posting traceroute initially after reading the FAQ, but this 
seems like a case where it can help in diagnostics. Loss starts at the 
affected device (hop 3) and persists. See mtr report for affected and 
unaffected gateways:

Host                                      Loss%   Snt   Last Avg  Best  
Wrst StDev
 1. Wireless_Broadband_Router.home          0.0%   673    0.5 0.5   
0.4   0.6   0.0
 2. L100.WASHDC-VFTTP-123.verizon-gni.net   0.0%   673    6.5 8.1   
3.7  90.1   9.6
 3. G1-7-3-5.WASHDC-LCR-22.verizon-gni.net 11.2%   673    8.2 8.7   
5.1  15.8   1.6
 4. ???
 5. 0.ae4.BR2.IAD8.ALTER.NET               10.4%   673    9.0 8.0   
5.4  22.1   1.4
 6. be3018.ccr41.iad02.atlas.cogentco.com  10.3%   673    7.1 8.2   
5.9  15.3   1.2
 7. 38.122.63.54                           12.3%   673    7.0 8.4   
6.2  29.3   2.8
 8. ae3-asbnvacyj41.lightower.net           9.4%   672    7.1 8.3   
5.8  32.0   2.2
 9. 172.85.52.94.lightower.net             10.7%   672   10.0 9.6   
7.5  14.3   1.0
10. te1-4.css-fw-r1.net.umd.edu            11.0%   672   12.2 14.8   7.6 
287.2  27.2
11. gi6-6.ptx-core-r1.net.umd.edu           7.9%   672   10.5 11.0   7.8 
340.5  15.2
12. gi3-2.ptx-dual-r1.net.umd.edu           9.7%   672   11.9 13.1   8.3 
265.6  14.3
13. 129.2.168.3                            12.6%   672   11.2 10.9   8.5 
188.4   7.7

Host                                      Loss%   Snt   Last Avg  Best  
Wrst StDev
 1. Wireless_Broadband_Router.home          0.0%   287    0.5 0.5   
0.5   1.2   0.1
 2. L100.WASHDC-VFTTP-123.verizon-gni.net   0.0%   287    6.6 14.9   
3.7 148.3  22.8
 3. G0-10-3-4.WASHDC-LCR-21.verizon-gni.ne  0.0%   287    9.4 14.9   
5.3  97.2  19.1
 4. ???
 5. 0.ae3.BR2.IAD8.ALTER.NET                0.0%   287    7.8 13.6   
5.7  93.6  17.7
 6. be3018.ccr41.iad02.atlas.cogentco.com   0.0%   287    8.1 14.0   
6.0  94.1  18.5
 7. 38.122.63.54                            0.0%   287    9.7 15.5   
6.0  97.1  19.1
 8. ae3-asbnvacyj41.lightower.net           0.0%   287    9.4 16.4   
5.9  95.7  19.6
 9. 172.85.52.94.lightower.net              0.0%   287    9.1 19.0   
7.7  99.7  22.6
10. te1-4.css-fw-r1.net.umd.edu             0.0%   287    9.4 23.9   7.7 
269.8  35.7
11. gi6-6.ptx-core-r1.net.umd.edu           0.0%   287   10.7 18.2   8.2 
100.4  20.6
12. gi3-2.ptx-dual-r1.net.umd.edu           0.0%   287   12.2 20.5   8.5 
172.2  23.7
13. 129.2.168.2                             0.0%   287   11.1 16.4   
7.7  95.8  19.3


On 02/17/2016 02:14 PM, chris wrote:
> packet loss in the middle of the path means nothing if you have no 
> loss at the destination. do you see packet loss on the last mtr hop 
> which is your destination machine?
>
> On Tue, Feb 16, 2016 at 8:54 PM, Paul Paukstelis 
> <shocksofmighty () gmail com <mailto:shocksofmighty () gmail com>> wrote:
>
>     Apologies for a post as a non-expert, but it was suggested that I
>     see if any Verizon techs are reading here and to solicit opinions
>     on fixing a peculiar problem.
>
>     I noticed about a month ago that all upload traffic from my home
>     router (Fios) to a specific work machine was extremely slow. I
>     first thought it was an issue with the destination NIC, or
>     something with the university gateway. I had normal upload speeds
>     to other destination machines in the same building on the same
>     subnet, etc. Eventually, I swapped IPs on the two machines, and
>     found that the problem had to do with routing to a specific IP,
>     not the destination machine itself. I confirmed first with
>     wireshark that there was packet loss resulting in constant TCP
>     Fast Retransmits when uploading to the affected host IP.
>     traceroute showed that the affected and unaffected traffic went
>     through different Verizon gateways
>     (G1-7-3-5.WASHDC-LCR-22.verizon-gni.net
>     <http://G1-7-3-5.WASHDC-LCR-22.verizon-gni.net> and
>     G0-10-3-4.WASHDC-LCR-21.verizon-gni.net
>     <http://G0-10-3-4.WASHDC-LCR-21.verizon-gni.net>, respectively;
>     I've ultimately found that four different gateways are used
>     depending on the destination IP). mtr confirmed consistent 10%
>     packet loss at G1-7-3-5.WASHDC-LCR-22.verizon-gni.net
>     <http://G1-7-3-5.WASHDC-LCR-22.verizon-gni.net>. This tells me
>     that there is clearly a problem with this device, but I have yet
>     to be able to convince Verizon there is a problem. I thought I
>     might just VPN around it, but as luck would have it the university
>     VPN IP gets routed through the bad gateway. This has to be
>     impacting a significant number of Fios customers in the DC/MD
>     area. 25% of all upload traffic (assuming a normal distribution of
>     destination IPs) gets routed through this bad device.
>
>     Any thoughts on how to get in touch with someone from Verizon that
>     can help with this or any specific ideas on what this problem
>     might be which I can try to pass along?
>
>     Thanks.