nanog mailing list archives

Verizon Fios, WashDC gateway problem


From: Paul Paukstelis <shocksofmighty () gmail com>
Date: Tue, 16 Feb 2016 20:54:31 -0500

Apologies for a post as a non-expert, but it was suggested that I see if any Verizon techs are reading here and to solicit opinions on fixing a peculiar problem.

I noticed about a month ago that all upload traffic from my home router (Fios) to a specific work machine was extremely slow. I first thought it was an issue with the destination NIC, or something with the university gateway. I had normal upload speeds to other destination machines in the same building on the same subnet, etc. Eventually, I swapped IPs on the two machines, and found that the problem had to do with routing to a specific IP, not the destination machine itself. I confirmed first with wireshark that there was packet loss resulting in constant TCP Fast Retransmits when uploading to the affected host IP. traceroute showed that the affected and unaffected traffic went through different Verizon gateways (G1-7-3-5.WASHDC-LCR-22.verizon-gni.net and G0-10-3-4.WASHDC-LCR-21.verizon-gni.net, respectively; I've ultimately found that four different gateways are used depending on the destination IP). mtr confirmed consistent 10% packet loss at G1-7-3-5.WASHDC-LCR-22.verizon-gni.net. This tells me that there is clearly a problem with this device, but I have yet to be able to convince Verizon there is a problem. I thought I might just VPN around it, but as luck would have it the university VPN IP gets routed through the bad gateway. This has to be impacting a significant number of Fios customers in the DC/MD area. 25% of all upload traffic (assuming a normal distribution of destination IPs) gets routed through this bad device.

Any thoughts on how to get in touch with someone from Verizon that can help with this or any specific ideas on what this problem might be which I can try to pass along?

Thanks.


Current thread: