is something weird going on with cox, level3, and/or cogent

[Miles Fidelman <mfidelman () meetinghouse net>]

Hi Folks,

I'm visiting AZ, and seeing some really really poor performance 
accessing some of our servers via Cox broadband.  The folks at Cox 
technical support are useless - all they say is "well you're on a DOCSIS 
2 modem."  Meanwhile, everything I'm seeing is several hops upstream of 
the local segment - and all that Cox level2 tech support will say is "if 
there was a backbone problem, our backbone people would have dealt with it."

I'm having problems reaching both our own server, and sites like google, 
facebook, windows update.

Traceroutes to and from our server are illustrative - note that for most 
of the past week, the average ping time was 85msec. Now we're seeing this:

From 98.177.135.186 - the public IP address on Cox's local broadband 
service.
To 107.154.13.58 (ntcorp.server - one of our servers, sitting in a 
Tierpoint data center, near Boston)
traceroute to ntcorp.com (207.154.13.58), 64 hops max, 52 byte packets
 1  10.128.128.1 (10.128.128.1)  199.893 ms  75.319 ms  27.295 ms
 2  100.127.69.178 (100.127.69.178)  38.710 ms  40.075 ms  43.598 ms
 3  72.215.229.22 (72.215.229.22)  39.674 ms  201.368 ms *
 4  lag-157.bear2.phoenix1.level3.net (4.28.82.53)  686.499 ms 1837.141 
ms  16.273 ms
 5  ae-1-60.edge1.losangeles6.level3.net (4.69.144.16)  35.498 ms 
964.377 ms *
 6  ae-3-80.edge1.losangeles6.level3.net (4.69.144.144)  551.760 ms  
525.014 ms
    ae-1-60.edge1.losangeles6.level3.net (4.69.144.16)  2061.191 ms
 7  be3036.ccr41.lax04.atlas.cogentco.com (154.54.14.129)  847.778 ms  
87.601 ms  71.504 ms
 8  be2965.ccr22.lax01.atlas.cogentco.com (154.54.45.1)  79.060 ms  
225.647 ms
    be2964.ccr21.lax01.atlas.cogentco.com (154.54.44.77)  60.306 ms
 9  * be2931.ccr21.phx02.atlas.cogentco.com (154.54.44.85) 2264.071 ms  
185.180 ms
10  be2929.ccr21.elp01.atlas.cogentco.com (154.54.42.66)  61.208 ms
    be2930.ccr21.elp01.atlas.cogentco.com (154.54.42.78)  386.149 ms  
1278.868 ms
11  be2928.ccr22.iah01.atlas.cogentco.com (154.54.30.161)  384.136 ms
    be2927.ccr41.iah01.atlas.cogentco.com (154.54.29.221) 2339.833 ms  
615.415 ms
12  be2690.ccr42.atl01.atlas.cogentco.com (154.54.28.129)  233.061 ms
    be2687.ccr41.atl01.atlas.cogentco.com (154.54.28.69)  87.902 ms
    be2690.ccr42.atl01.atlas.cogentco.com (154.54.28.129)  861.159 ms
13  be2113.ccr42.dca01.atlas.cogentco.com (154.54.24.221)  998.858 ms
    be2112.ccr41.dca01.atlas.cogentco.com (154.54.7.157)  249.930 ms *
14  be2807.ccr42.jfk02.atlas.cogentco.com (154.54.40.109)  768.461 ms
    be2806.ccr41.jfk02.atlas.cogentco.com (154.54.40.105)  136.772 ms
    be2807.ccr42.jfk02.atlas.cogentco.com (154.54.40.109)  288.225 ms
15  be2094.ccr21.bos01.atlas.cogentco.com (154.54.30.14)  271.736 ms  
166.224 ms
    be2096.ccr22.bos01.atlas.cogentco.com (154.54.30.42)  565.015 ms
16  te0-4-1-7.agr22.bos01.atlas.cogentco.com (154.54.80.34) 1944.479 ms
    te0-4-1-6.agr22.bos01.atlas.cogentco.com (154.54.80.10) 149.803 ms
    te0-4-1-6.agr21.bos01.atlas.cogentco.com (154.54.47.230) 897.115 ms
17  te0-0-2-0.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.82)  
107.207 ms
    te0-0-2-1.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.78)  
295.881 ms  185.453 ms
18  38.122.127.18 (38.122.127.18)  115.652 ms  461.168 ms  615.526 ms
19  207.154.0.57 (207.154.0.57)  1871.023 ms  1987.832 ms 2165.248 ms
20  server1.ntcorp.com (207.154.13.58)  587.560 ms  263.328 ms 333.542 ms


Traceroute in the reverse direction:
 1  207.154.13.47 (207.154.13.47)  0.000 ms  0.000 ms  0.000 ms
 2  * * *
 3  h130.207.190.173.static.ip.windstream.net (173.190.207.130) 0.000 
ms  0.000 ms  0.000 ms
 4  xe1-2-0-0.cr01.cley01-oh.us.windstream.net (40.128.250.166) 12.000 
ms  12.000 ms  12.000 ms
 5  et11-0-0-0.cr01.chcg01-il.us.windstream.net (40.128.248.71) 20.000 
ms  20.000 ms  20.000 ms
 6  10gigabitethernet4-1.core1.chi1.he.NET (206.223.119.37)  72.001 ms  
20.000 ms  16.000 ms
 7  chgobbrj01pos010100.r2.ch.cox.net (68.105.30.193)  20.000 ms 20.000 
ms  20.000 ms
 8  chnddsrj01-ae1.0.rd.ph.cox.net (68.1.5.211)  72.001 ms  72.001 ms  
72.001 ms
 9  * * *
10  * * *
11  * * *
<snip>
30  * * *
<looks like a lot of the later hops don't respond to pings>

Two things jump out at me:
1. The rather large number of hops from cox to ntcorp - with high 
delays  from several nodes in both the level3 and cogent networks.
2. That there's a rather more direct path from the datacenter to cox, 
that shows up in the reverse direction.

Some kind of routing or peering issue, perhaps?  (And I also note the 
earlier string of messages regarding youtube streaming problems - that 
also seemed to involve cox and level3.

Thanks for any insight (and better, for any fixes!).

Miles Fidelman




--
In theory, there is no difference between theory and practice.
In practice, there is.  .... Yogi Berra