RE: Level3 funkiness

["Dixon, Justin" <Justin.Dixon () BBandT com>]

> -----Original Message-----
> From: J. Oquendo [mailto:sil () infiltrated net] 
> Sent: Wednesday, April 15, 2009 15:36
> To: nanog () nanog org
> Subject: Level3 funkiness
>
>
> Anyone else experience sporadic funkiness via
> Level3? I can't even reach the main website from who
> knows how many networks I've tried. Also friends
> and former colleagues have tried to reach the site
> to no avail.
>
> One of my machines on AT&T:
> # traceroute level3.net
> traceroute to level3.net (63.211.236.36), 30 hops max, 40 byte packets
>
> 4  cr1.n54ny.ip.att.net (12.122.105.58)  11.285 ms  21.702 ms  21.477
ms
> 5  ggr2.n54ny.ip.att.net (12.122.131.141)  12.712 ms  10.194 ms
16.393 ms
> 6  so-8-0-0.car3.NewYork1.Level3.net (4.68.127.149)  9.975 ms  10.019
ms  10.833 ms
> 7  vlan79.csw2.NewYork1.Level3.net (4.68.16.126)  10.162 ms  10.189 ms
14.474 ms
> 8  ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69)  15.763 ms  11.166
ms  9.725 ms
> 9  ae-3-3.ebr4.Washington1.Level3.net (4.69.132.93)  16.139 ms  30.616
ms  16.275 ms
> 10  ae-64-64.csw1.Washington1.Level3.net (4.69.134.178)  15.684 ms
ae-74->74.csw2.Washington1.Level3.net (4.69.134.182)  21.870 ms
ae-84->84.csw3.Washington1.Level3.net (4.69.134.186)  28.729 ms
> 11  ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157)  17.035 ms
ae-62->62.ebr2.Washington1.Level3.net (4.69.134.145)  17.041 ms
ae-72->72.ebr2.Washington1.Level3.net (4.69.134.149)  21.940 ms
> 12  ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69)  31.671 ms  42.407 ms
45.774 ms
> 13  ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113)  31.922 ms  32.115
ms  38.135 ms
> 14  ae-3.ebr2.Denver1.Level3.net (4.69.132.61)  75.265 ms  67.528 ms
67.937 ms
> 15  ge-9-0.hsa1.Denver1.Level3.net (4.68.107.35)  62.587 ms !H
ge-9->1.hsa1.Denver1.Level3.net (4.68.107.99)  62.543 ms !H
ge-9-2.hsa1.Denver1.Level3.net >(4.68.107.163)  75.797 ms !H
> (From Texas through Above.net)
> $ traceroute level3.net|tail -n 1
> traceroute to level3.net (63.211.236.36), 64 hops max, 40 byte packets
> 11  ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131)  21.473 ms !H *
ge-6->0.hsa1.Denver1.Level3.net (4.68.107.3)  21.547 ms !H
> Confirmed it can't be reached from Travelers Ins, The
> Hartford, none of my connections. Anyone else seeing
> issues? I'm seeing drop off from clients going through
> their Atlanta interconnects with Charter and two other
> providers, which I can't make sense of. I DO KNOW they
> experienced some sort of issue with a TDM switch or so
> they said... Very broad statements: "We know teh
> interwebs are down please stand by"
>
> I know websites are one thing, but the chances of the
> website going down, a TDM switch being wacky and now
> clients traversing their networks complaining all at
> once seems a little out of the ordinary. 
>
> =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+>=+=+
> J. Oquendo
> SGFA, SGFE, C|EH, CNDA, CHFI, OSCP
>
> "It takes 20 years to build a reputation and five minutes to
> ruin it. If you think about that, you'll do things
> differently." - Warren Buffett
>
> 227C 5D35 7DCB 0893 95AA  4771 1DCE 1FD1 5CCD 6B5E
> http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x5CCD6B5E

Looks similar from Sprints perspective via
https://www.sprint.net/lg/lg_start.php as well...

Sprint Source Region: Anaheim, CA (sl-bb20-ana)
IP Destination: 63.211.236.36
Performing: ICMP Traceroute
Tracing the route to 63.211.236.36
  1 sl-crs2-ana-0-13-5-0.sprintlink.net (144.232.1.177) 4 msec 0 msec 0
msec
  2 144.232.19.227 4 msec 24 msec
    sl-st30-la-0-4-5-0.sprintlink.net (144.232.24.41) 0 msec
  3 xe-7-3-0.edge1.losangeles9.Level3.net (4.68.111.169) [AS 3356] 4
msec
    xe-7-2-0.edge1.losangeles9.Level3.net (4.68.111.89) [AS 3356] 0 msec
0 msec
  4 vlan69.csw1.LosAngeles1.Level3.net (4.68.20.62) [AS 3356] 16 msec
    vlan99.csw4.LosAngeles1.Level3.net (4.68.20.254) [AS 3356] 12 msec
    vlan89.csw3.LosAngeles1.Level3.net (4.68.20.190) [AS 3356] 24 msec
  5 ae-63-63.ebr3.LosAngeles1.Level3.net (4.69.137.33) [AS 3356] 4 msec
12 msec
    ae-93-93.ebr3.LosAngeles1.Level3.net (4.69.137.45) [AS 3356] 4 msec
  6 ae-2.ebr3.SanJose1.Level3.net (4.69.132.9) [AS 3356] 40 msec 16 msec
16 msec
  7 ae-73-73.csw2.SanJose1.Level3.net (4.69.134.230) [AS 3356] 16 msec
    ae-63-63.csw1.SanJose1.Level3.net (4.69.134.226) [AS 3356] 12 msec
12 msec
  8 ae-82-82.ebr2.SanJose1.Level3.net (4.69.134.217) [AS 3356] 12 msec
    ae-92-92.ebr2.SanJose1.Level3.net (4.69.134.221) [AS 3356] 32 msec
12 msec
  9 ae-3.ebr1.Denver1.Level3.net (4.69.132.58) [AS 3356] 60 msec 48 msec
52 msec
 10 ge-6-1.hsa1.Denver1.Level3.net (4.68.107.67) [AS 3356] !H 
    ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131) [AS 3356] !H  * 

Completed - Wed Apr 15 15:43:21 EDT 2009 

Sprint Source Region: Atlanta, GA (sl-bb20-atl)
IP Destination: 63.211.236.36
Performing: ICMP Traceroute
Tracing the route to 63.211.236.36
  1 sl-bb21-atl-14-0.sprintlink.net (144.232.12.142) 0 msec 0 msec 0
msec
  2 sl-crs1-atl-0-4-0-3.sprintlink.net (144.232.12.148) 4 msec 0 msec 0
msec
  3 sl-crs1-rly-0-0-5-0.sprintlink.net (144.232.20.177) 20 msec 16 msec
16 msec
  4 sl-crs1-dc-0-8-0-0.sprintlink.net (144.232.19.213) 20 msec 20 msec
24 msec
  5 sl-st31-ash-0-4-0-1.sprintlink.net (144.232.19.4) 20 msec 20 msec 24
msec
  6 4.68.63.205 [AS 3356] 20 msec
    te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) [AS 3356] 20
msec 24 msec
  7 vlan79.csw2.Washington1.Level3.net (4.68.17.126) [AS 3356] 24 msec
28 msec
    vlan89.csw3.Washington1.Level3.net (4.68.17.190) [AS 3356] 28 msec
  8 ae-72-72.ebr2.Washington1.Level3.net (4.69.134.149) [AS 3356] 24
msec
    ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) [AS 3356] 24
msec 24 msec
  9 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) [AS 3356] 28 msec 28
msec 28 msec
 10 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) [AS 3356] 28 msec
28 msec 28 msec
 11 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) [AS 3356] 48 msec 48 msec
52 msec
 12 ge-9-1.hsa1.Denver1.Level3.net (4.68.107.99) [AS 3356] !H 
    ge-9-2.hsa1.Denver1.Level3.net (4.68.107.163) [AS 3356] !H  * 
Completed - Wed Apr 15 15:44:17 EDT 2009 

Sprint Source Region: Chicago, IL (sl-bb20-chi)
IP Destination: 63.211.236.36
Performing: ICMP Traceroute
Tracing the route to 63.211.236.36
  1 sl-crs1-chi-0-0-0-0.sprintlink.net (144.232.26.115) 0 msec 4 msec 0
msec
  2 sl-st20-chi-4-0-0.sprintlink.net (144.232.18.153) 0 msec
    sl-st20-chi-13-0-0.sprintlink.net (144.232.20.3) 0 msec 0 msec
  3 144.232.19.174 0 msec 0 msec 4 msec
  4 ae-31-53.ebr1.Chicago1.Level3.net (4.68.101.94) [AS 3356] 8 msec
    ae-31-51.ebr1.Chicago1.Level3.net (4.68.101.30) [AS 3356] 8 msec
    ae-31-55.ebr1.Chicago1.Level3.net (4.68.101.158) [AS 3356] 4 msec
  5 ae-6.ebr1.Chicago2.Level3.net (4.69.140.190) [AS 3356] 0 msec 4 msec
0 msec
  6 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) [AS 3356] 44 msec 44 msec
36 msec
  7 ge-9-0.hsa1.Denver1.Level3.net (4.68.107.35) [AS 3356] !H 
    ge-9-2.hsa1.Denver1.Level3.net (4.68.107.163) [AS 3356] !H  * 

Completed - Wed Apr 15 15:44:49 EDT 2009 

Sprint Source Region: Relay, MD (sl-bb20-rly)
IP Destination: 63.211.236.36
Performing: ICMP Traceroute
Tracing the route to 63.211.236.36
  1 sl-crs2-rly-0-1-0-0.sprintlink.net (144.232.3.96) 0 msec 0 msec 4
msec
  2 sl-crs2-dc-0-12-2-0.sprintlink.net (144.232.19.221) 0 msec 0 msec 0
msec
  3 sl-st31-ash-0-8-0-0.sprintlink.net (144.232.19.6) 0 msec
    sl-st31-ash-0-1-0-0.sprintlink.net (144.232.19.230) 0 msec
    sl-st31-ash-0-8-0-0.sprintlink.net (144.232.19.6) 4 msec
  4 4.68.63.205 [AS 3356] 4 msec
    te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) [AS 3356] 0
msec
    4.68.63.205 [AS 3356] 0 msec
  5 vlan69.csw1.Washington1.Level3.net (4.68.17.62) [AS 3356] 0 msec
    vlan89.csw3.Washington1.Level3.net (4.68.17.190) [AS 3356] 4 msec 12
msec
  6 ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) [AS 3356] 4 msec
    ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157) [AS 3356] 0 msec
    ae-82-82.ebr2.Washington1.Level3.net (4.69.134.153) [AS 3356] 4 msec
  7 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) [AS 3356] 20 msec 20
msec 20 msec
  8 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) [AS 3356] 20 msec
24 msec 20 msec
  9 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) [AS 3356] 52 msec 64 msec
52 msec
 10 ge-9-1.hsa1.Denver1.Level3.net (4.68.107.99) [AS 3356] !H  *  !H 

Completed - Wed Apr 15 15:45:37 EDT 2009