nanog mailing list archives
AW: Level3 routing issue US west coast
From: Jürgen Jaritsch <jj () anexia at>
Date: Fri, 10 Jul 2015 13:06:45 +0000
Hi Joseph, in the meantime I have ~20 verified paths which are affected and Level3 is simply not competent enough to reroute/drop the affected path ... FYI: my private ticket # is 9446435 ### There is also a new global ticket available: Network Event Detail Network Event Summary: Multiple devices were unreachable in Europe impacting IP services. Event Ticket ID: 9446797 Market Area Affected: Multiple Markets in Europe Ticket Create Date: 7/10/15 11:15:39 AM GMT Impacted For: 12 minutes Event Status: Active Time Since Last Update: 1 hour 39 minutes 7/10/15 11:26:39 AM GMT The IP NOC reported multiple devices were simultaneously unreachable in Europe impacting IP services. The devices recovered without any intervention from Level 3 after an interruption lasting twelve minutes. Due to the possibility of additional impacted, the IP NOC will monitor services until the completion of multiple ongoing events in the region. A final update will be se ### best regards Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: jj () anexia at Web: http://www.anexia.at Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces () nanog org] Im Auftrag von Joseph Jenkins Gesendet: Freitag, 10. Juli 2015 14:55 An: nanog () nanog org Betreff: Re: Level3 routing issue US west coast Level3 had an issue with one of their core routers in Los Angeles last night(7pm Pacific) and early this morning(1am Pacific). Last update to my trouble ticket had the issue still being reviewed by engineering, but that a core router was dropping packets.
On Jul 10, 2015, at 3:59 AM, Jürgen Jaritsch <jj () anexia at> wrote:
Hi,
does anyone else experience issues with the Level3 network at the US west coast? We see lots of broken paths like
this:
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. er-01.0v-00-03.anx01.klu.at.anexia-it.com 0.0% 231 0.6 0.5 0.2 18.1 1.2
2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com 0.0% 231 0.5 9.9 0.3 361.1 40.1
3. cr-04.01-01-04.anx03.vie.at.anexia-it.com 0.0% 230 6.5 7.7 6.3 49.7 5.3
4. win-b4-link.telia.net 0.0% 230 6.6 6.8 6.4 20.2 1.5
5. level-ic-1573273-wien-b4.c.telia.net 0.0% 230 6.6 9.3 6.3 69.1 9.6
6. ae-2-70.edge1.SanJose3.Level3.net 38.4% 230 164.8 165.0 164.5 194.9 2.6
7. ae-2-70.edge1.SanJose3.Level3.net 45.9% 230 164.7 164.8 164.5 174.1 0.9
8. 4.53.208.102 34.3% 230 634.9 310.7 168.5 680.1 199.7
9. TenGE5-4.br01.seo01.pccwbtn.net 34.1% 230 412.0 455.2 304.9 954.6 203.4
10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 40.6% 230 323.4 441.4 323.1 822.0 182.1
11. 211.115.201.92 38.4% 230 289.8 412.9 289.6 846.7 185.4
12. 61.250.89.2 35.8% 230 290.6 439.4 290.2 804.3 205.1
13. ???
Trace from NYC is also broken:
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. cr-01.0v-00-05.anx32.nyc.us.anexia-it.com 0.0% 30 0.4 4.3 0.4 57.8 13.3
2. nyk-b5-link.telia.net 0.0% 30 0.3 0.4 0.3 0.9 0.1
3. ???
4. ae-3-80.edge1.SanJose3.Level3.net 17.2% 30 71.7 73.2 71.7 98.7 5.5
5. ae-3-80.edge1.SanJose3.Level3.net 0.0% 30 71.8 71.8 71.7 72.0 0.1
6. 4.53.208.102 31.0% 30 569.6 250.7 70.5 579.3 231.2
7. 63.218.250.73 31.0% 30 672.6 355.5 178.0 672.6 232.1
At 10:24 UTC+2 it was even more broken:
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. er-01.0v-00-03.anx01.klu.at.anexia-it.com 0.0% 326 0.4 0.5 0.3 39.7 2.2
2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com 0.0% 326 0.5 6.7 0.3 198.1 26.3
3. cr-04.01-01-04.anx03.vie.at.anexia-it.com 0.0% 326 6.6 7.6 6.4 43.6 4.5
4. win-b4-link.telia.net 0.0% 326 6.7 7.4 6.3 43.1 3.2
5. level-ic-1573273-wien-b4.c.telia.net 0.0% 326 6.9 9.2 6.3 73.2 10.1
6. ae-1-60.edge5.LosAngeles1.Level3.net 62.6% 326 164.7 165.5 164.5 176.7 1.5
ae-2-70.edge1.SanJose3.Level3.net
7. ae-1-60.edge5.LosAngeles1.Level3.net 63.1% 326 164.8 165.8 164.6 204.2 3.9
ae-2-70.edge1.SanJose3.Level3.net
8. 205.129.5.70 74.2% 326 799.9 487.2 169.0 799.9 305.7
4.53.208.102
9. TenGE5-4.br01.seo01.pccwbtn.net 77.2% 326 1359. 701.0 308.7 3716. 510.6
10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 75.1% 326 960.4 643.0 323.4 960.4 307.6
11. 211.115.201.92 68.9% 326 925.3 674.2 289.8 932.3 296.6
12. 61.250.89.2 72.9% 326 928.5 637.2 291.9 928.5 304.3
13. ???
best regards
Jürgen Jaritsch
Head of Network & Infrastructure
ANEXIA Internetdienstleistungs GmbH
Telefon: +43-5-0556-300
Telefax: +43-5-0556-500
E-Mail: jj () anexia at<mailto:jj () anexia at>
Web: http://www.anexia.at<http://www.anexia.at/>
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Current thread:
- Level3 routing issue US west coast Jürgen Jaritsch (Jul 10)
- Re: Level3 routing issue US west coast Joseph Jenkins (Jul 10)
- AW: Level3 routing issue US west coast Jürgen Jaritsch (Jul 10)
- Re: Level3 routing issue US west coast Ca By (Jul 10)
- AW: Level3 routing issue US west coast Jürgen Jaritsch (Jul 10)
- AW: Level3 routing issue US west coast Jürgen Jaritsch (Jul 10)
- AW: Level3 routing issue US west coast Jens Hoffmann (Jul 10)
- AW: Level3 routing issue US west coast Jürgen Jaritsch (Jul 10)
- AW: Level3 routing issue US west coast Jürgen Jaritsch (Jul 10)
- Re: Level3 routing issue US west coast Christopher Morrow (Jul 10)
- Re: Level3 routing issue US west coast Mr. NPP (Jul 11)
- Re: Level3 routing issue US west coast? Donn Lasher (Jul 12)
- AW: Level3 routing issue US west coast? Jürgen Jaritsch (Jul 12)
- Re: Level3 routing issue US west coast Joseph Jenkins (Jul 10)