nanog mailing list archives
Re: AW: Level3 routing issue US west coast
From: joel jaeggli <joelja () bogus com>
Date: Sat, 11 Jul 2015 13:11:29 -0700
On 7/10/15 7:54 AM, Jürgen Jaritsch wrote:
Wow .... Level3 responded to me that they had an issue last night .... but they simply did nothing ... for at least 10 hours they did nothing to fix the issue:
It's more likely that there's a certain amount of nonsense and a lot of loose ends in the ticketing system; than that the issue resolved itself without intervention. You might reasonably conclude that the corrective action and the ticket are unrelated at least as far as the noc is concerned.
### Event Case ID: 9446216 Location: Los Angeles, CA Impacted For: 10 hours 52 minutes ETR: Unknown Bridge: N/A 08:52 GMT - Event Conclusion Summary Start Time: 07/09 19:48 GMT Stop Time: 07/10 06:40 GMT Root Cause: A packet loss issue in Los Angeles, CA. Fix Action: The packet loss issue resolved before any action was taken. Summary: The IP NOC is currently investigated a packet loss issue in Los Angeles, CA that was impacting IP services. The packet loss issue resolved before any action was taken and the IP NOC deemed services are stable after monitoring was concluded. 07:53 GMT - The IP NOC is currently monitoring services after the packet loss issue resolved before any action was taken. An update will be sent when traffic is deemed stable or if status changes. 06:43 GMT - The IP NOC is currently investigating a packet loss issue in Los Angeles, CA that is impacting IP services. Troubleshooting efforts are ongoing with no estimated time to restore services available at this point. Please be advised that updates for this event will be relayed hourly unless otherwise noted. ### best regards Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: jj () anexia at Web: http://www.anexia.at Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: Jürgen Jaritsch Gesendet: Freitag, 10. Juli 2015 15:47 An: 'Ca By'; Joseph Jenkins Cc: nanog () nanog org Betreff: AW: Level3 routing issue US west coast Hi, sitting here and watching the packet loss coming and going :(. It changes every 10-25min. Looks like an massive issue in San Jose - routers out there sometimes have an latency from 5-6 SECONDS ... best regards Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: jj () anexia at Web: http://www.anexia.at Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces () nanog org] Im Auftrag von Ca By Gesendet: Freitag, 10. Juli 2015 15:42 An: Joseph Jenkins Cc: nanog () nanog org Betreff: Re: Level3 routing issue US west coast On Friday, July 10, 2015, Joseph Jenkins <joe () breathe-underwater com> wrote:Level3 had an issue with one of their core routers in Los Angeles last night(7pm Pacific) and early this morning(1am Pacific). Last update to my trouble ticket had the issue still being reviewed by engineering, but that a core router was dropping packets.I have seen this several times with level3. They confirm packets are dropping and sevice is degraded yet they refuse to take tactical corrective action for hours and hours. Makes me furious. CBOn Jul 10, 2015, at 3:59 AM, Jürgen Jaritsch <jj () anexia at<javascript:;>> wrote:Hi, does anyone else experience issues with the Level3 network at the USwest coast? We see lots of broken paths like this:PacketsPingsHost Loss% Snt LastAvg Best Wrst StDev1. er-01.0v-00-03.anx01.klu.at.anexia-it.com 0.0% 231 0.60.5 0.2 18.1 1.22. cr-01.0v-08-06.anx01.klu.at.anexia-it.com 0.0% 231 0.59.9 0.3 361.1 40.13. cr-04.01-01-04.anx03.vie.at.anexia-it.com 0.0% 230 6.57.7 6.3 49.7 5.34. win-b4-link.telia.net 0.0% 230 6.66.8 6.4 20.2 1.55. level-ic-1573273-wien-b4.c.telia.net 0.0% 230 6.69.3 6.3 69.1 9.66. ae-2-70.edge1.SanJose3.Level3.net 38.4% 230 164.8165.0 164.5 194.9 2.67. ae-2-70.edge1.SanJose3.Level3.net 45.9% 230 164.7164.8 164.5 174.1 0.98. 4.53.208.102 34.3% 230 634.9310.7 168.5 680.1 199.79. TenGE5-4.br01.seo01.pccwbtn.net 34.1% 230 412.0455.2 304.9 954.6 203.410. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 40.6% 230 323.4441.4 323.1 822.0 182.111. 211.115.201.92 38.4% 230 289.8412.9 289.6 846.7 185.412. 61.250.89.2 35.8% 230 290.6439.4 290.2 804.3 205.113. ??? Trace from NYC is also broken: PacketsPingsHost Loss% SntLast Avg Best Wrst StDev1. cr-01.0v-00-05.anx32.nyc.us.anexia-it.com 0.0% 300.4 4.3 0.4 57.8 13.32. nyk-b5-link.telia.net 0.0% 300.3 0.4 0.3 0.9 0.13. ??? 4. ae-3-80.edge1.SanJose3.Level3.net 17.2% 3071.7 73.2 71.7 98.7 5.55. ae-3-80.edge1.SanJose3.Level3.net 0.0% 3071.8 71.8 71.7 72.0 0.16. 4.53.208.102 31.0% 30569.6 250.7 70.5 579.3 231.27. 63.218.250.73 31.0% 30672.6 355.5 178.0 672.6 232.1At 10:24 UTC+2 it was even more broken: PacketsPingsHost Loss% Snt LastAvg Best Wrst StDev1. er-01.0v-00-03.anx01.klu.at.anexia-it.com 0.0% 326 0.40.5 0.3 39.7 2.22. cr-01.0v-08-06.anx01.klu.at.anexia-it.com 0.0% 326 0.56.7 0.3 198.1 26.33. cr-04.01-01-04.anx03.vie.at.anexia-it.com 0.0% 326 6.67.6 6.4 43.6 4.54. win-b4-link.telia.net 0.0% 326 6.77.4 6.3 43.1 3.25. level-ic-1573273-wien-b4.c.telia.net 0.0% 326 6.99.2 6.3 73.2 10.16. ae-1-60.edge5.LosAngeles1.Level3.net 62.6% 326 164.7165.5 164.5 176.7 1.5ae-2-70.edge1.SanJose3.Level3.net 7. ae-1-60.edge5.LosAngeles1.Level3.net 63.1% 326 164.8165.8 164.6 204.2 3.9ae-2-70.edge1.SanJose3.Level3.net 8. 205.129.5.70 74.2% 326 799.9487.2 169.0 799.9 305.74.53.208.102 9. TenGE5-4.br01.seo01.pccwbtn.net 77.2% 326 1359.701.0 308.7 3716. 510.610. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 75.1% 326 960.4643.0 323.4 960.4 307.611. 211.115.201.92 68.9% 326 925.3674.2 289.8 932.3 296.612. 61.250.89.2 72.9% 326 928.5637.2 291.9 928.5 304.313. ??? best regards Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: jj () anexia at <javascript:;><mailto:jj () anexia at <javascript:;>> Web: http://www.anexia.at<http://www.anexia.at/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: ATU63216601
Attachment:
signature.asc
Description: OpenPGP digital signature
Current thread:
- AW: Level3 routing issue US west coast, (continued)
- AW: Level3 routing issue US west coast Jens Hoffmann (Jul 10)
- AW: Level3 routing issue US west coast Jürgen Jaritsch (Jul 10)
- AW: Level3 routing issue US west coast Jürgen Jaritsch (Jul 10)
- Re: Level3 routing issue US west coast Christopher Morrow (Jul 10)
- Re: Level3 routing issue US west coast Mr. NPP (Jul 11)
- Re: Level3 routing issue US west coast? Donn Lasher (Jul 12)
- AW: Level3 routing issue US west coast? Jürgen Jaritsch (Jul 12)
- Re: Level3 routing issue US west coast? Yang Yu (Jul 12)
- AW: Level3 routing issue US west coast? Jürgen Jaritsch (Jul 12)
- Re: Level3 routing issue US west coast? Joe Hamelin (Jul 13)
- Re: AW: Level3 routing issue US west coast joel jaeggli (Jul 11)