nanog mailing list archives

Re: Level3 routing issue US west coast

From: Joseph Jenkins <joe () breathe-underwater com>
Date: Fri, 10 Jul 2015 05:55:07 -0700

Level3 had an issue with one of their core routers in Los Angeles last night(7pm Pacific) and early this morning(1am 
Pacific). Last update to my trouble ticket had the issue still being reviewed by engineering, but that a core router 
was dropping packets.

On Jul 10, 2015, at 3:59 AM, Jürgen Jaritsch <jj () anexia at> wrote:

Hi,

does anyone else experience issues with the Level3 network at the US west coast? We see lots of broken paths like 
this:

                                                     Packets               Pings
Host                                               Loss%   Snt   Last   Avg  Best  Wrst StDev
1. er-01.0v-00-03.anx01.klu.at.anexia-it.com        0.0%   231    0.6   0.5   0.2  18.1   1.2
2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com        0.0%   231    0.5   9.9   0.3 361.1  40.1
3. cr-04.01-01-04.anx03.vie.at.anexia-it.com        0.0%   230    6.5   7.7   6.3  49.7   5.3
4. win-b4-link.telia.net                            0.0%   230    6.6   6.8   6.4  20.2   1.5
5. level-ic-1573273-wien-b4.c.telia.net             0.0%   230    6.6   9.3   6.3  69.1   9.6
6. ae-2-70.edge1.SanJose3.Level3.net               38.4%   230  164.8 165.0 164.5 194.9   2.6
7. ae-2-70.edge1.SanJose3.Level3.net               45.9%   230  164.7 164.8 164.5 174.1   0.9
8. 4.53.208.102                                    34.3%   230  634.9 310.7 168.5 680.1 199.7
9. TenGE5-4.br01.seo01.pccwbtn.net                 34.1%   230  412.0 455.2 304.9 954.6 203.4
10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net     40.6%   230  323.4 441.4 323.1 822.0 182.1
11. 211.115.201.92                                  38.4%   230  289.8 412.9 289.6 846.7 185.4
12. 61.250.89.2                                     35.8%   230  290.6 439.4 290.2 804.3 205.1
13. ???

Trace from NYC is also broken:

                                                         Packets               Pings
Host                                                   Loss%   Snt   Last   Avg  Best  Wrst StDev
1. cr-01.0v-00-05.anx32.nyc.us.anexia-it.com            0.0%    30    0.4   4.3   0.4  57.8  13.3
2. nyk-b5-link.telia.net                                0.0%    30    0.3   0.4   0.3   0.9   0.1
3. ???
4. ae-3-80.edge1.SanJose3.Level3.net                   17.2%    30   71.7  73.2  71.7  98.7   5.5
5. ae-3-80.edge1.SanJose3.Level3.net                    0.0%    30   71.8  71.8  71.7  72.0   0.1
6. 4.53.208.102                                        31.0%    30  569.6 250.7  70.5 579.3 231.2
7. 63.218.250.73                                       31.0%    30  672.6 355.5 178.0 672.6 232.1


At 10:24 UTC+2 it was even more broken:

                                                     Packets               Pings
Host                                               Loss%   Snt   Last   Avg  Best  Wrst StDev
1. er-01.0v-00-03.anx01.klu.at.anexia-it.com        0.0%   326    0.4   0.5   0.3  39.7   2.2
2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com        0.0%   326    0.5   6.7   0.3 198.1  26.3
3. cr-04.01-01-04.anx03.vie.at.anexia-it.com        0.0%   326    6.6   7.6   6.4  43.6   4.5
4. win-b4-link.telia.net                            0.0%   326    6.7   7.4   6.3  43.1   3.2
5. level-ic-1573273-wien-b4.c.telia.net             0.0%   326    6.9   9.2   6.3  73.2  10.1
6. ae-1-60.edge5.LosAngeles1.Level3.net            62.6%   326  164.7 165.5 164.5 176.7   1.5
   ae-2-70.edge1.SanJose3.Level3.net
7. ae-1-60.edge5.LosAngeles1.Level3.net            63.1%   326  164.8 165.8 164.6 204.2   3.9
   ae-2-70.edge1.SanJose3.Level3.net
8. 205.129.5.70                                    74.2%   326  799.9 487.2 169.0 799.9 305.7
   4.53.208.102
9. TenGE5-4.br01.seo01.pccwbtn.net                 77.2%   326  1359. 701.0 308.7 3716. 510.6
10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net     75.1%   326  960.4 643.0 323.4 960.4 307.6
11. 211.115.201.92                                  68.9%   326  925.3 674.2 289.8 932.3 296.6
12. 61.250.89.2                                     72.9%   326  928.5 637.2 291.9 928.5 304.3
13. ???


best regards

Jürgen Jaritsch
Head of Network & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300
Telefax: +43-5-0556-500

E-Mail: jj () anexia at<mailto:jj () anexia at>
Web: http://www.anexia.at<http://www.anexia.at/>

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

Current thread:

Level3 routing issue US west coast Jürgen Jaritsch (Jul 10)
- Re: Level3 routing issue US west coast Joseph Jenkins (Jul 10)
  - AW: Level3 routing issue US west coast Jürgen Jaritsch (Jul 10)
  - Re: Level3 routing issue US west coast Ca By (Jul 10)
    - AW: Level3 routing issue US west coast Jürgen Jaritsch (Jul 10)
    - AW: Level3 routing issue US west coast Jürgen Jaritsch (Jul 10)
    - AW: Level3 routing issue US west coast Jens Hoffmann (Jul 10)
    - AW: Level3 routing issue US west coast Jürgen Jaritsch (Jul 10)
    - AW: Level3 routing issue US west coast Jürgen Jaritsch (Jul 10)
    - Re: Level3 routing issue US west coast Christopher Morrow (Jul 10)
    - Re: Level3 routing issue US west coast Mr. NPP (Jul 11)
    - Re: Level3 routing issue US west coast? Donn Lasher (Jul 12)

(Thread continues...)