nanog mailing list archives

Previous By Date Next
Previous By Thread Next

AW: Level3 routing issue US west coast

From: Jürgen Jaritsch <jj () anexia at>
Date: Fri, 10 Jul 2015 14:54:52 +0000
Wow .... Level3 responded to me that they had an issue last night .... but they simply did nothing ... for at least 10 
hours they did nothing to fix the issue:

###

Event Case ID: 9446216
Location: Los Angeles, CA
Impacted For: 10 hours 52 minutes
ETR: Unknown
Bridge: N/A


08:52 GMT - Event Conclusion Summary 

Start Time: 07/09 19:48 GMT 
Stop Time: 07/10 06:40 GMT 

Root Cause: A packet loss issue in Los Angeles, CA. 

Fix Action: The packet loss issue resolved before any action was taken. 
Summary: The IP NOC is currently investigated a packet loss issue in Los Angeles, CA that was impacting IP services. 
The packet loss issue resolved before any action was taken and the IP NOC deemed services are stable after monitoring 
was concluded.

07:53 GMT - The IP NOC is currently monitoring services after the packet loss issue resolved before any action was 
taken. An update will be sent when traffic is deemed stable or if status changes.

06:43 GMT - The IP NOC is currently investigating a packet loss issue in Los Angeles, CA that is impacting IP services. 
Troubleshooting efforts are ongoing with no estimated time to restore services available at this point. Please be 
advised that updates for this event will be relayed hourly unless otherwise noted.

###


best regards

Jürgen Jaritsch
Head of Network & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300
Telefax: +43-5-0556-500

E-Mail: jj () anexia at 
Web: http://www.anexia.at

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601


-----Ursprüngliche Nachricht-----
Von: Jürgen Jaritsch 
Gesendet: Freitag, 10. Juli 2015 15:47
An: 'Ca By'; Joseph Jenkins
Cc: nanog () nanog org
Betreff: AW: Level3 routing issue US west coast

Hi,

sitting here and watching the packet loss coming and going :(.

It changes every 10-25min. Looks like an massive issue in San Jose - routers out there sometimes have an latency from 
5-6 SECONDS ...

best regards

Jürgen Jaritsch
Head of Network & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300
Telefax: +43-5-0556-500

E-Mail: jj () anexia at 
Web: http://www.anexia.at

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601


-----Ursprüngliche Nachricht-----
Von: NANOG [mailto:nanog-bounces () nanog org] Im Auftrag von Ca By
Gesendet: Freitag, 10. Juli 2015 15:42
An: Joseph Jenkins
Cc: nanog () nanog org
Betreff: Re: Level3 routing issue US west coast

On Friday, July 10, 2015, Joseph Jenkins <joe () breathe-underwater com> wrote:


Level3 had an issue with one of their core routers in Los Angeles last
night(7pm Pacific) and early this morning(1am Pacific). Last update to my
trouble ticket had the issue still being reviewed by engineering, but that
a core router was dropping packets.



I have seen this several times with level3. They confirm packets are
dropping and sevice is degraded yet they refuse to take tactical corrective
action for hours and hours.

Makes me furious.

CB



On Jul 10, 2015, at 3:59 AM, Jürgen Jaritsch <jj () anexia at

<javascript:;>> wrote:


Hi,

does anyone else experience issues with the Level3 network at the US

west coast? We see lots of broken paths like this:


                                                     Packets

   Pings

Host                                               Loss%   Snt   Last

 Avg  Best  Wrst StDev

1. er-01.0v-00-03.anx01.klu.at.anexia-it.com        0.0%   231    0.6

 0.5   0.2  18.1   1.2

2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com        0.0%   231    0.5

 9.9   0.3 361.1  40.1

3. cr-04.01-01-04.anx03.vie.at.anexia-it.com        0.0%   230    6.5

 7.7   6.3  49.7   5.3

4. win-b4-link.telia.net                            0.0%   230    6.6

 6.8   6.4  20.2   1.5

5. level-ic-1573273-wien-b4.c.telia.net             0.0%   230    6.6

 9.3   6.3  69.1   9.6

6. ae-2-70.edge1.SanJose3.Level3.net               38.4%   230  164.8

165.0 164.5 194.9   2.6

7. ae-2-70.edge1.SanJose3.Level3.net               45.9%   230  164.7

164.8 164.5 174.1   0.9

8. 4.53.208.102                                    34.3%   230  634.9

310.7 168.5 680.1 199.7

9. TenGE5-4.br01.seo01.pccwbtn.net                 34.1%   230  412.0

455.2 304.9 954.6 203.4

10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net     40.6%   230  323.4

441.4 323.1 822.0 182.1

11. 211.115.201.92                                  38.4%   230  289.8

412.9 289.6 846.7 185.4

12. 61.250.89.2                                     35.8%   230  290.6

439.4 290.2 804.3 205.1

13. ???

Trace from NYC is also broken:

                                                         Packets

       Pings

Host                                                   Loss%   Snt

 Last   Avg  Best  Wrst StDev

1. cr-01.0v-00-05.anx32.nyc.us.anexia-it.com            0.0%    30

0.4   4.3   0.4  57.8  13.3

2. nyk-b5-link.telia.net                                0.0%    30

0.3   0.4   0.3   0.9   0.1

3. ???
4. ae-3-80.edge1.SanJose3.Level3.net                   17.2%    30

 71.7  73.2  71.7  98.7   5.5

5. ae-3-80.edge1.SanJose3.Level3.net                    0.0%    30

 71.8  71.8  71.7  72.0   0.1

6. 4.53.208.102                                        31.0%    30

569.6 250.7  70.5 579.3 231.2

7. 63.218.250.73                                       31.0%    30

672.6 355.5 178.0 672.6 232.1



At 10:24 UTC+2 it was even more broken:

                                                     Packets

   Pings

Host                                               Loss%   Snt   Last

 Avg  Best  Wrst StDev

1. er-01.0v-00-03.anx01.klu.at.anexia-it.com        0.0%   326    0.4

 0.5   0.3  39.7   2.2

2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com        0.0%   326    0.5

 6.7   0.3 198.1  26.3

3. cr-04.01-01-04.anx03.vie.at.anexia-it.com        0.0%   326    6.6

 7.6   6.4  43.6   4.5

4. win-b4-link.telia.net                            0.0%   326    6.7

 7.4   6.3  43.1   3.2

5. level-ic-1573273-wien-b4.c.telia.net             0.0%   326    6.9

 9.2   6.3  73.2  10.1

6. ae-1-60.edge5.LosAngeles1.Level3.net            62.6%   326  164.7

165.5 164.5 176.7   1.5

   ae-2-70.edge1.SanJose3.Level3.net
7. ae-1-60.edge5.LosAngeles1.Level3.net            63.1%   326  164.8

165.8 164.6 204.2   3.9

   ae-2-70.edge1.SanJose3.Level3.net
8. 205.129.5.70                                    74.2%   326  799.9

487.2 169.0 799.9 305.7

   4.53.208.102
9. TenGE5-4.br01.seo01.pccwbtn.net                 77.2%   326  1359.

701.0 308.7 3716. 510.6

10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net     75.1%   326  960.4

643.0 323.4 960.4 307.6

11. 211.115.201.92                                  68.9%   326  925.3

674.2 289.8 932.3 296.6

12. 61.250.89.2                                     72.9%   326  928.5

637.2 291.9 928.5 304.3

13. ???


best regards

Jürgen Jaritsch
Head of Network & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300
Telefax: +43-5-0556-500

E-Mail: jj () anexia at <javascript:;><mailto:jj () anexia at <javascript:;>>
Web: http://www.anexia.at<http://www.anexia.at/>

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT

U63216601
Previous By Date Next
Previous By Thread Next

Current thread: