nanog mailing list archives

Previous By Date Next
Previous By Thread Next

Re: Verizon Issues? East Coast US

From: Mike Tancsa <mike () sentex net>
Date: Wed, 03 Aug 2011 15:34:53 -0400


Hi,
        Yeah, I was just seeing some issues through TATA (AS6453) with routes
being blackholed in Newark, or at least that was where the packets
stopped. I had to shut my peer with them and I just finished opening a
trouble ticket.

A traceroute to 209.167.35.0/24 from a source addr in 205.211.164.0/24
byte packets
 1  teleglobe-vl38-tor (205.211.165.121)  0.259 ms  0.465 ms  0.488 ms
 2  if-2-3-513.core4.TNK-Toronto.as6453.net (209.58.16.21)  0.987 ms
0.981 ms  0.565 ms
 3  if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2)  53.390
ms  2.973 ms  2.991 ms
 4  if-3-1-0-0.tcore1.NJY-Newark.as6453.net (216.6.98.34)  20.481 ms
62.938 ms  20.967 ms
 5  if-2-2.tcore2.NJY-Newark.as6453.net (66.198.70.2)  20.986 ms  21.059
ms  20.871 ms
 6  * * *

Not sure if it was after them or coming back to them.  Same source addr
through AS174 is fine



        ---Mike


On 8/3/2011 3:27 PM, James Smallacombe wrote:


I'm having issues through Verizon too...I have a server colocated in
Vancouver...could it be a Canadian thing with Verizon?

 2  l100.phlapa-vfttp-60.verizon-gni.net (98.114.95.1)  8.910 ms  8.760
ms 7.026 ms
 3  g3-0-2-860.phlapa-lcr-08.verizon-gni.net (130.81.139.120)  10.711 ms
8.466 ms  10.698 ms
 4  * * *
 5  so-13-2-0-0.res-bb-rtr2.verizon-gni.net (130.81.19.118)  14.937 ms
15.975 ms  15.148 ms
 6  0.ae2.br2.iad8.alter.net (152.63.34.73)  14.346 ms  13.943 ms 
14.833 ms
 7  * * *
 8  * * *

I can ssh to the box from other networks, and here's a traceroute back
to my Verzon FIOS IP...other Verizon customers (DSL, etc) report same
problem:

 2  static-209-17-142-114.gtcust.grouptelecom.net (209.17.142.114) 
0.744 ms  0.642 ms  0.620 ms
 3  static-66-38-255-9.gtcust.grouptelecom.net (66.38.255.9)  0.750 ms
0.657 ms  0.649 ms
 4  GE3-0.PEERA-VANCBC.IP.GROUPTELECOM.NET (66.59.190.6)  0.730 ms 
0.694 ms  0.693 ms
 5  bx4-vancouver_G1-1-6.net.bell.ca (67.69.199.105)  0.847 ms  0.836 ms
0.828 ms
 6  core4-vancouver_ge8-0-0.net.bell.ca (64.230.183.109)  4.637 ms
    core3-vancouver_ge8-0-0.net.bell.ca (64.230.183.105)  113.794 ms
17.328 ms
 7  core2-seattle_pos4-0-0_core.net.bell.ca (64.230.144.97)  23.686 ms
    core1-seattle_pos6-0-0_core.net.bell.ca (64.230.144.89)  4.672 ms
    core2-seattle_pos4-0-0_core.net.bell.ca (64.230.144.97)  5.720 ms
 8  bx2-seattle_POS10-0-0.net.bell.ca (64.230.186.22)  4.358 ms
    bx2-seattle_POS11-0-0.net.bell.ca (64.230.186.26)  4.396 ms
    bx2-seattle_POS10-0-0.net.bell.ca (64.230.186.22)  4.353 ms
 9  Comcast-peering.net.bell.ca (67.69.246.198)  4.991 ms  4.795 ms 
4.795 ms
10  pos-0-4-0-0-cr01.seattle.wa.ibone.comcast.net (68.86.86.137)  8.285
ms 5.258 ms  4.919 ms
11  pos-0-6-0-0-cr01.denver.co.ibone.comcast.net (68.86.87.49)  46.892
ms 46.901 ms  46.952 ms
12  pos-0-13-0-0-cr01.chicago.il.ibone.comcast.net (68.86.85.245) 
57.243 ms  57.359 ms  57.333 ms
13  pos-2-13-0-0-cr01.newyork.ny.ibone.comcast.net (68.86.87.25)  85.080
ms  85.020 ms  85.069 ms
14  te-2-1-pe01.philadelphia.pa.ibone.comcast.net (68.86.84.194)  87.264
ms  86.372 ms  86.453 ms
15  75.149.230.250 (75.149.230.250)  86.548 ms  86.489 ms  86.439 ms


On Mon, 28 Feb 2011, Mike Tancsa wrote:


I was just looking at an issue between 701 in Toronto.   Seems to be
resolved now-- at least the issue I was seeing.


the bad traceroute, looked like
....
3  if-3-0-2.core3.TNK-Toronto.as6453.net (64.86.81.3)  0.988 ms  0.978
ms  1.578 ms
4  209.58.94.10 (209.58.94.10)  1.902 ms  71.416 ms  3.472 ms
5  if-3-1-0-0.tcore1.NJY-Newark.as6453.net (216.6.98.34)  22.286 ms 
21.957 ms  29.472 ms
6  if-12-0.mcore3.NJY-Newark.as6453.net (66.198.70.14)  67.961 ms
   if-3-0-0.mcore3.NJY-Newark.as6453.net (216.6.57.121)  21.449 ms 
20.956 ms
7  if-10-0.core3.NTO-NewYork.as6453.net (216.6.57.66)  21.975 ms 
22.467 ms  21.977 ms
8  Vlan1297.icore1.NTO-NewYork.as6453.net (209.58.26.49)  20.977 ms * 
30.520 ms
9  0.ae20.BR2.NYC4.ALTER.NET (204.255.168.173)  21.478 ms  21.458 ms 
21.477 ms
10  *
11  *

Now its working

....
3  if-3-0-2.core3.TNK-Toronto.as6453.net (64.86.81.3)  0.975 ms
4  209.58.94.10 (209.58.94.10)  1.975 ms
5  if-3-1-0-0.tcore1.NJY-Newark.as6453.net (216.6.98.34)  21.445 ms
6  if-12-0.mcore3.NJY-Newark.as6453.net (66.198.70.14)  54.472 ms
7  if-10-0.core3.NTO-NewYork.as6453.net (216.6.57.66)  112.426 ms
8  Vlan1297.icore1.NTO-NewYork.as6453.net (209.58.26.49)  22.964 ms
9  0.ae20.BR2.NYC4.ALTER.NET (204.255.168.173)  21.455 ms
10  0.ae2.XL4.NYC4.ALTER.NET (152.63.3.117)  21.466 ms
11  0.so-5-1-0.XT2.TOR2.ALTER.NET (152.63.128.121)  34.958 ms
12  0.POS7-1.GW2.TOR2.ALTER.NET (152.63.131.205)  33.958 ms


When it was not working, packets would not get from my AS (11647) to
the target in IP in AS701.  But packets from 701 would get back to my
AS.  The AS path in both directions are 701-6453-11647 and 11647 6453
701...  I saw a similar outage to VPNs I have in AS15290 which I see
as 11647 6453 701 15290.  However, I did not have time to check if it
was the same behaviour with loss being in one direction. In both
cases, IPs that follow 11647 174 701 and 701 174 11647 and 11647 174
7018 15290  and 15290 7018 174 11647 were not impacted.

    ---Mike



On 2/28/2011 9:53 PM, ML wrote:

Seeing some packet loss via Cogent.

www.internetpulse.net seems to be lighting up.





-- 
-------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mike () sentex net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada   http://www.tancsa.com/




James Smallacombe              PlantageNet, Inc. CEO and Janitor
up () 3 am                                http://3.am
=========================================================================





-- 
-------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mike () sentex net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada   http://www.tancsa.com/
Previous By Date Next
Previous By Thread Next

Current thread: