RE: Cogent <-> Verizon peering congestion

[Ben Bridges <bbridges () springnet net>]

We've been having trouble with congestion between Verizon and Cogent in Chicago since May 2013.  We had to move some 
traffic off of our Verizon connection to get around it.  Verizon has apparently had an internal ticket open for the 
problem since February 2013.  Their response in August 2013 was "Cogent is designing an upgrade to turn up the 
bandwidth".  Their response a few weeks ago was "Verizon is working with Cogent on increasing more capacity there is no 
deadline of completion, issue ongoing since 2013 possible completion in 2014".  In other words, they're blaming Cogent.

Ben Bridges
SpringNet

-----Original Message-----
From: Robert Glover [mailto:robertg () garlic com] 
Sent: Tuesday, February 04, 2014 5:44 PM
To: nanog () nanog org
Subject: Cogent <-> Verizon peering congestion

Hello,

For the last several months, we have been tracking a congestion issue between Cogent <-> Verizon

  Host Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. router.garlic.com 0.0%    29    0.3   6.1   0.2 160.6  29.7
  2. vl203.mag03.sfo01.atlas.cogentco.com 0.0%    29    2.2   8.1   2.1 
161.1  29.5
  3. te0-0-0-14.ccr22.sfo01.atlas.cogentco.com 0.0%    29    2.9   2.7   
2.4   3.6   0.2
  4. be2165.ccr22.sjc01.atlas.cogentco.com 0.0%    29    4.1   4.0   
3.7   4.8   0.2
  5. be2047.ccr21.sjc03.atlas.cogentco.com 0.0%    29    4.5   4.7   
4.3   5.5   0.3
  6. verizon.sjc03.atlas.cogentco.com 22.2%    28  169.3 171.5 168.1 
193.5   6.9
  7. so-1-0-0-0.SJC01-CORE-RTR2.verizon-gni.net 37.0%    28  205.8 180.6 
171.6 271.6  24.8
  8. A12-0-135.SNFCCA-DSL-01.verizon-gni.net 33.3%    28  172.3 177.5 
171.7 250.8  18.3
  9. pool-71-116-122-235.snfcca.btas.verizon.net 25.0%    28  197.9 
197.6 195.5 199.2   0.8

We have smokeping's from our side showing 30%+ packet loss from us
(AS4307) to Verizon.

All I have gotten from Cogent is a canned response:

---
The latency and/or packet loss that you are experiencing to this destination is due to occasional high traffic with 
Verizon. We have repeatedly requested augments to these congestion points and hope Verizon will comply soon.  While 
this has been escalated internally to the CEO level, we encourage you to also contact Verizon customer support with 
your concerns and complaints.  Their delay is a major impediment to internet traffic overall and contrary to net 
neutrality requirements.  
Our peering engineers will continue to address this on a daily basis until resolved.
---

It seems to have gotten a lot worse in recent days, to the point where we have customers who are trying to access us 
from Verizon's network (i.e. they have Verizon DSL, or via Verizon 3G/4GLTE) complaining they are having a very hard to 
checking their email, etc.

Has anyone else been experiencing these issues?  Or does anyone have more information that what Cogent provided me in 
their canned statement?

-Bobby