nanog mailing list archives

Re: GNi/365main Above.net peering problems?


From: Rusty Hodge <rusty () hodge com>
Date: Sat, 30 Aug 2008 13:55:28 -0700

Received this update for GNi:

At this time we believe we have found a Cisco day 0 network vulnerability.

We have 20+ routers in our core network - 6 of the 20 have the identical route
processor and IOS version.  These 6 have been affected in 3 separate
geographical locations in the past several days. The network issues range from simple SNMP failure to loss of BGP or OSPF communications. These are creating black holes intermittently across our network. We are experiencing as
much as 20% of networks not available at this time.

We have been coordinating with Cisco for the past two days and have deployed patches to address the problem. Overall, we have determined that the fastest, surest path to restoring 100% network normality is to replace these 6 routers.
We will continue this process to replace these 6 routers.

Two of the six affected routers have already been replaced and clients have
been moved.  We are now replacing the remaining 4 affected routers.
We know that this week has been very painful and frustrating for you. We will provide a detailed and open RFO Reason For Outage post mortem document as soon
as we have the replaced the affected routers.

Also, I will be updating all customers who have opened tickets at least every two hours from this time to ensure that all customers have the latest status information. If you would prefer a phone call status report, please note it
on the ticket and I will also give you a call to answer any additional
questions you may have. In addition, you may call the NOC at your convenience
at (415) 979-9786.

Thank you for your patience.


Current thread: