nanog mailing list archives
Re: L3 East cost maint / fiber 05FEB2012 maintenance
From: Jason Lixfeld <jason () lixfeld ca>
Date: Tue, 5 Feb 2013 11:00:56 -0500
I got notification of their maintenance window, albeit with < 24 hours notice. Notice came in at 11:00GMT-5 yesterday, maintenance was scheduled for 00:00GMT-5 this morning. That said, the notice said that the maintenance was in Phoenix but I got a notice about my IPT circuit at 60 Hudson which I found confusing. Based on my logs, our BGP session with them went down at 03:06GMT-5 and back up at 03:15GMT-5. Down again at 03:37GMT-5 until 04:20GMT-5. A third time at 06:41GMT-5 and back at 06:45GMT-5. Traffic graphs tell a bit of a different story. Just before 05:00GMT-5, our outbound traffic to Level 3 dropped substantially. About that time, I started getting reports about issues to Level 3 destinations. Traces seemed to indicate a black hole condition within Level 3's network in NYC, seemingly at, or just past csw3.NewYork1.Level3.net. Stuff seemed to correct itself by about 06:45GMT-5, but due to Level 3 sending only about 180k routes. About 20 minutes later, the table was back to ~431K and all's been fine since. On 2013-02-05, at 10:39 AM, Josh Reynolds <esseph () gmail com> wrote:
I know a lot of you are out of the office right now, but does anybody have any info on what happened with L3 this morning? They went into a 5 hour maintenance window with expected downtime of about 30 minutes while they upgraded something like *40* of their "core routers" (their words), but also did this during some fiber work and completely cut off several of their east coast peers for the entirety of the 5 hour window. If anybody has any more info on this, on a NOC contact for them on the East Coast for future issues, you can hit me off off-list if you don't feel comfortable replying with that info here. Thanks, and I hope hope you guys are enjoying Orlando. -- *Josh Reynolds* esseph () gmail com - (270) 302-3552
Current thread:
- L3 East cost maint / fiber 05FEB2012 maintenance Josh Reynolds (Feb 05)
- Re: L3 East cost maint / fiber 05FEB2012 maintenance Viral Vira (Feb 05)
- RE: L3 East cost maint / fiber 05FEB2012 maintenance David Hubbard (Feb 05)
- Re: L3 East cost maint / fiber 05FEB2012 maintenance Jon Lewis (Feb 05)
- Re: L3 East cost maint / fiber 05FEB2012 maintenance Jason Lixfeld (Feb 05)
- Re: L3 East cost maint / fiber 05FEB2012 maintenance Jonathan Lassoff (Feb 05)
- Re: L3 East cost maint / fiber 05FEB2012 maintenance Jason Biel (Feb 05)
- Re: L3 East cost maint / fiber 05FEB2012 maintenance Jonathan Lassoff (Feb 05)
- Re: L3 East cost maint / fiber 05FEB2012 maintenance Jason Biel (Feb 05)
- Re: L3 East cost maint / fiber 05FEB2012 maintenance joel jaeggli (Feb 05)
- Re: L3 East cost maint / fiber 05FEB2012 maintenance Jason Biel (Feb 05)
- <Possible follow-ups>
- RE: L3 East cost maint / fiber 05FEB2012 maintenance Nick Olsen (Feb 05)
- RE: L3 East cost maint / fiber 05FEB2012 maintenance 2asx1y702 (Feb 05)
- Re: L3 East cost maint / fiber 05FEB2012 maintenance Ben Bartsch (Feb 05)