nanog mailing list archives
RE: Followup British Telecom outage reason
From: Kevin Gannon <kgannon () lancomms ie>
Date: Sat, 24 Nov 2001 19:15:26 -0000
Anyone have the Bug ID's ? Regards, Kevin -----Original Message----- From: owner-nanog () merit edu [mailto:owner-nanog () merit edu]On Behalf Of Sean Donelan Sent: 24 November 2001 19:17 To: Neil J. McRae Cc: nanog () merit edu Subject: Re: Followup British Telecom outage reason On Sat, 24 Nov 2001, Neil J. McRae wrote:
I'd be surprised if it was the GSR, and in anycase that doesn't absolve anyone. If it was a software issue- why wasn't the software properly tested? Why was such a critical upgrade rolled out across the entire network at the same time? It doesn't add up.
It appears to be yet another CEF bug. If you want to use a GSR you are stuck using some version of IOS with a CEF bug. The question is which bug do you want. Each version of IOS has a slightly different set. Several US network providers have also been bitten by CEF bugs too. While trying to fix one set of bugs, BT upgraded of their network. I'm not sure if they were upgrading at 9am in the morning, or had upgraded earlier and the bug finally came out under load at 9am. When the BT network melted down, Cisco suggested installing a different version of IOS, which had previously been tested. At noon, BT found the new version had an even worse bug, sending packets out the wrong interface. It was until 2200 (13 hours later), BT and Cisco found a version of IOS which stablized the network. "Stablized" not fixed. The running version of IOS still has a bug, but it isn't as severe.
Current thread:
- Re: Followup British Telecom outage reason, (continued)
- Re: Followup British Telecom outage reason Ian Duncan (Nov 26)
- RE: Followup British Telecom outage reason Alex Bligh (Nov 26)
- Re: Followup British Telecom outage reason Christopher A. Woodfield (Nov 26)
- Re: Followup British Telecom outage reason jerry scharf (Nov 26)
- Re: Followup British Telecom outage reason Christopher A. Woodfield (Nov 26)
- Re: Followup British Telecom outage reason Brett Frankenberger (Nov 26)
- Re: Followup British Telecom outage reason Ryan O'Connell (Nov 27)
- Re: Followup British Telecom outage reason Alex Bligh (Nov 26)
- Re: Followup British Telecom outage reason Paul Vixie (Nov 26)
- RE: Followup British Telecom outage reason Joel Jaeggli (Nov 29)
- RE: Followup British Telecom outage reason Vadim Antonov (Nov 29)