Re: CenturyLink RCA?

[John Von Essen <john () essenz com>]

One thing that is troubling when reading that URL is that it appears 
several steps of restoration required teams to go onsite for local 
login, etc.,. Granted, to troubleshoot hardware you need to be 
physically present to pop a line card in and out, but CTL/LVL3 should 
have full out-of-band console and power control to all core devices, we 
shouldn't be waiting for someone to drive to a location to get console 
or do power cycling. And I would imagine the first step to alot of the 
troubleshooting was power cycling and local console logs.


-John



On 12/30/18 10:42 AM, Mike Hammett wrote:
> It's technical enough so that laypeople immediately lose interest, yet 
> completely useless to anyone that works with this stuff.
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
> ------------------------------------------------------------------------
> *From: *"Saku Ytti" <saku () ytti fi>
> *To: *"nanog list" <nanog () nanog org>
> *Sent: *Sunday, December 30, 2018 7:42:49 AM
> *Subject: *CenturyLink RCA?
>
> Apologies for the URL, I do not know official source and I do not
> share the URLs sentiment.
> https://fuckingcenturylink.com/
>
> Can someone translate this to IP engineer? What did actually happen?
> From my own history, I rarely recognise the problem I fixed from
> reading the public RCA. I hope CenturyLink will do better.
>
> Best guess so far that I've heard is
>
> a) CenturyLink runs global L2 DCN/OOB
> b) there was HW fault which caused L2 loop (perhaps HW dropped BPDU,
> I've had this failure mode)
> c) DCN had direct access to control-plane, and L2 congested
> control-plane resources causing it to deprovision waves
>
> Now of course this is entirely speculation, but intended to show what
> type of explanation is acceptable and can be used to fix things.
> Hopefully CenturyLink does come out with IP-engineering readable
> explanation, so that we may use it as leverage to support work in our
> own domains to remove such risks.
>
> a) do not run L2 DCN/OOB
> b) do not connect MGMT ETH (it is unprotected access to control-plane,
> it  cannot be protected by CoPP/lo0 filter/LPTS ec)
> c) do add in your RFP scoring item for proper OOB port (Like Cisco CMP)
> d) do fail optical network up
>
> --
>   ++ytti