Re: Famous operational issues

[George Herbert <george.herbert () gmail com>]

Northridge quake.  I was #2 and on call at CRL.  That One Guy on dialup in Atlanta playing MUDs 23x7 pages that things 
are down.  I wander out to my computer to dial in and see what’s up, turned on TV walking past it, sat down and turned 
computer on, as it was booting on comes a live helicopter shot over Northridge showing the 1.5 remaining floors of the 
3-story Cable and Wireless building our east coast connector went through.

Took a second to listen and make sure I understood what was happening, changed channels to verify it wasn’t a stunt, 
logged  on and pinged our router there to confirm nothing there, call & wake up Jim: “East coast’s down because 
earthquake in Northridge and the C&W center fell down.”

“....oh.”

And then there was the Sidekick outage...


-George 

Sent from my iPhone

> On Feb 18, 2021, at 4:37 PM, Patrick W. Gilmore <patrick () ianai net> wrote:
>
> On Feb 18, 2021, at 6:10 PM, Karl Auer <kauer () biplane com au> wrote:
> > I think it was Macchiavelli who said that one should not ascribe to
> > malice anything adequately explained by incompetence…
>
> https://en.wikipedia.org/wiki/Hanlon%27s_razor
>    Never attribute to malice that which is adequately explained by stupidity.
>
> I personally prefer this version from Robert A. Heinlein:
>    Never underestimate the power of human stupidity.
>
> And to put it on topic, cover your EPOs
>
> In 1994, there was a major earthquake near the city of Los Angeles. City hall had to be evacuated and it would take 
> over a year to reinforce the building to make it habitable again. My company moved all the systems in the basement of 
> city hall to a new datacenter a mile or so away. After the install, we spent more than a week coaxing their ancient 
> (even for 1994) machines back online, such as a Prime Computer and an AS400 with tons of DASD. Well, tons of 
> cabinets, certainly less storage than my watch has now.
>
> I was in the DC going over something with the lady in charge when someone walked in to ask her something. She said 
> “just a second”. That person took one step to the side of the door and leaned against the wall - right on an EPO 
> which had no cover.
>
> Have you ever heard an entire row of DASD spin down instantly? Or taken 40 minutes to IPL an AS400? In the middle of 
> the business day? For the second most populous city in the country?
>
>    Me: Maybe you should get a cover for that?
>    Her: Good idea.
>
> Couple weeks later, in the same DC, going over final checklist. A fedex guy walks in. (To this day, no idea how he 
> got in a supposedly locked DC.) She says “just a second”, and I get a very strong deja vu feeling. He takes one step 
> to the side and leans against the wall.
>
>    Me: Did you order that EPO cover?
>    Her: Nope.
>
> -- 
> TTFN,
> patrick