nanog mailing list archives

Re: Famous operational issues


From: Warren Kumari <warren () kumari net>
Date: Tue, 23 Feb 2021 14:06:01 -0500

On Mon, Feb 22, 2021 at 7:31 PM <bzs () theworld com> wrote:


At Boston Univ we discovered the hard way that a security guard's
walkie-talkie could cause a $5,000 (or $10K for the big machine room)
Halon dump.


At one of the AOL datacenters there was some convoluted fire marshal reason
why a specific door could not be locked "during business hours" (?!), and
so there was a guard permanently stationed outside. The door was all the
way around the back of the building, and so basically never used - and so
the guard would fall asleep outside it with a piece of cardboard saying
"Please wake me before entering". He was a nice guy (and it was less faff
than the main entrance), and so we'd either sneak in and just not tell
anyone, or talk loudly while going round the corner so he could pretend to
have been awake the whole time...

W





Took a couple of times before we figured out the connection tho once
someone made it to the hold button before it actually dumped.

Speaking of halon one very hot day I'm goofing off drinking coffee at
a nearby sub shop when the owner tells me someone from the computing
center was on the phone, that never happened before.

Some poor operator was holding the halon shot, it's a deadman's switch
(well, button) and the building was doing its 110db thing could I come
help? The building is being evac'd.

So my boss who wasn't the sharpest knife in the drawer follows me down
as I enter and I'm sweating like a pig with a floor panel sucker
trying to figure out which zone tripped.

And he shouts at me over the alarms: WHY TF DOES IT DO THIS?! Angrily.

I answered: well, maybe THERE'S A FIRE!!!

At which point I notice the back of my shoulder is really bothering
me, which I say to him, and he says hmmm there's a big bee on your
back maybe he's stinging you?

Fun day.

--
        -Barry Shein

Software Tool & Die    | bzs () TheWorld com             |
http://www.TheWorld.com
Purveyors to the Trade | Voice: +1 617-STD-WRLD       | 800-THE-WRLD
The World: Since 1989  | A Public Information Utility | *oo*



-- 
The computing scientist’s main challenge is not to get confused by the
complexities of his own making.
  -- E. W. Dijkstra

Current thread: