nanog mailing list archives
Re: Famous operational issues
From: Alain Hebert <ahebert () pubnix net>
Date: Thu, 18 Feb 2021 15:31:57 -0500
A few I remember:. Some monitoring server SCSI drive failed (we're talking State/Province level govt)... Got a return back stating it will take 6 month delay to get a replacement...
Ended up choosing to use my own drive instead of leaving something that could be have been deadly, unmonitored.
. Metro interruption during rush hour (for a pop of 4M) due to overload power bar in a MMR (Meet Me Room) during a unplanned deployment;
. Cherry red and very angry looking 520-600V bus bar =D;. Fire fighters hitting the building generator emergency STOP button because some neighbor reported smoke on top of the building during a black out...
( not their fault, local gov failure as usual ). Some idiots poured gasoline into a large pipe under a bridge... ended up demonstrating the lack of diversity to the DCs on that urban island;
. Underground transformer blow up downtown Mtl and took out the entire fiber bundle, demonstrating to those customers that their diversity was actually real =D.
(took them a year to get that fixed) and . Obviously: Any rack cabling I do... ----- Alain Hebert ahebert () pubnix net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443 On 2/18/21 2:37 PM, tim () pelican org wrote:
On Thursday, 18 February, 2021 16:23, "Seth Mattinen" <sethm () rollernet us> said:I had a customer that tried to stack their servers - no rails except the bottom most one - using 2x4's between each server. Up until then I hadn't imagined anyone would want to fill their cabinet with wood, so I made a rule to ban wood and anything tangentially related (cardboard, paper, plastic, etc.). Easier to just ban all things. Fire reasons too but mainly I thought a cabinet full of wood was too stupid to allow.On the "stupid racking" front, I give you most of a rack dedicated to a single server. Not all that high a server, maybe 2U or so, but *way* too deep for the rack, so it had been installed vertically. By looping some fairly hefty chain through the handles on either side of the front of the chassis, and then bolting the four chain ends to the four rack posts. I wish I'd kept pictures of that one. Not flammable, but a serious WTF moment. Cheers, Tim.
Current thread:
- Re: Famous operational issues, (continued)
- Re: Famous operational issues Jen Linkova (Feb 19)
- Re: Famous operational issues Andrey Kostin (Feb 19)
- Re: Famous operational issues Owen DeLong (Feb 22)
- Re: Famous operational issues tim () pelican org (Feb 22)
- Re: Famous operational issues Warren Kumari (Feb 22)
- Re: Famous operational issues Patrick W. Gilmore (Feb 22)
- Re: Famous operational issues Justin Streiner (Feb 23)
- Re: Famous operational issues Seth Mattinen (Feb 18)
- Re: Famous operational issues Erik Sundberg (Feb 18)
- Re: Famous operational issues tim () pelican org (Feb 18)
- Re: Famous operational issues Alain Hebert (Feb 18)
- Re: Famous operational issues Eric Kuhnke (Feb 20)
- Re: Famous operational issues Jörg Kost (Feb 20)
- Re: Famous operational issues Henry Yen (Feb 18)
- Re: Famous operational issues bzs (Feb 18)
- Re: Famous operational issues Owen DeLong (Feb 19)
- Re: Famous operational issues Mark Andrews (Feb 16)