Re: Never push the Big Red Button (New York City subway failure)

[richey goldberg <richey.goldberg () gmail com>]

I once worked for a provider who had a company next door that ran a small datacenter of about a dozen or so racks.    
They had been sold and all of their infrastructure had been virtualized and moved to the new owner’s network.    The 
last task of the local admin was to just get rid of everything.   They didn’t care how just get it gone.    So he came 
over and asked us to come take a look and we could have anything we wanted.    I picked up a few servers for lab a 
bunch of racks and stuff.

While we were working I asked the guy  “So there is absolutely nothing that’s in production in here anymore?”    He 
said “Yep” so I asked “Then if the power went off in here it wouldn’t be a big deal”   and he said “Not at all”.    
Then I asked “Can I hit the red button?”    He said “Sure, I always wondered what happened”.     I hit the button and 
with a loud booming sound the room went dead silent and then the UPS started beeping.      It was at that moment 
everyone realized that you just don’t pull the button out to restart the room.      It took us 20 minutes to figure out 
how to turn it all back on.

And with that when I got back to our office I made sure someone knew how to restart everything if we ever had to hit 
our red button.


-richey

From: NANOG <nanog-bounces+richey.goldberg=gmail.com () nanog org> on behalf of Roy <r.engehausen () gmail com>
Date: Thursday, September 16, 2021 at 12:41 AM
To: nanog <nanog () nanog org>
Subject: Re: Never push the Big Red Button (New York City subway failure)
Miy story in the late 1970s I was working in a large computer facility
with both mainframes and mil-spec 400hz computers.
Management decided that the EPO should be tested.  So we powered down
the disk and tapes.  The electrician pressed
the EPO button and NOTHING.  Everything kept running.

Turns out a wire had come loose and the fuse in the EPO circuit had blown.

Roy