nanog mailing list archives

Re: Never push the Big Red Button (New York City subway failure)


From: Fred Baker <fredbaker.ietf () gmail com>
Date: Wed, 15 Sep 2021 12:20:21 -0700



On Sep 10, 2021, at 1:33 PM, Warren Kumari <warren () kumari net> wrote:

The utility let them know that they were going to be doing some maintenance work in the area. No impact expected, but 
out of an abundance of caution, they transfer over to generators. After the utility lets them know that the 
maintenance work is all finished, they want to switch back. If the generators are "emergency power", and you need to 
switch back to "utility power", obviously the way to do this must be the big red button, clearly marked as "EMERGENCY 
POWER OFF", no?!

One of the many stories that came out of 9/11 was a switching center in NY City that had a diesel generator as a power 
backup - which of course acted as primary when the city power is off. After a few days of operation, it needed to be 
refueled, so a truck was sent in carrying gasoline. The generator was refueled and restarted, and - oops - diesel != 
gasoline. So then they needed to bring in a new generator.

Yup, it happens, and it happened.

Current thread: