nanog mailing list archives
Re: Never push the Big Red Button (New York City subway failure)
From: Christopher Morrow <morrowc.lists () gmail com>
Date: Fri, 10 Sep 2021 14:48:05 -0400
On Fri, Sep 10, 2021 at 1:49 PM Matthew Huff <mhuff () ox com> wrote:
Reminds me of something that happened about 25 years ago when an elementary school visited our data center of the insurance company where I worked. One of our operators strategically positioned himself between the kids and the mainframe, leaned back and hit it's EPO button.
Or when your building engineering team cuts themselves a new key for the 'main breaker' for the facility... and tests it at 2pm on a tuesday. Or when that same team cuts a second key (gotta have 2 keys!) and tests that key on the same 'main breaker' ... at 2pm on the following tuesday. <quadruple face palm> not fakenews, a real story from a large building full of gov't employees and computers and all manner of 'critical infrastructure' for the agency occupying said building. Matthew Huff | Director of Technical Operations | OTA Management LLC
Office: 914-460-4039 mhuff () ox com | www.ox.com ........................................................................................................................................... -----Original Message----- From: NANOG <nanog-bounces+mhuff=ox.com () nanog org> On Behalf Of Sean Donelan Sent: Friday, September 10, 2021 12:38 PM To: nanog () nanog org Subject: Never push the Big Red Button (New York City subway failure) NEW YORK CITY TRANSIT RAIL CONTROL CENTER POWER OUTAGE ISSUE ON AUGUST 29, 2021 Key Findings September 8, 2021 https://www.governor.ny.gov/sites/default/files/2021-09/WSP_Key_Findings_Summary-for_release.pdf Key Findings [...] 3. Based on the electrical equipment log readings and the manufacturer’s official assessment, it was determined that the most likely cause of RCC shutdown was the “Emergency Power Off” button being manually activated. Secondary Findings 1. The “Emergency Power Off” button did not have a protective cover at the time of the shutdown or the following WSP investigation. [...] Mitigation Steps 1. Set up the electrical equipment Control and Communication systems properly to stay active so that personnel can monitor RCC electrical system operations. [...]
Current thread:
- Never push the Big Red Button (New York City subway failure) Sean Donelan (Sep 10)
- RE: Never push the Big Red Button (New York City subway failure) Matthew Huff (Sep 10)
- Re: Never push the Big Red Button (New York City subway failure) Christopher Morrow (Sep 10)
- Re: Never push the Big Red Button (New York City subway failure) Chris Kane (Sep 10)
- RE: Never push the Big Red Button (New York City subway failure) Matthew Huff (Sep 10)
- Re: Never push the Big Red Button (New York City subway failure) Baldur Norddahl (Sep 10)
- Re: Never push the Big Red Button (New York City subway failure) Warren Kumari (Sep 10)
- Re: Never push the Big Red Button (New York City subway failure) Tom Beecher (Sep 15)
- Re: Never push the Big Red Button (New York City subway failure) Fred Baker (Sep 15)
- Re: Never push the Big Red Button (New York City subway failure) Warren Kumari (Sep 15)
- Re: Never push the Big Red Button (New York City subway failure) William Herrin (Sep 15)
- Re: Never push the Big Red Button Stephen Satchell (Sep 15)
- Re: Never push the Big Red Button Keith Stokes (Sep 15)
- Re: Never push the Big Red Button (New York City subway failure) Christopher Morrow (Sep 10)
- RE: Never push the Big Red Button (New York City subway failure) Matthew Huff (Sep 10)