nanog mailing list archives

Never push the Big Red Button (New York City subway failure)


From: Sean Donelan <sean () donelan com>
Date: Fri, 10 Sep 2021 12:37:54 -0400 (EDT)

NEW YORK CITY TRANSIT RAIL CONTROL CENTER POWER
OUTAGE ISSUE ON AUGUST 29, 2021
Key Findings
September 8, 2021


https://www.governor.ny.gov/sites/default/files/2021-09/WSP_Key_Findings_Summary-for_release.pdf

Key Findings
[...]

3. Based on the electrical equipment log readings and the manufacturer’s official assessment, it was determined that the most likely cause of RCC shutdown was the “Emergency Power Off” button being manually activated.

Secondary Findings

1. The “Emergency Power Off” button did not have a protective cover at the time of the shutdown or the following WSP investigation.

[...]
Mitigation Steps

1. Set up the electrical equipment Control and Communication systems properly to stay active so that personnel can monitor RCC electrical system operations.

[...]


Current thread: