nanog mailing list archives

Re: DHS letters for fuel and facility access


From: Alexandre Petrescu <alexandre.petrescu () gmail com>
Date: Tue, 17 Mar 2020 18:35:52 +0100

At my work place there is enough generators, fuel generators.

There is enough time to power things down properly.

The IT infra seems to be working ok, although some remote workers complain about a few things about VPN.

There is however worry that the IT infra might not keep up, or that not all employees might have access to emails.  To address that, they have built a website facing to the Internet with internal announcement info to employees. They have also created a registry where the employees record their external email addresses so we receive internal announcements but on external email addresses, a thing which was more or less prohibited in normal times by IT policy.

The internal emergency phone number (two digit phone number only available to internals only by landline) has just been shut down.  The info circulated announcing it so.  IT is standard procedure in case of issues.

My desk voicemail is still active and I can consult it remotely, but not sure for how long.  The re-start of desk power typically resets the phone and I lose voicemail forever.  I expect that re-start of desk power in a few weeks or so, part of standard procedure to re-start power routinely.  But I dont expect me to go to my desk any time since now in one month to press the button on the phone to set the voicemail active.

Alex

Le 17/03/2020 à 18:21, Hiers, David a écrit :
Good reminder to test, test, test...


-----Original Message-----
From: NANOG [mailto:nanog-bounces () nanog org] On Behalf Of Warren Kumari
Sent: Tuesday, March 17, 2020 10:08 AM
To: Paul Nash <paul () nashnetworks ca>
Cc: Untitled 3 <nanog () nanog org>
Subject: Re: DHS letters for fuel and facility access

On Tue, Mar 17, 2020 at 12:44 PM Paul Nash <paul () nashnetworks ca> wrote:
September 2001.  Just after the 9/11 attacks, all of lower Manhattan was shut down.  Out link (IIRC) was to a satellite 
farm on Staten island, across the bay to 60 Hudson.  Power went off, diesels kicked in, fuel trucks was not allowed in, 
and a few days later we lost all international connectivity.
We had some interesting failures during 9/11 as well -- for some reason, the UPS didn't kick in, so everything went 
down - and then came back a few minutes later as the generators came online -- and then went down again ~2 hours later -- 
turns out that the genset air filters got clogged with dust, and suffocated the diesel.
This was "fixed" a few days later by brushing them off with brooms and paintbrushes -- by this point they had 
completely discharged the 24V starter batteries, and so someone (not me!) had to lug a pair of car batteries and jumper cables. 
They restarted, and ran for a while, and then stopped again.

It turns out that getting a permit to store lots of diesel on the roof is hard (fair enough), and so there was only a 
small holding tank on the roof, and the primary tanks were in the basement -- and the transfer pump from the basement 
to roof storage was not, as we had been told, on generator power....

We had specified that the transfer pump be on the generator feed, there was a schematic showing at is being on the generator 
feed, there was even a breaker with a cable marked  "Transfer Pump (HP4,5)" --- but it turned out to just be a ~3ft 
piece of cable stuffed into a conduit, and not actually, you know, running all the way down to the basement and connected to the 
transfer pump.

W



Lots of important people lost power as well, so the feds decided to let the diesel tankers in after a few days’ 
deliberations.

         paul

On Mar 17, 2020, at 11:21 AM, Mark Tinka <mark.tinka () seacom mu> wrote:



On 17/Mar/20 17:15, Paul Nash wrote:

That same fuel shortage killed all Internet traffic to sub-Saharan Africa.  Took us a while to figure out what was 
wrong with the satellite link to the US.
What year was that :-)?

Mark.

--
I don't think the execution is relevant when it was obviously a bad idea in the first place.
This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid 
weasels and that pair of pants.
    ---maf

----------------------------------------------------------------------
This message and any attachments are intended only for the use of the addressee and may contain information that is 
privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative 
of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. 
If you have received this communication in error, notify the sender immediately by return email and delete the message 
and any attachments from your system.

Current thread: