FC: NYC network problems after 9-11, and how providers are responding

[Declan McCullagh <declan () well com>]

[This is a heartening thread from the nanog network operators list. Kudos 
to those firms like XO Europe who are taking extra steps to help their 
colleagues keep the Internet, especially New York City's portions, running 
during this difficult time. Without the Net, it would be more difficult for 
friends and family to keep in touch with their loved ones in the city. And, 
as Sean Donelan points out, sites like nyc.gov (now, alas, offline) are 
being used to provide vital information about emergency response and 
shelters. --Declan]

***********

From: "Vincent J. Bono" <vbono () vinny org>
Subject: Re: 32 Old Slip AC Outage and 60 Hudson Verizon failure
Date: Wed, 12 Sep 2001 09:28:24 -0400

Status:

We are at 32 Old Slip on the 4th floor.  AC power is down and our generator
air intake is clogged and the sensors shut it down.  Batteries are fine (we
have 27 hours left) but the HVAC is on generator only.  Since we are south
of 14th street we are awaiting permission to enter the area to effect
repairs.  Luckily our staff members who were outside during the collapse are
fine which was our main concern.

Sincerely,
GLOBAL NAPS, INC.

Vincent J. Bono
Vice President
Engineering

***********

From: "Vincent J. Bono" <vbono () vinny org>
Subject: Re: 32 Old Slip AC Outage and 60 Hudson Verizon failure
Date: Wed, 12 Sep 2001 12:58:18 -0400

We have a man back at 32 Old Slip now.  Most of our issues came from
overheat.  The generators are now up but utility power is still out.  There
is emergency lighting only.  The generator ran our of fuel once and has been
refilled from the building reserve but we are unsure about getting
addiitonal fuel at this time.

Sincerely,
GLOBAL NAPS, INC.

Vincent J. Bono
Vice President
Engineering

PS

FOCAL is on the 4th floor with us as well as TCG (AT&T Local).  AT&T seems
to have an equipment failure.

-vb

***********

From: robert () tellurian com
Subject: 32 Old Slip AC Outage and 60 Hudson Verizon failure
Date: Tue, 11 Sep 2001 22:36:18 -0400

Our equipment at 32 Old Slip is not operational. This is apparently
due to AC power failure and an inability to start the generators due
to ash in the air (this may be a choice as well). Anyone running at
32 Old Slip (TCG/Eagle/Focal/GNAPs are the four I know of on the
fourth floor) on DC power is on battery or offline.

Our POP at 60 Hudson is up with the MFS/WorldCom provided backhaul
circuit to our 34 Exchnage Place Jersey City location, but all
Verizon circuits terminating at 60 Hudson are down/down. All MFS/WCom
circuits seem to be ok so far...

-Robert

***********

From: Sean Donelan <sean () donelan com>
Subject: NYC city net 161.185.0.0 off net
Date: 11 Sep 2001 20:21:54 -0700

New York City's primary network 161.185.0.0/16 including the
www.nyc.gov web site is off the net this evening.  It was up
all day, and providing public information about emergency
response, shelters and transit status.  I've tried from
two different network providers.

Applied Theory, NYC's primary name server, is up.

***********

From: Alex Bligh <alex.bligh () uk xo com>
Subject: [nom-steer] Disaster in New York; offer of assistance
Date: Wed, 12 Sep 2001 09:43:00 +0100

No doubt everybody reading this will alread be aware of the disastrous
events unfold in New York and WDC, and untold human cost.

As we try and focus back on running networks, it has become evident that
much of downtown NY is currently without power, and many telco hotels are
running on backup generators. Some are not being started, or cannot start,
due to the ash in the air, and some may stop. At least three colo
facilities (WTC obviously, 7 WTC next door, Old slip) are down. 25 Broadway
(at least) is running on generator power. It is unknown at this stage how
much diesel existing facilities have. At least one C/O is gone, and Verizon
is having extensive problems in the downtown area.

Most transatlantic circuits land in Manhatten, and most are routed through
down town area. Though we didn't see much circuit disruption yesterday
(apparently UUnet lost some transatlantic capacity - no doubt they can
provide more details), I expect more disruption as generators run dry.

This is likely to be a difficult week for transatlantic operators. In the
spirit of cooperation in adverse circumstances, XO Europe would like to
offer free transit to any European ISP with a presence in the UK, in case
of any emergency outage caused by the NY / WDC events (i.e. total lack of
transit or severe degredation through). We can bring this up quickly in
either Telehouse or Harbour Exchange. Obviously we will need to capacity
plan this. Please email or ring me first. Don't just point default.
XchangePoint has similarly offered to put in private interconnects at
minimal notice if required under similar circuimstances (please contact
Brett Wilde there). I would encourage other large ISPs with transatlantic
capacity spare to join us in this offer.

Please feel free to redistribute this message intact.

Alex Bligh
Senior Technology Officer, Europe
XO Communications - http://www.xo.com/
(formerly Nextlink Inc, Concentric Network Corporation
GX Networks, Xara Networks)

***********

From: dennis opacki <dopacki () adotout com>
Subject: Reducing power consumption in generator-supplied facilities
Date: Wed, 12 Sep 2001 07:31:21 -0700 (PDT)

        What steps are folks taking to maximize run time of emergency
generator equipment and battery in NY facilities currently facing diesel
supply issues? Are second halves of redundant systems being temporarily
taken off line and non-essential loads being shed in a effort to stretch
the time to failure? In a previous life, such precutions were referred to
as "rigging for reduced electrical power" and proved quite effective.

Thanks,

-Dennis

***********

From: "Vincent J. Bono" <vbono () vinny org>
Subject: Re: Reducing power consumption in generator-supplied facilities
To: "dennis opacki" <dopacki () adotout com>, <nanog () merit edu>
Date: Wed, 12 Sep 2001 10:47:23 -0400

Hundreds of generators are not even running thanks to debris in the intake
paths.  At least in the area as close as our building was.  I shudder to
think of what's happening in the way of Carbon Monoxide gas release and
general overheating issues for those units that do not have smart sensors on
them.

-vb

***********

Return-Path: <jlapolit () drama navipath com>
Subject: Re: World Trade Center attack
Date: Tue, 11 Sep 2001 22:07:50 -0400 (EDT)

Avi can you post this to nanog for me, my post access is gone:

We have started to lose equipment at 32 Old Slip.  The site is not
currently off the air, but it is getting hot and several boxes have
shut off due to heat.  The site is currently (and understandably)
unmanned, so we cannot get a status on the site from GNAPS our
provider.

We are a dial provider and have started to see a drastic reduction in
inbound calls.  It's unclear what is causing this at this point.

-Joe

***********

From: "Stephen J. Wilcox" <steve () opaltelecom co uk>
Subject: Re: Horrible world trade center crash, traffic effect (fwd)
Date: Wed, 12 Sep 2001 13:57:29 +0100 (BST)

So let see, this concerns North America, theres operators
involved.. sounds nanogish. Just to double check the relevance heres a
snippet from Telehouse

The commercial power supply to 25 Broadway was lost at about 16:35 EDT
today. We are continuing to support those loads on generator.  We are
conservatively stating that we have sufficient fuel for 24 hours or more,
but actual fuel usage seems to indicate that could possibly continue for
as long as 70 hours.  Unfortunately, if the commercial power outage
continues for a longer period, we may not be able to replenish the fuel
given the state of the streets, etc. in the area.  We'll watch this
situation closely

So yup, theres a good chance everyone will be affected sooner or later.

Does that qualify as relevant!?

FYI, good stories, pictures and video on www.sky.com

Steve

***********




-------------------------------------------------------------------------
POLITECH -- Declan McCullagh's politics and technology mailing list
You may redistribute this message freely if you include this notice.
Declan McCullagh's photographs are at http://www.mccullagh.org/
To subscribe to Politech: http://www.politechbot.com/info/subscribe.html
This message is archived at http://www.politechbot.com/
-------------------------------------------------------------------------