NJ impact

[Alex Rubenstein <alex () corp nac net>]

I had to summarize this recently for a news article I was interviewed for, so I figured I forward:


--

Of our three datacenters, this is what we saw:

Parsippany 1 (OCT) - The worst we saw here was several sub-second power hits. UPS's held without problem, and we did 
not transfer to generator at all yet.

Parsippany 2 (WBR) - Transferred to generator at about 7:55 PM EST Monday as a precautionary measure due to ongoing 
utility power hits. However, shortly after transfer, utility voltage went to 0 on all phases; around 10p power 
returned, but abnormally high (seeing about 550 volts on 480 volt bus). We retransferred last night as utility voltage 
settled down.

Cedar Knolls 1 (MMU) - Briefly transferred to generator around 7:10, then back to utility. We then force transferred to 
generator around 8pm and stayed until this morning. Returned to utility and all systems are normal.

We have accommodated many customers (and non-customers) by allowing them to use our conference rooms and some offices, 
mainly because they lost power at their buildings, lost access to their buildings, or generators ran out of fuel. We 
literally have card tables setup in hallways and common areas where people sit and use Wifi to conduct business.

There has been no service interruption of any sort to any of our datacenter customers, as of this writing.

We lost one span of fiber optics between MMU and 165 Halsey, Newark (NWR), network healed in less than 1 second. It was 
imperceptible. That was early though, Monday afternoon. This has since restored today.

We have sales people onsite who are actively taking sales calls, and there is a measurable amount of customers moving 
equipment into our facilities as I write this. Many are moving equipment from offices with no power and no ETR in 
sight. We have electricians onsite to help accommodate.

The facilities department has been manning all locations around the clock.

All sites remain accessible by road. There was never a situation where you could not over the last 48 hours.

We have no equipment in 111 8th, so I cannot offer any telemetry there. However, our gear in 60 Hudson St (Tel-X 9th 
floor) is still online.

We saw no interruption at 165 Halsey, except for some very brief utility hits. UPS's held.

We are seeing hundreds, if not thousands, of DSL / T1 / T3 / Ethernet customers down - presumably mostly power loss on 
the CPE. We lost our connection to Covad, presumably they have gear in a building which lost power. These are only just 
starting to come back.

We have seen substantial interruption to the cellular networks; in many cases, ATT Wireless (who NAC uses) is unable to 
complete a call. In certain areas, there is no signal at all.

Where I live, in 07874 (Byram), many roads are impassable and most everyone does not have power. Trees are down 
everywhere. I heard that a Byram squad car had a tree land on it while responding to a call. Schools are closed, 
Halloween has been "rescheduled" according to our governor. My cable is down, cable modem is down, POTS service is out 
- yet I still have xDSL (loop by Verizon, IP by NAC) and I run on that successfully. I am unable to get any ATT cell 
service at my house (usually at least three bars). I am on generator at my house, consuming about 2 gal/hour (propane), 
with a little over 500 gallons left.

Gasonline is becoming an issue. As I understand, having talked to our oil supplier, there is 0 refinery capacity in NJ 
at this time. All gasoline is being imported from PA, DE, MD, etc. Most stations don't have power. Ones that do are 
generally out of gas. If they have gas, they have mile long lines and are rationing.

Diesel is unavailable if you had not pre-arranged for it.

Widespread power outages remain. I am running on generator at my house (Generac QT048) on a 1000 gallon tank of 
propane, have been since Monday about 7pm. It is estimated at least 7 to 10 days before power restoration. I have yet 
to see one JCPL truck anywhere near where I live.

That is all I have.

This is the road leading to the Byram Police Dept - I took this yesterday morning:
https://sphotos-a.xx.fbcdn.net/hphotos-prn1/12696_4311181250657_900557870_n.jpg

This is typically what Northern NJ looks like:

https://sphotos-a.xx.fbcdn.net/hphotos-ash3/548544_4311185050752_1011136048_n.jpg

https://sphotos-b.xx.fbcdn.net/hphotos-ash3/61115_4317167440308_289655722_n.jpg

(not the top of the telephone pole hanging from the wires)