Re: San Francisco Power Outage

[George William Herbert <gherbert () retro com>]

Michael Dillon writes:
> > And the stories that the power guy I'm working with tells 
> > about foreign facilities, particularly in middle east war 
> > zones, are really scary...
>
> > We fundamentally do not have the facilities problem 
> > completely nailed down to the point that things will never 
> > drop.  Level 4
> > datacenters can, and will, fail.   Nothing you can do including
> > just doing 48V DC for everything are truly foolproof solutions.
>
> A single level 4 datacenter is a Single Point of Failure!
>
> Two of those middle-eastern style facilities is... ?
> Has anyone actually kept track of all these data center failures over
> the years and done some statistical analysis on it? Maybe two half-baked
> data centers is better than one over the long run?
>
> Remember that one 10-12 years ago in (Palo Alto, Mountainview?) where a
> lady in a car caused a backhoe driver to move out of the way which
> resulted in him cutting a gas line which resulted in the fire department
> evacuating the data center, cutting off electricity in the area, and
> forbidding the diesel generators to be switched on? 

Santa Clara.

I was working right outside the evacuation radius.

Which exchange point was in the building?  PB-NAP?  CIX?  I remember
we had a net-dark event associated, but not which one.

It was a bad day...

The lesson, as you point out, is that geographical redundancy
is sometimes necessary.  This is as true for providers as for
datacenter end-users...  



-george william herbert
gherbert () retro com