Re: OVH datacenter SBG2 in Strasbourg on fire 🔥

[Rod Beck <rod.beck () unitedcablecompany com>]

It is amazing that they can burn so well. Would never have imagined it.

Regards,

Roderick.

________________________________
From: NANOG <nanog-bounces+rod.beck=unitedcablecompany.com () nanog org> on behalf of George Herbert <george.herbert () 
gmail com>
Sent: Thursday, March 11, 2021 11:37 AM
To: Andy Ringsmuth <andy () andyring com>
Cc: nanog () nanog org list <nanog () nanog org>
Subject: Re: OVH datacenter SBG2 in Strasbourg on fire 🔥

Sent from my iPhone

On Mar 10, 2021, at 7:45 AM, Andy Ringsmuth <andy () andyring com<mailto:andy () andyring com>> wrote:

Sad to see of course, but also a little surprising that fire suppression systems didn’t, well, suppress the fire.

Unless they didn’t exist?


I am assuming you haven’t had a real datacenter fire before.

I’ve had one fire, seen another, and had an accidental system firing of the gas system.

In the actual fire, caused by cooling system partial failure, there was no gas and it turned out the system to disable 
power on sprinkler discharge failed, then the power circuit breakers stayed live despite significant electrical short 
circuiting in the room as sprinklers fired, and then after 5 minutes the fire department arrived and sprayed water in 
for over 10 minutes without reducing arcing and fire before building power was successfully disabled and they could put 
the rest of the fire out.  This one could easily have destroyed its building, also the building Palo Alto Frys was in.

Fun fact: a rack of burning servers displaces sprinkler water around the rack, if it has a top on it, and even if not a 
vertical stack of burning servers pushes water down the front and back and slightly to the sides of burning servers, 
not through the systems themselves.

Fun fact: motherboards burn, as does chip encapsulation epoxy and all the wiring, the fan frames, board standoffs, 
essentially all the RAM and PCI board slots, some capacitors and surface mount devices... hard drives melt and the 
aluminum casings burn, SSD plastic casings generally burn.  GBICs and other laser diodes smell awful after they catch 
fire, though it can be hard to tell with everything else that smells.

Fun fact: It does not necessarily take many burning servers to put the room integrity at risk, even with sprinklers 
going and the fire department spraying water in.

Fun fact: All the servers that get wet but don’t burn will rust.  And everything in the room near the sprinklers that 
are going off will get wet.  This is very sad as you watch many millions of dollars of brand new 42U racks prestuffed 
with HP enterprise servers oxidizing away while you wait for the garbage truck.

Fun fact: Combustion soot is conductive and even things that didn’t get wet probably are dead.

Fun fact: late era Sun Microsystems server boxes were very nice waxed cardboard, very well made, apparently fire 
resistant more than anyone would have thought but that were also water resistant enough that you’d have a new in box 
server submerged in a box still full to the brim with water days later.

Fun engineering advice: The window for critical data recovery from hard drives that are visibly corroding from water 
damage short of immersion is probably 48-72 hours, but run them outside any casing on a fire resistant table and have 
all of CO2 and dry chemical fire extinguishers ready... and a fire hose, and handy building fire alarm.  Corroding 
water damaged SSDs are lower power draw and somewhat less likely to start another fire, but take the same precautions.


The fire I saw but that wasn’t mine burned a building with halon and pre action sprinklers more or less to the steel 
columns and roof beams, despite the fire department arriving in 5 minutes and trying to actively suppress.  Not enough 
openings for them to safely get to the fire before it was out of control, and not enough water flow available in all 
the sprinklers once it took off.

That one had evidently burned aluminum rack posts by the time it was over, not just melted them...


Systems and datacenters aren’t built to eliminate fire risk.  They just aren’t.  You can contain or control most office 
fires with sprinklers, and certainly evacuate.  Datacenters with emergency power batteries in the envelope often have 
enough stored energy to set the room on fire despite sprinklers.  If AC cutoffs fail and circuit breakers don’t trip 
mains power will as well.  Too many systems and storage and networking hardware components can burn.


-george