Re: Famous operational issues

[Warren Kumari <warren () kumari net>]

On Mon, Feb 22, 2021 at 12:50 PM Regis M. Donovan <regis-nanog () offhand org>
wrote:

> On Thu, Feb 18, 2021 at 07:34:39PM -0500, Patrick W. Gilmore wrote:
> > And to put it on topic, cover your EPOs
>
> I worked somewhere with an uncovered EPO, which was okay until we had a
> telco tech in who was used to a different data center where a similar
> looking button controlled the door access, so he reflexively hit it
> on his way out to unlock the door.  Oops.
>
> Also, consider what's on generator and what's not.  I worked in a corporate
> data center where we lost power.  The backup system kept all the machines
> running, but the ventilation system was still down, so it was very warm
> very
> fast as everyone went around trying to shut servers down gracefully while
> other folks propped the doors open to get some cooler air in.

That reminds me of another one...

In parts of NYC, there are noise abatement requirements, and so many places
have their generators mounted on the roof -- it's cheap real-estate, the
exhaust is easier, the noise issues are less, etc.

The generators usually have a smallish diesel tank, and then a much larger
one in the basement (diesel is heavy)...

So, one of the buildings that I was in was really good about testing thier
gensets - they'd do weekly tests (usually at night), and the generators
always worked perfectly -- right up until the time that it was actually
needed.
The generator fired up, the lights kept blinking, the disks kept spinning -
but the transfer pump that pumped diesel from the basement to the roof was
one of the few things that was not on the generator....

W



> --r

-- 
The computing scientist’s main challenge is not to get confused by the
complexities of his own making.
  -- E. W. Dijkstra