nanog mailing list archives

Re: Famous operational issues


From: "Patrick W. Gilmore" <patrick () ianai net>
Date: Thu, 18 Feb 2021 19:34:39 -0500

On Feb 18, 2021, at 6:10 PM, Karl Auer <kauer () biplane com au> wrote:

I think it was Macchiavelli who said that one should not ascribe to
malice anything adequately explained by incompetence…

https://en.wikipedia.org/wiki/Hanlon%27s_razor
        Never attribute to malice that which is adequately explained by stupidity.

I personally prefer this version from Robert A. Heinlein:
        Never underestimate the power of human stupidity.

And to put it on topic, cover your EPOs

In 1994, there was a major earthquake near the city of Los Angeles. City hall had to be evacuated and it would take 
over a year to reinforce the building to make it habitable again. My company moved all the systems in the basement of 
city hall to a new datacenter a mile or so away. After the install, we spent more than a week coaxing their ancient 
(even for 1994) machines back online, such as a Prime Computer and an AS400 with tons of DASD. Well, tons of cabinets, 
certainly less storage than my watch has now.

I was in the DC going over something with the lady in charge when someone walked in to ask her something. She said 
“just a second”. That person took one step to the side of the door and leaned against the wall - right on an EPO which 
had no cover.

Have you ever heard an entire row of DASD spin down instantly? Or taken 40 minutes to IPL an AS400? In the middle of 
the business day? For the second most populous city in the country?

        Me: Maybe you should get a cover for that?
        Her: Good idea.

Couple weeks later, in the same DC, going over final checklist. A fedex guy walks in. (To this day, no idea how he got 
in a supposedly locked DC.) She says “just a second”, and I get a very strong deja vu feeling. He takes one step to the 
side and leans against the wall.

        Me: Did you order that EPO cover?
        Her: Nope.

-- 
TTFN,
patrick


Current thread: