nanog mailing list archives

Re: Mitigating human error in the SP


From: Larry Sheldon <LarrySheldon () cox net>
Date: Thu, 04 Feb 2010 17:13:30 -0600

On 2/4/2010 3:30 PM, Scott Weeks wrote:

A recent organizational change at my company has put someone in charge
who is determined to make things perfect.  We are a service provider,

isn't a common occurrence, and the engineer in question has a pristine
track record.

This outage, of a high profile customer, triggered upper management to
react by calling a meeting just days after.  Put bluntly, we've been
told "Human errors are unacceptable, and they will be completely
eliminated.  One is too many."
----------------------------------------------------------------



From experience...

At one point this will become overwhelming.  You'll wake up every morning dreading going to
work instead of looking forward to it.  Chain shot will be put in the 'blame cannon' and
blasted regularly and at everyone.  Update your resume and get everything in place just in
case it gets to the point you can't take it anymore sooner than you expect.  ;-)


This is a golden opportunity.

Prepare a pan for building the lab necessary to pre-test EVERYTHING.

Cost it out.

Present the cost and the plan in a public forum or widely distributed memorandum (including as a minimum everybody that was at the meeting and everybody in the chain(s) of command between you and the edict giver.


--
"Government big enough to supply everything you need is big enough to take everything you have."

Remember:  The Ark was built by amateurs, the Titanic by professionals.

Requiescas in pace o email
Ex turpi causa non oritur actio
Eppure si rinfresca

ICBM Targeting Information: http://tinyurl.com/4sqczs http://tinyurl.com/7tp8ml
        


Current thread: