Re: Mitigating human error in the SP

[Mark Smith <nanog () 85d5b20a518b8f6864949bd940457dc124746ddc nosense org>]

On Thu, 04 Feb 2010 17:22:23 -0600
Larry Sheldon <LarrySheldon () cox net> wrote:

> On 2/4/2010 5:13 PM, Larry Sheldon wrote:
> > On 2/4/2010 3:30 PM, Scott Weeks wrote:
> > > A recent organizational change at my company has put someone in charge
> > > who is determined to make things perfect. We are a service provider,
> > >
> > > isn't a common occurrence, and the engineer in question has a pristine
> > > track record.
> > >
> > > This outage, of a high profile customer, triggered upper management to
> > > react by calling a meeting just days after. Put bluntly, we've been
> > > told "Human errors are unacceptable, and they will be completely
> > > eliminated. One is too many."
> > > ----------------------------------------------------------------
> > >
> > >
> > >
> > > > From experience...
> > >
> > > At one point this will become overwhelming. You'll wake up every
> > > morning dreading going to
> > > work instead of looking forward to it. Chain shot will be put in the
> > > 'blame cannon' and
> > > blasted regularly and at everyone. Update your resume and get
> > > everything in place just in
> > > case it gets to the point you can't take it anymore sooner than you
> > > expect. ;-)
> >
> >
> > This is a golden opportunity.
> >
> > Prepare a pLan for building the lab necessary to pre-test EVERYTHING.
>
> Plan.  Prepare a plan.
>
> > Cost it out.
> >
> > Present the cost and the plan in a public forum or widely distributed
> > memorandum (including as a minimum everybody that was at the meeting and
> > everybody in the chain(s) of command between you and the edict giver.

Problem is, when the inevitable human error does occur, the expensive
lab then just looks like it was a huge waste of money, and that the
networking people took advantage of the situation to build a play
ground. They'll then likely be shown the door.

The only way to completely eliminate human error is to eliminate the
humans - from everything - hardware design, software design, deployment
and maintenance.

Actually, come to think of it, there is a way to eliminate human error.
Staff netops with monkeys, and as long as management don't work out
that they've now got "monkey error", they'll be happy. When they cotton
on to that, replace monkey error with goat error. Then sheep error.
Then hamster error. etc. etc. By the time they run out of types of
animal error, they'll realise how wonderful human error really is!


> -- 
> "Government big enough to supply everything you need is big enough to 
> take everything you have."
>
> Remember:  The Ark was built by amateurs, the Titanic by professionals.
>
> Requiescas in pace o email
> Ex turpi causa non oritur actio
> Eppure si rinfresca
>
> ICBM Targeting Information:  http://tinyurl.com/4sqczs 
> http://tinyurl.com/7tp8ml
>