Re: Network Operations Luminaries?

[Jeff Rizzo <riz () boogers sf ca us>]

On Mon, Dec 10, 2001 at 11:15:25AM -0800, Clayton Fiske wrote:
> On Mon, Dec 10, 2001 at 10:18:48AM -0700, Joel Baker wrote:
> > I don't have a timeline to know which happened first; 2551 was down, or at
> > least the majority of it was, for something on the order of 48-72 hours.
> > The rendition I heard assigned the moniker because one of the major news
> > outlets said that the mistake which triggered it was "like a misplaced
> > ampersand". It was certainly the one on the wall beside his legacy Chevy's
> > sombrero.
>
> My oh my, how the versions differ.
>
> As was recounted to me, the outage was about 19 hours, and was due to
> the semantics of Cisco config mode. Something like:
>
> router ospf 1234
> redistribute bgp subnets route-map blah
>
> (everything fine, now let's turn it off...)
>
> no redistribute bgp subnets route-map blah
>
> So tell me, does this turn off the redistribution, or just remove the
> route-map... :)
>
> And this is certainly worth remembering. Had I not known of this, I
> could likely have made the same mistake at some point.
>
> -c


Ye gods.   Guess I'd better keep a closer eye on NANOG.


For the record:  This version (Clay's) is more or less correct.  :) 
The "oops" was pretty much *exactly* that... removing a route-map
when intending to remove a redistribution.  

In my defense, however, we discovered the problem fairly quickly, but
OSPF bug extant in IOS 10.whatever that we were running prevented us
from cleaning it up in a timely manner.  We basically wound up
partitioning the network and rebooting *every* ospf-speaking device
on it, because they were getting poisoned with gobs of LSA data that was
never getting cleared, and was propagating the bogus information.

Also for the record:  I have *no idea* where Bob Metcalfe got that 
"ampersand" thing.    It did make for an amusing engraving on
a going-away gift I was given some years later:

!!!!!!!!!!!!!!!!!!!!&................!!!!!!!!!!!!!

:)

+j

--
Jeff Rizzo                                         http://boogers.sf.ca.us/~riz