nanog mailing list archives
Re: Fiber cut in SF area
From: "Patrick W. Gilmore" <patrick () ianai net>
Date: Fri, 10 Apr 2009 16:19:25 -0400
On Apr 10, 2009, at 3:41 PM, Scott Doty wrote:
George William Herbert wrote:Scott Doty wrote:(Personally, I can think of a "MAE-Clueless" episode that was worse than this, but that was in the 90's...)The gas main strike out front of the building in Santa Clara? Or something else? -george william herbert gherbert () retro com
No, it was when an AS took their full bgp feed & fed it into their igp (which used RIP, iirc), which generated (de-aggregated) routes into /24's, which they then announced back into bgp...
That was Vinny Bono of FLIX, the Fat man Little man Internet eXchange, as7007. Happened in 1997, IIRC. He used a Bay Networks router to redistribute BGP on one card into RIPv1 on another card, stripping the CIDR notations off each prefix, making them classful, and stripping the AS Path. This means, for instance, 96.0.0.0 was a /8, not a /24. It also means He then re-redistributed RIP into BGP on a third card, which then originated each route from as7007.
I have it on most excellent authority (the "Fat man" himself) that this was not possible on ciscos. Wonder if it is now ... ?
Anyway, I did not know people were calling this the "MAE-Clueless" incident. I've always called it the "7007 incident". In fact, some people still have as7007 filtered.
iirc, part of the chaos than ensued was due to a router bug, so that the routes "stuck around" in global views, even after the AS killed their announcements, and even after physically disconnecting from their provider.
That was Sprint, as7007's transit provider. Sprint only did AS Path filtering, and as every single prefix was ^7007$, they all passed the filter.
Vinny literally unplugged the router, no power, no fiber, no copper, but the prefixes were still bouncing around the 'Net for hours. Sprint kept the routes around for a long time as their routers would not honor withdrawals - or so the rumors said. The rumors also claimed the IOS version was named "$FOO-sean". Sean Doran was CTO of Sprint's Internet company at the time, and he supposedly specifically asked for the 'feature' of ignoring withdrawals to lower CPU on their AGS+s. I have absolutely no way of confirming this as I haven't spoken to Sean in years & years, and wouldn't even know where to find him any more.
The most interesting rumor I heard is that Sprint had to shut down every single router simultaneously to clear the routes out of their network. Personally I think that's probably a bit exaggerated, but who knows?
We told our customers "the Internet is broken, please try again later"...which was acceptable back then. (But I doubt we would get away with just that nowadays... ;-) )
Really? That's what some broadband providers say nearly daily. -- TTFN, patrick
Current thread:
- Re: Fiber cut in SF area, (continued)
- Re: Fiber cut in SF area David Edwards (Apr 09)
- Message not available
- Re: Fiber cut in SF area Mike Lyon (Apr 09)
- Message not available
- Re: Fiber cut in SF area Joel Jaeggli (Apr 09)
- Message not available
- Re: Fiber cut in SF area Scott Doty (Apr 09)
- Re: Fiber cut in SF area George William Herbert (Apr 09)
- Re: Fiber cut in SF area Scott Doty (Apr 10)
- BGP FlowSpec support on provider networks Fouant, Stefan (Apr 10)
- Re: BGP FlowSpec support on provider networks Seth Mattinen (Apr 10)
- Re: BGP FlowSpec support on provider networks Charles Wyble (Apr 10)
- Re: BGP FlowSpec support on provider networks McDonald Richards (Apr 10)
- Re: Fiber cut in SF area Patrick W. Gilmore (Apr 10)
- RE: Fiber cut in SF area Jo¢ (Apr 10)
- Re: Fiber cut in SF area Joel Jaeggli (Apr 10)
- Re: Fiber cut in SF area Joe Greco (Apr 11)
- Re: Fiber cut in SF area Florian Weimer (Apr 11)
- Re: Fiber cut in SF area Christopher Morrow (Apr 11)
- Re: Fiber cut in SF area Jorge Amodio (Apr 11)
- Re: Fiber cut in SF area Peter Beckman (Apr 12)
- [OT] Re: Fiber cut in SF area Lamar Owen (Apr 11)
- Re: [OT] Re: Fiber cut in SF area Joe Greco (Apr 11)
- Re: [OT] Re: Fiber cut in SF area Christopher Morrow (Apr 11)