nanog mailing list archives
Followup: anyone else seeing very long AS paths?
From: "Ivan Pepelnjak" <ip () ioshints info>
Date: Fri, 20 Feb 2009 21:55:35 +0100
I know this is getting boring, but I don't want to see information lying around hinting that the ISPs around the world were to blame for the Monday incident due to their sloppy software upgrade policies. That's not the case; a lot of very recent IOS releases were affected. There was lots of conflicting information flowing around. Rodney did an excellent job with the bug description, but I was still wondering what exactly was going on, so I've tested all potential combinations of inbound/outbound IBGP/EBGP prepending/no-prepending cases to identify exactly what can happen; you should be able to figure out which one affected your network: http://blog.ioshints.info/2009/02/oversized-as-paths-cisco-ios-bug.html And it should be reiterated that anyone that has already configured bgp maxas-limit has avoided this problem. Ivan
-----Original Message----- From: Rodney Dunn [mailto:rodunn () cisco com] Sent: Thursday, February 19, 2009 9:15 PM To: Rodney Dunn Cc: German Martinez; Ivan Pepelnjak; nanog () nanog org Subject: Re: anyone else seeing very long AS paths? We are working on a document for Cisco.com but in the interim here is the bug that will fix the issue of a Cisco IOS device sending an incorrectly formatted BGP update when as a result of prepending it goes over 255 AS hops. Note: The Title and Release-note on bug toolkit may be a bit different as I just updated it to be more accurate. Of all the scenarios I've looked at (thanks to those that responded offline) there wasn't a condition found where this could happen without AS path prepending being used. Please respond offline or let's move the discussion over to cisco-nsp at this point. CSCsx73770 Invalid BGP formatted update causes peer reset with AS prepending <B>Symptom:</B> A Cisco IOS device that receives a BGP update message and as a result of AS prepending needs to send an update downstream that would have over 255 AS hops will send an invalid formatted update. This update when received by a downstream BGP speaker triggers a NOTIFICATION back to the sender which results in the BGP session being reset. <B>Conditions:</B> This problem is seen when a Cisco IOS device receives a BGP update and due to a combination of either inbound, outbound, or both AS prepending it needs to send an update downstream that has more than 255 AS hops. <B>Workaround:</B> The workaround is to implement <CmdBold> bgp maxas-limit X <NoCmdBold> on the device that after prepending would need to send an update with over 255 AS hops. Since IOS limits the inbound prepending value to 10 the most that could be added iss 11 AS hops (10 on ingress, 10 on egress, and 1 for normal eBGP AS hop addition). Therefore, a conservative value to configure would be 200 to prevent this condition. Full support of Section 5.1.2 of RFC4271 is being tracked under CSCsx75937 Add BGP support of AS paths longer than 255 per Section 5.1.2 of RFC4271 Thanks to those that worked offline with us to verify the field results reported. Rodney On Tue, Feb 17, 2009 at 05:27:01PM -0500, Rodney Dunn wrote:If you want to take this offline send it unicast or wecould move itto cisco-nsp. What scenarios are you seeing that appear broken other than when a notification is sent when a > 255 hop update is received? That's the one I'm working on right now. Rodney On Tue, Feb 17, 2009 at 05:31:49PM -0500, German Martinez wrote:On Tue Feb 17, 2009, Rodney Dunn wrote: Hello Rodney, It will be great if you can share with us your findings.It seemslike we are hitting different bugs in different platforms. Thanks GermanIvan, It is confusing but from what I have tested you have it correct. The confusing part comes from multiple issues. a) The documentation about the default maxas limitbeing 75 appears to beincorrect. I'll get that fixed. b) Prior to CSCee30718 there was a hard limit of 255.After that fixAS sets of more than 255 should work. c) CSCeh13489 implemented the maxas command to mark itas invalid andnot send. There does appear to be an issue when you cross the 255boundaryand the next hop router sends a notification back. I've got it recreated in the lab and we are working to clearly understand why that is. I'll post an update once we have more. The way to prevent it is the upstream device thatcrosses the 255boundary on sending needs to use the maxas limitcommand to keep it less than 255.It doesn't work on the device that receives the updatewith the ASpath larger than 255. Rodney On Tue, Feb 17, 2009 at 08:58:48PM +0100, Ivan Pepelnjak wrote:We were dropping ALL prefixes and the eBGP sessionwas stillresetting.Upstream or downstream?1) "bgp maxas-limit 75" had no effect mitigatingthis problemon the IOS we were using. That is: it waspreviously verifiedto be working just fine to drop paths longer than75, but oncewe started receiving paths > 255 then BGP started resetting.I was able to receive BGP paths longer than 255 onIOS release12.2SRC. The paths were generated by Quagga BGP daemon. 12.2SRC causes the downstream session to break when the installed AS-path length is close to 255 and you usedownstream AS-path prepending.In your case, I'm assuming you were hit with an older bug (probably at the 128 AS-path length boundary). It would be very hardto generatejust the right AS-path length to unintentionally break your upstream EBGP session (as I said before, it's a nice targeted attack if you know your downstream topology). If your IOS is vulnerable to the older bugs thatbreak inboundprocessing of AS paths longer than 128, there'snothing you cando on your end. The internal BGP checks reject the inbound update before the inbound filters (or bgpmaxas-limit) can touch it and reset the upstream BGP session.Hope this helps Ivan Disclaimer: as I don't have internal access to Cisco,all of theabove is a result of lab tests.
Current thread:
- Re: anyone else seeing very long AS paths?, (continued)
- Re: anyone else seeing very long AS paths? Leland E. Vandervort (Feb 17)
- RE: anyone else seeing very long AS paths? Ivan Pepelnjak (Feb 17)
- Re: anyone else seeing very long AS paths? Jack Bates (Feb 17)
- Re: anyone else seeing very long AS paths? Mike Lewinski (Feb 17)
- RE: anyone else seeing very long AS paths? Ivan Pepelnjak (Feb 17)
- Re: anyone else seeing very long AS paths? Rodney Dunn (Feb 17)
- Re: anyone else seeing very long AS paths? German Martinez (Feb 17)
- Re: anyone else seeing very long AS paths? Rodney Dunn (Feb 17)
- Re: anyone else seeing very long AS paths? Rodney Dunn (Feb 19)
- Re: anyone else seeing very long AS paths? Rodney Dunn (Feb 20)
- Followup: anyone else seeing very long AS paths? Ivan Pepelnjak (Feb 20)
- RE: anyone else seeing very long AS paths? Ivan Pepelnjak (Feb 17)
- Re: anyone else seeing very long AS paths? Steven Saner (Feb 17)
- Re: anyone else seeing very long AS paths? Jack Bates (Feb 17)
- Re: anyone else seeing very long AS paths? German Martinez (Feb 17)
- Re: anyone else seeing very long AS paths? Hank Nussbacher (Feb 17)