nanog mailing list archives

Global Crossing Contact / BGP and SONET interaction question


From: Forrest W Christian <fwc () mt net>
Date: Tue, 25 Jul 2006 22:55:20 -0600


Two somewhat intertwined questions.  I'll ask the second part first.

I buy transit from Global Crossing and another carrier on HDLC encapsulated DS3's.

Recently my BGP session has started flapping on the GX circuit... It looks something like this:

Jul 21 21:17:43.731 UTC: %BGP-3-NOTIFICATION: received from neighbor 67.17.168.73 6/6 (cease) 0 bytes Jul 21 21:17:43.731 UTC: %BGP-5-ADJCHANGE: neighbor 67.17.168.73 Down BGP Notification received
Jul 21 21:18:25.439 UTC: %BGP-5-ADJCHANGE: neighbor 67.17.168.73 Up
Jul 21 21:29:52.315 UTC: %BGP-3-NOTIFICATION: received from neighbor 67.17.168.73 6/6 (cease) 0 bytes Jul 21 21:29:52.315 UTC: %BGP-5-ADJCHANGE: neighbor 67.17.168.73 Down BGP Notification received
Jul 21 21:30:38.511 UTC: %BGP-5-ADJCHANGE: neighbor 67.17.168.73 Up
Jul 21 21:31:34.411 UTC: %BGP-3-NOTIFICATION: received from neighbor 67.17.168.73 6/6 (cease) 0 bytes Jul 21 21:31:34.411 UTC: %BGP-5-ADJCHANGE: neighbor 67.17.168.73 Down BGP Notification received
Jul 21 21:32:20.535 UTC: %BGP-5-ADJCHANGE: neighbor 67.17.168.73 Up
Jul 21 21:32:52.547 UTC: %BGP-5-ADJCHANGE: neighbor 67.17.168.73 Down Peer closed the session
Jul 21 21:33:32.703 UTC: %BGP-5-ADJCHANGE: neighbor 67.17.168.73 Up

There are no other log entries during the periods when this occur. Unfortunately this causes enough prefix flaps that any prefixes which are preferred through GX are damped for like a half hour by certain providers as my BGP routes get added/withdrawn through the GX link.

GX claims (although I'm not sure they really know) that these are caused by SONET ring switches. I can believe this, since I haven't seen any real circuit flaps, and my understanding is that a SONET switch should generally be fast enough that you normally won't see the transition other than perhaps an error counter or two cranking up. However, it seems strange that I'm getting a 6/6 (cease) notification which I read as "configuration change" from their router. GX also seems to be at a loss to explain why my BGP is flapping - other than to point at the SONET switches.

I guess I'm trying to find out if someone on the list recognizes what this might be so I can perhaps help GX find and fix this. I'm also kinda curious as to whether or not typically a SONET ring switch event would actually propagate into a router in such a way that BGP would try to shut down the BGP sessions. I'm just having a hard time visualizing how a supposedly below-layer-two switch would cause bgp to reset in this manner. Not being a SONET expert even by any long stretch of the imagination leaves me with some holes here, but I thought the whole goal of SONET when used to provide DS3 circuits was to hide the ring switches as much as possible from the DS3 circuits - realizing that framing may be hard to preserve on a ring switch which would cause momentary loss of sync or similar - which usually shows up as an error instead of a interface flap.

And finally, does anyone have a contact within GX with a clue? So far I'm not sure I've talked to anyone who knows anything but how to spell BGP. I'd really like to talk to someone about the real cause of these flaps and try to resolve them so they don't reoccur.

-forrest


Current thread: