Re: L3 East cost maint / fiber 05FEB2012 maintenance

[Jason Lixfeld <jason () lixfeld ca>]

I got notification of their maintenance window, albeit with < 24 hours notice.  Notice came in at 11:00GMT-5 yesterday, 
maintenance was scheduled for 00:00GMT-5 this morning.

That said, the notice said that the maintenance was in Phoenix but I got a notice about my IPT circuit at 60 Hudson 
which I found confusing.

Based on my logs, our BGP session with them went down at 03:06GMT-5 and back up at 03:15GMT-5.  Down again at 
03:37GMT-5 until 04:20GMT-5.  A third time at 06:41GMT-5 and back at 06:45GMT-5.

Traffic graphs tell a bit of a different story.  Just before 05:00GMT-5, our outbound traffic to Level 3 dropped 
substantially.  About that time, I started getting reports about issues to Level 3 destinations.  Traces seemed to 
indicate a black hole condition within Level 3's network in NYC, seemingly at, or just past csw3.NewYork1.Level3.net.  
Stuff seemed to correct itself by about 06:45GMT-5, but due to Level 3 sending only about 180k routes.  About 20 
minutes later, the table was back to ~431K and all's been fine since.

On 2013-02-05, at 10:39 AM, Josh Reynolds <esseph () gmail com> wrote:

> I know a lot of you are out of the office right now, but does anybody have
> any info on what happened with L3 this morning? They went into a 5 hour
> maintenance window with expected downtime of about 30 minutes while they
> upgraded something like *40* of their "core routers" (their words), but
> also did this during some fiber work and completely cut off several of
> their east coast peers for the entirety of the 5 hour window.
>
> If anybody has any more info on this, on a NOC contact for them on the East
> Coast for future issues, you can hit me off off-list if you don't feel
> comfortable replying with that info here.
>
> Thanks, and I hope hope you guys are enjoying Orlando.
>
> -- 
> *Josh Reynolds*
> esseph () gmail com - (270) 302-3552