RE: Time Warner outage?

["Naslund, Steve" <SNaslund () medline com>]

I don’t buy that excuse either.  If Level 3 is doing fiber maintenance on any route and takes down your entire network, 
then you have a pretty poor backbone design.  It is not Level 3s fault if you design your network such that a single 
route loss causes a huge outage.

Steven Naslund
Chicago IL


From: Chris Lane [mailto:clane1875 () gmail com]
Sent: Thursday, August 28, 2014 8:51 AM
To: Warren Bailey
Cc: Chris Garrett; Naslund, Steve; nanog () nanog org
Subject: Re: Time Warner outage?

Agreed on DHCP, just passing along something i had heard about.....   With that said, why wouldn't the TW guys just 
post something oh right they did, blame the other guy "Level3"

On Thu, Aug 28, 2014 at 9:42 AM, Warren Bailey <wbailey () satelliteintelligencegroup com<mailto:wbailey () 
satelliteintelligencegroup com>> wrote:
Sounds more likely than "we broke the dhcp server".


Sent from my T-Mobile 4G LTE Device



-------- Original message --------
From: Chris Garrett <chris () aperturefiber com<mailto:chris () aperturefiber com>>
Date: 08/28/2014 6:39 AM (GMT-08:00)
To: "Naslund, Steve" <SNaslund () medline com<mailto:SNaslund () medline com>>
Cc: nanog () nanog org<mailto:nanog () nanog org>
Subject: Re: Time Warner outage?


I have never seen BGP drop peers because of a DHCP failure.

I would love to see the breakdown on that.

TW NOC had said it was a Level 3 fiber maintenance gone bad that took down their peering connections when we called in 
at the time of the event.

Can anyone confirm one way or the other?


On Aug 28, 2014, at 8:29 AM, Naslund, Steve <SNaslund () medline com<mailto:SNaslund () medline com>> wrote:

> Something sounds really unlikely about that.  Lack of DHCP would not cause reachability problems except for the 
> clients.  The trace below looks like a transit connection that should be unaffected by DHCP.  Looks more like a 
> routing issue.  Also sounds unlikely that one DHCP server would be covering this large a scope.
>
> Steven Naslund
> Chicago IL
>
> -----Original Message-----
> From: NANOG [mailto:nanog-bounces () nanog org<mailto:nanog-bounces () nanog org>] On Behalf Of Chris Lane
> Sent: Thursday, August 28, 2014 8:23 AM
> To: Stephen Satchell
> Cc: nanog () nanog org<mailto:nanog () nanog org>
> Subject: Re: Time Warner outage?
>
> I heard the following,
>
> It was actually an engineer doing maintenance on the dhpc server that stopped customers from getting an IP address 
> when the connected to the network between 5:30 and 7:00.  The funny part is upper Managment heard about it on the 
> today show
>
>
> On Thu, Aug 28, 2014 at 8:46 AM, Stephen Satchell <list () satchell net<mailto:list () satchell net>> wrote:
>
> > This just keeps getting better and better:
> >
> > "Yahoo Logo
> > Will be right back...
> >
> > Thank you for your patience.
> >
> > Our engineers are working quickly to resolve the issue."
> >
> > My upstream is Charter Business...
> >
> > On 08/28/2014 04:05 AM, Chris Garrett wrote:
> > > I believe this is what is commonly referred to in our industry as a
> > “resume generating event”.
> > > On Aug 27, 2014, at 8:24 AM, Alexandru Suciu
> > > <alex.suciu.sum () gmail com<mailto:alex.suciu.sum () gmail com>>
> > wrote:
> > > > Can't wait to see the postmortem report on this one.
> > > >
> > > > On 8/27/2014 3:05 PM, Adam Greene wrote:
> > > > > Came back for us, too, at about 7am.
> > > > >
> > > > > Outage was a biggie this time:
> > https://news.yahoo.com/time-warner-cable-suffering-massive-nationwide-
> > outage
> > > > > -110549606.html
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: NANOG [mailto:nanog-bounces () nanog org<mailto:nanog-bounces () nanog org>] On Behalf Of David
> > Coulson
> > > > > Sent: Wednesday, August 27, 2014 6:59 AM
> > > > > To: Rick Coloccia
> > > > > Cc: <nanog () nanog org<mailto:nanog () nanog org>>
> > > > > Subject: Re: Time Warner outage?
> > > > >
> > > > > Just came back up for me.
> > > > >
> > > > > Sent from my iPad
> > > > >
> > > > > > On Aug 27, 2014, at 6:48 AM, Rick Coloccia <coloccia () geneseo edu<mailto:coloccia () geneseo edu>>
> > wrote:
> > > > > > BGPMON shows my routes falling off the net at around 5:49am.
> > > > > >
> > > > > > We now sit at their mercy....
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Sent from my iPhone
> > > > > >
> > > > > > > On Aug 27, 2014, at 6:46 AM, "Adam Greene"
> > > > > > > <maillist () webjogger net<mailto:maillist () webjogger net>>
> > > > > wrote:
> > > > > > > Same here. Seems like no traffic is exiting TWC:
> > > > > > >
> > > > > > > Tracing route to ns03.savvis.net<http://ns03.savvis.net> [204.70.25.234<tel:%5B204.70.25.234>] over a maximum
> > > > > > > of
> > 30
> > > > > > > hops:
> > > > > > >
> > > > > > > 1     1 ms     1 ms     1 ms  10049.webjogger.net<http://10049.webjogger.net> [204.8.80.49]
> > > > > > > 2    27 ms     6 ms     5 ms  cpe-24-29-112-25.nyc.res.rr.com<http://cpe-24-29-112-25.nyc.res.rr.com>
> > > > > > > [24.29.112.25]
> > > > > > > 3    12 ms     8 ms     7 ms  rdc-69-193-225-74.nyc.bc.twcable.com<http://rdc-69-193-225-74.nyc.bc.twcable.com>
> > > > > > > [69.193.225.74]
> > > > > > > 4     9 ms     9 ms    12 ms  
> > > > > > > rdc-69-193-225-137.nyc.bc.twcable.com<http://rdc-69-193-225-137.nyc.bc.twcable.com>
> > > > > > > [69.193.225.137]
> > > > > > > 5     9 ms    10 ms     8 ms  
> > > > > > > rdc-69-193-225-222.nyc.bc.twcable.com<http://rdc-69-193-225-222.nyc.bc.twcable.com>
> > > > > > > [69.193.225.222]
> > > > > > > 6     *        *        *     Request timed out.
> > > > > > > 7     *        *        *     Request timed out.
> > > > > > > 8     *        *        *     Request timed out.
> > > > > > > 9     *        *        *     Request timed out.
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: NANOG [mailto:nanog-bounces () nanog org<mailto:nanog-bounces () nanog org>] On Behalf Of Rick
> > > > > > > Coloccia
> > > > > > > Sent: Wednesday, August 27, 2014 6:43 AM
> > > > > > > To: nanog () nanog org<mailto:nanog () nanog org>
> > > > > > > Subject: Re: Time Warner outage?
> > > > > > >
> > > > > > > My whole campus (~10000 users) is down... Since roughly 6am.
> > > > > > > TWC is our upstream.
> > > > > > >
> > > > > > > --
> > > > > > > Sent from my iPhone
> > > > > > >
> > > > > > > > On Aug 27, 2014, at 6:28 AM, Rob Barbeau
> > > > > > > > <rob.barbeau () gmail com<mailto:rob.barbeau () gmail com>>
> > wrote:
> > > > > > > > David,
> > > > > > > >
> > > > > > > > I have a branch office in Syracuse,NY that appears to be down
> > > > > > > > at the moment that uses a time warner business connection for
> > > > > > > > internet
> > access.
> > > > > > > > -rob
> > > > > > > > On Aug 27, 2014 5:20 AM, "David Hubbard"
> > > > > > > > <dhubbard () dino hostasaurus com<mailto:dhubbard () dino hostasaurus com>>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hey all, anyone else having issues with Time Warner
> > > > > > > > > residential or business connections?  One of our offices is
> > > > > > > > > down and the route is not currently in bgp.
> > > > > > > > > http://downdetector.com/status/time-warner-cable
> > > > > > > > > shows thousands of reports of outages on the consumer side
> > > > > > > > > starting an hour or so ago so I figure it's a larger issue
> > > > > > > > > than just my one office; couldn't reach anyone by phone.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > David
>
>
> --
> //CL



--
//CL