RE: NANOG Digest, Vol 46, Issue 15

["Steve Dispensa" <dispensa () phonefactor com>]

Level 3 was down in KC, Chi, and San Jose (at least) for us between
about 8:10 and 8:40, plus or minus. Brought down SureWest in KC too.

 -Steve

> -----Original Message-----
> From: nanog-request () nanog org [mailto:nanog-request () nanog org]
> Sent: Monday, November 07, 2011 10:05 AM
> To: nanog () nanog org
> Subject: NANOG Digest, Vol 46, Issue 15
>
> Send NANOG mailing list submissions to
>       nanog () nanog org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>       https://mailman.nanog.org/mailman/listinfo/nanog
> or, via email, send a message with subject or body 'help' to
>       nanog-request () nanog org
>
> You can reach the person managing the list at
>       nanog-owner () nanog org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of NANOG digest..."
>
>
> Today's Topics:
>
>    1. Re: Time Warner Telecom problems (Jon Lewis)
>    2. Re: Performance Issues - PTR Records (Leigh Porter)
>    3. Re: Time Warner Telecom problems (Ray Van Dolson)
>    4. General Internet Instability (Jared Mauch)
>    5. Re: TATA problems? (Pierre-Yves Maunier)
>    6. Re: TATA problems? (Leigh Porter)
>    7. Re: Time Warner Telecom problems (Joe Greco)
>    8. Re: TATA problems? (Kelly Kane)
>    9. Re: Time Warner Telecom problems (Blake Hudson)
>   10. RE: Time Warner Telecom problems (Thomas York)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 7 Nov 2011 10:12:30 -0500 (EST)
> From: Jon Lewis <jlewis () lewis org>
> To: Peter Pauly <ppauly () gmail com>
> Cc: nanog () nanog org
> Subject: Re: Time Warner Telecom problems
> Message-ID: <Pine.LNX.4.61.1111071005540.24418 () soloth lewis org>
> Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
>
> On Mon, 7 Nov 2011, Peter Pauly wrote:
>
> > Gizmodo is reporting problems at Time Warner Telecom .... we're
> suffering
> > from it too and calls to the NOC have not been answered so far...
does
> > anyone have any further information?
> >
> > http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
>
> I noticed just a little while ago that we're having a lot of DNS fail.
> Initial findings were that several of the root-servers we were trying
to
> reach via our TWTelecom link were unreachable after 2 hops into TWT.
>
>   4  64-128-130-233.static.twtelecom.NET (64.128.130.233)  2.399 ms
2.298
> ms  2.338 ms
>   5  mia2-pr1-xe-1-3-0-0.us.twtelecom.net (66.192.253.18)  11.571 ms
> 11.552 ms  9.467 ms
>   6  * * *
>   7  * * *
>   8  * * *
>
> For instance, a.root-servers.net is pingable from a rackspace server,
but
> not from our network (unless I shut off TWT, at which point it is, but
> it's apparently not the same a.root-servers.net instance rackspace
sees).
> I assume this is one of the root-servers being anycast.
>
> Shutting off our BGP with TWT didn't appear to help (though the
> root-servers became reachable)...so I assume there's more going on
than
> just TWT routing fail.
>
> ----------------------------------------------------------------------
>   Jon Lewis, MCP :)           |  I route
>   Senior Network Engineer     |  therefore you are
>   Atlantic Net                |
> _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
>
>
>
> ------------------------------
>
> Message: 2
> Date: Mon, 7 Nov 2011 15:29:30 +0000
> From: Leigh Porter <leigh.porter () ukbroadband com>
> To: Bj?rn Mork <bjorn () mork no>
> Cc: "nanog () nanog org" <nanog () nanog org>
> Subject: Re: Performance Issues - PTR Records
> Message-ID: <53A4963F-4969-4A60-BF06-E690C7324863 () ukbroadband com>
> Content-Type: text/plain; charset="iso-8859-1"
>
>
>
> On 7 Nov 2011, at 14:03, "Bj?rn Mork" <bjorn () mork no> wrote:
>
> > Leigh Porter <leigh.porter () ukbroadband com> writes:
> >
> > > Indeed, there is no way I would allow that either. But really,
> > > providing a reverse zone and forward zone to match is a case of
five
> > > minutes and a shell script or a DNS that as Steinar said, will
> > > synthesise results.
> > >
> > > It's really not all that difficult..
> >
> > No, not at all.  It's just totally pointless.  Any IPv6 address is
just
> > as pretty as a synthesized name.  Maybe even prettier. Do you prefer
> > "2001:db8:1::2" or
"20010db8000100000000000000000002.rev.example.com"?
> > If we're going to provide any reverse DNS for end users, then it is
> > because we can create names which actually improves something.
> >
> >
> > Bj?rn
>
> Yup it is pointless.. Mine are all ipadrress.domain which is of
course,
> pointless.. I suppose at least somebody would glean that perhaps its a
> home user rather than a business or server on that address but that's
all.
> With IPv6 arguably even more pointless as you say.
>
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email
> ______________________________________________________________________
>
>
>
> ------------------------------
>
> Message: 3
> Date: Mon, 7 Nov 2011 07:28:18 -0800
> From: Ray Van Dolson <rvandolson () esri com>
> To: nanog () nanog org
> Subject: Re: Time Warner Telecom problems
> Message-ID: <20111107152817.GA29715 () esri com>
> Content-Type: text/plain; charset=us-ascii
>
> On Mon, Nov 07, 2011 at 07:04:19AM -0800, Peter Pauly wrote:
> > Gizmodo is reporting problems at Time Warner Telecom .... we're
> suffering
> > from it too and calls to the NOC have not been answered so far...
does
> > anyone have any further information?
> >
> > http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
>
> FWIW, my home TWC connection dropped this morning for about 15 minutes
> (Southern California around 6:30AM'ish).  Still could ping the default
> gateway, but packets weren't traversing much beyond that.
>
> Didn't investigate further, just headed into work.
>
> Ray
>
>
>
> ------------------------------
>
> Message: 4
> Date: Mon, 7 Nov 2011 10:31:31 -0500
> From: Jared Mauch <jared () puck nether net>
> To: Tom Hill <tom () ninjabadger net>
> Cc: nanog () nanog org
> Subject: General Internet Instability
> Message-ID: <B6567BC5-75E9-4E58-AFA9-5ADC0C2A7BDD () puck nether net>
> Content-Type: text/plain; charset=us-ascii
>
> On Nov 7, 2011, at 10:08 AM, Tom Hill wrote:
>
> > On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote:
> > > We seem to be having some problems with our tata links - first seen
in
> EU
> > > about 45 minutes ago, now we're seeing problems in NA.  I'm focused
on
> DNS,
> > > so I'm seeing a lot of timeouts/servfails, but our networking folks
are
> > > talking about links dropping.
> > >
> > > Anyone else seeing oddness on the NA Internet right now?
> > >
> > > http://downrightnow.com/ confirms - something is up.
> >
> > There are widespread issues across the Internet; certain versions of
> > Juniper firmware have core dumped after seeing a particular BGP
'UPDATE'
> > message.
> >
> > (That's the running theory at least).
> >
> > It's affected multiple service providers, globally, not just those
> > connected to TATA.
>
>
> Pretty much any major BGP event will impact multiple providers.
>
> A threshold you should use to view the general instability (which I
find
> valuable, you may as well) is route views data.
>
> If you look at the BGP UPDATES archive sizes, you can see when
something
> happens, e.g.:
>
> http://archive.routeviews.org/bgpdata/2011.11/UPDATES/
>
> Take a look at the size of the updates.20111107.1400.bz2 file and the
1415
> file.  They are abnormally large compared to a normal period of time.
> This shows there were a lot of updates out there being processed and a
> reference to levels of instability.
>
> If you are not feeding route views or similar community projects,
please
> consider doing so.  It helps paint the view for those doing analysis.
>
> - Jared
>
>
> ------------------------------
>
> Message: 5
> Date: Mon, 7 Nov 2011 16:33:15 +0100
> From: Pierre-Yves Maunier <nanog () maunier org>
> To: Tom Hill <tom () ninjabadger net>
> Cc: nanog () nanog org
> Subject: Re: TATA problems?
> Message-ID:
<CAEGdXrM+2rFXG2=d80kO_1ObtAv=24npNqZsLCTXUxeKS9q9yA () mail gmail com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> 2011/11/7 Tom Hill <tom () ninjabadger net>
>
> > On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote:
> > > We seem to be having some problems with our tata links - first
seen in
> EU
> > > about 45 minutes ago, now we're seeing problems in NA.  I'm
focused on
> > DNS,
> > > so I'm seeing a lot of timeouts/servfails, but our networking
folks
> are
> > > talking about links dropping.
> > >
> > > Anyone else seeing oddness on the NA Internet right now?
> > >
> > > http://downrightnow.com/ confirms - something is up.
> >
> > There are widespread issues across the Internet; certain versions of
> > Juniper firmware have core dumped after seeing a particular BGP
'UPDATE'
> > message.
> >
> > (That's the running theory at least).
> >
> > It's affected multiple service providers, globally, not just those
> > connected to TATA.
> >
> > Tom
> On our side all our 10.3R2.11 core dumped which made all our
interfaces
> flapped.
> I've been told 10.4R1.9 is affected too.
>
> --
> Pierre-Yves Maunier
>
>
> ------------------------------
>
> Message: 6
> Date: Mon, 7 Nov 2011 15:45:18 +0000
> From: Leigh Porter <leigh.porter () ukbroadband com>
> To: Pierre-Yves Maunier <nanog () maunier org>
> Cc: "nanog () nanog org" <nanog () nanog org>
> Subject: Re: TATA problems?
> Message-ID: <7994AF08-0622-434F-974F-FC9269469176 () ukbroadband com>
> Content-Type: text/plain; charset="us-ascii"
>
>
> My 10.4r1.9 boxes died also but I saw interfaces go down whilst bgpd
> seemed stable.
>
> --
> Leigh
>
>
> On 7 Nov 2011, at 15:34, "Pierre-Yves Maunier" <nanog () maunier org>
wrote:
> > 2011/11/7 Tom Hill <tom () ninjabadger net>
> >
> > > On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote:
> > > > We seem to be having some problems with our tata links - first
seen in
> EU
> > > > about 45 minutes ago, now we're seeing problems in NA.  I'm
focused on
> > > DNS,
> > > > so I'm seeing a lot of timeouts/servfails, but our networking
folks
> are
> > > > talking about links dropping.
> > > >
> > > > Anyone else seeing oddness on the NA Internet right now?
> > > >
> > > > http://downrightnow.com/ confirms - something is up.
> > >
> > > There are widespread issues across the Internet; certain versions
of
> > > Juniper firmware have core dumped after seeing a particular BGP
> 'UPDATE'
> > > message.
> > >
> > > (That's the running theory at least).
> > >
> > > It's affected multiple service providers, globally, not just those
> > > connected to TATA.
> > >
> > > Tom
> > On our side all our 10.3R2.11 core dumped which made all our
interfaces
> > flapped.
> > I've been told 10.4R1.9 is affected too.
> >
> > --
> > Pierre-Yves Maunier
______________________________________________________________________
> > This email has been scanned by the MessageLabs Email Security
System.
> > For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email
> ______________________________________________________________________
>
>
>
> ------------------------------
>
> Message: 7
> Date: Mon, 7 Nov 2011 09:54:25 -0600 (CST)
> From: Joe Greco <jgreco () ns sol net>
> To: ppauly () gmail com (Peter Pauly)
> Cc: nanog () nanog org
> Subject: Re: Time Warner Telecom problems
> Message-ID: <201111071554.pA7FsPHb045359 () aurora sol net>
> Content-Type: text/plain; charset=us-ascii
>
> > Gizmodo is reporting problems at Time Warner Telecom .... we're
> suffering
> > from it too and calls to the NOC have not been answered so far...
does
> > anyone have any further information?
> >
> > http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
>
> Actually, it looks to me like they mean "Time Warner", because that's
> what they said.
>
> The company once known as "Time Warner Telecom" has always been a
> different entity, and hasn't been known as that in some time, now
> being called "twtelecom."  Much of that company is what was once
> known as inc.net, a Milwaukee area provider of the '90's.
>
> Time Warner Cable appears to have experienced an implosion this
morning,
> being out of service for about 11 minutes.  During that time, packets
> originating here in Milwaukee quickly died in Chicago;
>
>  1  76.46.192.1  8.320 ms  9.900 ms  7.974 ms
>  2  24.160.230.32  7.967 ms  5.975 ms  8.479 ms
>  3  24.160.229.132  8.471 ms  7.969 ms  10.991 ms
>  4  24.160.229.193  9.972 ms  9.973 ms
>     24.160.229.197  9.985 ms
>  5  * * *
>  6  * * *
>
> while packets destined for RR all seemed to be headed out to SJC, from
> what I can tell.
>
> ... JG
> --
> Joe Greco - sol.net Network Services - Milwaukee, WI -
http://www.sol.net
> "We call it the 'one bite at the apple' rule. Give me one chance [and]
> then I
> won't contact you again." - Direct Marketing Ass'n position on e-mail
> spam(CNN)
> With 24 million small businesses in the US alone, that's way too many
> apples.
>
>
>
> ------------------------------
>
> Message: 8
> Date: Mon, 7 Nov 2011 07:55:33 -0800
> From: Kelly Kane <kelly () hawknetworks com>
> To: Tim Vollebregt <tim () interworx nl>
> Cc: nanog () nanog org
> Subject: Re: TATA problems?
> Message-ID:
<CAKfXD0zrH9TJkN=7doGe1uAq8zxVszyvtJYfLLTbiC6+UMy4ww () mail gmail com>
> Content-Type: text/plain; charset=UTF-8
>
> On Mon, Nov 7, 2011 at 07:06, Tim Vollebregt <tim () interworx nl> wrote:
> > On #IX there are rumours about Junos version 10.3R2.11 being core
dumped
> and
> > rebooted, which makes sense.
>
> Perhaps related to Juniper PSN-2011-08-327? Did the whole router
> reboot, or just the service module?
>
> We saw one TATA session, and one Abovenet session flap.
>
> Kelly
>
>
>
> ------------------------------
>
> Message: 9
> Date: Mon, 07 Nov 2011 10:02:13 -0600
> From: Blake Hudson <blake () ispn net>
> To: nanog () nanog org
> Subject: Re: Time Warner Telecom problems
> Message-ID: <4EB80105.8060703 () ispn net>
> Content-Type: text/plain; charset=UTF-8; format=flowed
>
>
> Joe Greco wrote the following on 11/7/2011 9:54 AM:
> > > Gizmodo is reporting problems at Time Warner Telecom .... we're
> suffering
> > > from it too and calls to the NOC have not been answered so far...
does
> > > anyone have any further information?
> > >
> > > http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
> > Actually, it looks to me like they mean "Time Warner", because
that's
> > what they said.
> >
> > The company once known as "Time Warner Telecom" has always been a
> > different entity, and hasn't been known as that in some time, now
> > being called "twtelecom."  Much of that company is what was once
> > known as inc.net, a Milwaukee area provider of the '90's.
> >
> > Time Warner Cable appears to have experienced an implosion this
morning,
> > being out of service for about 11 minutes.  During that time,
packets
> > originating here in Milwaukee quickly died in Chicago;
>
> Using the looking glass from TWtelecom, we saw 30-60min outage
(roughly
> 8:30AM to 9:30AM CST) between the Kansas City location and our own
> server room in Kansas City. Other TWtelecom locations appeared to be
> unaffected. Perhaps TWtelecom is served by Timewarner or shares
> equipment in KC. Either way, none of our KC customers who were served
> via TWtelecom or Timewarner were able to reach us. Packets would hit
> Level 3 Communications and die in either direction at the border
between
> L3 and TW. FWIW, TW was showing a good BGP route to us and vise versa.
> http://lglass.twtelecom.net/
>
>
>
> ------------------------------
>
> Message: 10
> Date: Mon, 7 Nov 2011 11:04:59 -0500
> From: "Thomas York" <straterra () fuhell com>
> To: "'Blake Hudson'" <blake () ispn net>,     <nanog () nanog org>
> Subject: RE: Time Warner Telecom problems
> Message-ID: <c23401cc9d67$046baea0$0d430be0$@fuhell.com>
> Content-Type: text/plain;     charset="utf-8"
>
> FWIW, We saw issues here in Indianapolis between TWTC and L3 up until
a
> few minutes ago.
>
> --Thomas York
>
> -----Original Message-----
> From: Blake Hudson [mailto:blake () ispn net]
> Sent: Monday, November 07, 2011 11:02 AM
> To: nanog () nanog org
> Subject: Re: Time Warner Telecom problems
>
>
> Joe Greco wrote the following on 11/7/2011 9:54 AM:
> > > Gizmodo is reporting problems at Time Warner Telecom .... we're
> > > suffering from it too and calls to the NOC have not been answered
so
> > > far...  does anyone have any further information?
> > >
> > > http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
> > Actually, it looks to me like they mean "Time Warner", because
that's
> > what they said.
> >
> > The company once known as "Time Warner Telecom" has always been a
> > different entity, and hasn't been known as that in some time, now
> > being called "twtelecom."  Much of that company is what was once
known
> > as inc.net, a Milwaukee area provider of the '90's.
> >
> > Time Warner Cable appears to have experienced an implosion this
> > morning, being out of service for about 11 minutes.  During that
time,
> > packets originating here in Milwaukee quickly died in Chicago;
>
> Using the looking glass from TWtelecom, we saw 30-60min outage
(roughly
> 8:30AM to 9:30AM CST) between the Kansas City location and our own
server
> room in Kansas City. Other TWtelecom locations appeared to be
unaffected.
> Perhaps TWtelecom is served by Timewarner or shares equipment in KC.
> Either way, none of our KC customers who were served via TWtelecom or
> Timewarner were able to reach us. Packets would hit Level 3
Communications
> and die in either direction at the border between
> L3 and TW. FWIW, TW was showing a good BGP route to us and vise versa.
> http://lglass.twtelecom.net/
>
>
>
>
>
> End of NANOG Digest, Vol 46, Issue 15
> *************************************