Re: Verizon connectivity issues

[Christopher Morrow <morrowc.lists () gmail com>]

On Thu, Aug 28, 2014 at 10:18 AM, Bacon, Ricky (RJ)
<rj.bacon () verizon com> wrote:
> Hi Chris!

howdy!

> @Ryan, XO was accurate, the peering connection between 6 and 7 is not


I had thought Ryan wasn't at XO but an XO customer... I could be
mistaken though.

congested, nor is it taking any sort of errors.
> I took a look.  The significant increase in rtt seems to happen between 7 and 8 (an Verizon LSP consisting of 
> multiple physical routers).  That hop traverses the

yikes :) lsp's from XT -> LCR now, neat. it's also possible (and
likely) that the return path from VZ customer -> Ryan is across either
the NYC or DCA links to XO (nyc4 or dca6?)...  I agree that the ~65ms
cross-country seems on-par with SJC -> PHL on a good day though.

continental US from San Jose to Philadelphia, so you would expect
that, and I don't see anything going on in that path.  The end to end
rtt is pretty normal, and in any case, it shouldn't prevent your
customers from connecting.  I am not seeing any packet loss on that
path.  It's possible that whatever they saw was a transient issue and
may be over now.
> So, please double check with your customers to verify that they still have problems.  If so, continue to escalate the 
> ticket.

hopefully ryan's having a better day today :)

> -- rj
>
> -----Original Message-----
> From: NANOG [mailto:nanog-bounces () nanog org] On Behalf Of Christopher Morrow
> Sent: Wednesday, August 27, 2014 10:52 PM
> To: ryanruuska+nanog () gmail com
> Cc: nanog list
> Subject: Re: Verizon connectivity issues
>
> On Mon, Aug 25, 2014 at 1:54 PM, Ryan Ruuska <ryanruuska+nanog () gmail com> wrote:
> > Hello all,
> >
> > We have heard from many of our customers in the Northeast region,
> > specifically PA, MD, and VA who have difficulty connecting to our
> > website
>
> there's probably vz customer folk on-list, maybe 'what should they test for you' would be nice? :)
>
> > (very slow loading times).  We have noticed that if our data center
> > provider specifically routes our outbound traffic over Hurricane
> > Electric rather than XO Communications, that connectivity is restored
> > to normal for our customers.  The HE path goes from Salt Lake City to
> > Denver where it is handed off to TeliaSonera, and they send it to
> > Chicago where it gets handed off to Verizon.  This works great and
> > generally has a ping response of 20-25ms less than the bad route as posted below from one of our servers:
> >
> > Tracing route to pool-108-52-xxx-xxx.phlapa.fios.verizon.net
> >  [108.52.xxx.xxx]
> > over a maximum of 30 hops:
> >
> >   1    <1 ms    <1 ms    <1 ms  172.xxx.xxx.xxx (Internal IP)
> >   2    <1 ms    10 ms    <1 ms  209-41-xxx-xxx.c7dc.com [209.41.xxx.xxx]
> >   3    <1 ms    <1 ms    <1 ms  e1-5.rt08.gp1.c7dc.com [192.41.0.97]
> >   4     2 ms     1 ms     1 ms  ip65-46-51-113.z51-46-65.customer.algx.net
> >  [65.46.51.113]
> >   5    19 ms    18 ms    19 ms  vb1611.rar3.sanjose-ca.us.xo.net
> >  [216.156.0.5]
> >   6    21 ms    18 ms    18 ms  207.88.14.226.ptr.us.xo.net [207.88.14.226]
> >   7    18 ms    18 ms    18 ms  206.111.6.122.ptr.us.xo.net [206.111.6.122]
> >  >>>> Verizon owned device with an XO IP address
> >   8    87 ms    86 ms    87 ms  b100.phlapa-lcr-22.verizon-gni.net
> > [130.81.209.187]
> >  >>>> Next hop goes from CA to PA, probably just hidden routing info
> >   9     *        *        *     Request timed out.
> >  10    88 ms    89 ms    90 ms  pool-108-52-xxx-xxx.phlapa.fios.verizon.net
> >  [108.52.xxx.xxx]
> >
> > As you can see, XO sends this traffic from Salt Lake City to San Jose
> > where it is handed off to Verizon.  We've worked with XO and
> > determined that there are no connectivity issues along their path,
> > which seems to point to an issue within Verizon's network.  I have a
> > couple of tickets open with Verizon at the moment on behalf of some of
> > our customers, but we have had trouble breaking through Tier 1.
> >
> > Our first thought was saturation of the peering point, but XO states
> > that the link between hops 6 and 7 is using 5Gbps out of 20Gbps
> > capacity, and the latency on hop 7 (the Verizon owned device with an
> > XO IP) seems normal whenever we test.
> >
> > Any Verizon engineers out there willing to take a quick peek at this
> > route for any obvious issues?  It would be a lot easier for us if
> > Verizon provided Looking Glass servers, but alas...
>
> they have route data sent to routeviews at least...