[Story] When IPv6 Fixes IPv4 Peering Issues

[Brielle <bruns () 2mbit com>]

Hi all,

So fun story for you all, and a good lesson as to why spending the time to set up IPv6 can save your ass in a pinch.

The players in this story are Me (and the company I consult with for when they have problems like this), Comcast (gig 
biz fiber), and CenturyLink (1/4th gig biz fiber).

Now, some of you who have Comcast and/or CenturyLink/Lumen probably remember issues last year regarding IPSec traffic 
getting heavily fouled up at peering points somewhere.  And, if you were like me, you probably remember that it was, 
well, lets be honest, impossible to get it looked into or dealt with (in reality).

We resolved the issue ourselves between the three offices by switching to WireGuard which magically made the problems 
go away.

Things have been going great until last week, when we noticed one of our WireGuard peers between CL/Lumen in Cheyenne 
and Comcast Denver was down.  Packets from Den -> Cys were going through, but not Cys -> Den.  Cys -> Boise on CL was 
still working perfectly fine and was acting as a backup connection to the Den office.

I did my usual testing - changed ports, same behavior, changed IPs on the WireGuard endpoints on each end, same 
behavior.  Even temp changed destination of the tunnel on Cys end to another off network node, and packets were going 
through, so we knew it had to be something relating going CL/Lumen -> Comcast.

Weird thing was, I could dump iperf udp traffic over the same ports from same devices Cys -> Den, and the packets would 
go through perfectly fine...  So.. sounds like there's some sort of throttling or IDS in the way somewhere toying with 
things.

As expected, our first dealing with Comcast was less than spectacular where the tech tried to tell us that the live IPs 
they had assigned us, because they were a /27, they wouldn't work for VPN traffic (what?).  I had to walk away from 
that call and let my partner finish it.

We went to dinner, and as we were returning home and pulling into the driveway, I remembered we had 'wasted' (as some 
of you would put it) a bunch of time setting up IPv6 on the outward facing devices at each office...  including the 
WireGuard boxes.

I quickly reconfigured the Cys WireGuard node to connect to the Den node over IPv6 and, after WireGuard did its magic 
dynamically reconfiguring endpoints, suddenly the connection was back up and routing at full speed.  Hell yeah!

So, moral / TLDR of the story?

Don't discount taking the time to set up IPv6, even if it's just for your important devices.  Also, WireGuard > IPsec.

-- Brie

Sent from my iPad