Re: IPv6 woes - RFC

[Owen DeLong via NANOG <nanog () nanog org>]

> On Sep 19, 2021, at 16:17 , Victor Kuarsingh <victor () jvknet com> wrote:
>
> Owen,
>
>
> On Sat, Sep 18, 2021 at 23:51 Owen DeLong <owen () delong com <mailto:owen () delong com>> wrote:
> > On Sep 18, 2021, at 12:34 , Victor Kuarsingh <victor () jvknet com <mailto:victor () jvknet com>> wrote:
> >
> > On Sat, Sep 18, 2021 at 2:39 PM John Levine <johnl () iecc com <mailto:johnl () iecc com>> wrote:
> > It appears that Owen DeLong via NANOG <owen () delong com <mailto:owen () delong com>> said:
> >
> >
> > Glad you noted this.  Thinking this was/is purely a hardware cycle problem related to normal/forced upgrade 
> > strategies.  On that point, most hardware I know of from 2004 in larger networks is long fully depreciated and 
> > sweating assets 15+ years can happen, but I don't personally think this is the biggest issue.
>
> It’s certainly not purely hardware, but it also doesn’t require solving the entire problem across the entire backend.
>
> What is urgently needed is to fix the customer-facing front end so that it speaks both protocols (or at least speaks 
> IPv6).
>
> This is a great question.  So when we approached this (going back a while now) this is what I thought too.  But I was 
> responsible to solve this end to end.  So just updating front end (CPEs, network routers, DHCP, provisioning, IP 
> address planning, security, peering/transit, staff training, test harnesses/plans, etc) was not sufficient to turn on 
> IPv6. 
>
> The high cost and time challenge issues were not upgrading backend servers to IPv6, but backend provisioning systems, 
> tools, customer support tools, their training, and related items.  These systems were far more complex and costly to 
> address (especially when considering opportunity cost).   Without all of this in place, we could not actually deploy 
> IPv6 for production use (it’s not just Turing it on, its our ability to operate it down to the person answer the 
> phone, the technician that installs and/or fixes/replaces items in home). 

This sounds like an eyeball environment… Note that my question was in the context of the laggard content providers.

Eyeball providers are going to have to do something eventually whether they like it or not and I’m a whole lot less 
worried about a forcing function for them.

The major eyeball providers have basically completed this. Hopefully that means that the vendors you’re using have been 
forced to upgrade their code and such to accommodate by now.

> Also at that time vendor CPE hardware was not as far along on IPv6 readiness (approx mid-decade 2000s).  Getting 
> reliable code at that time was hard with a lot of brokenness (which also could not go into production until it was 
> reliable).  Perhaps this a non-issue today, but it certainly was not a something that was “ready to go” even 10-15 
> years.   This (IPv6 readiness in CPE code) was also competing with other priorities often blowing up timelines which 
> meant it had to wait as not releasing code into production on a strict timeline was not an option for business 
> reasons.

Sure, but a lot of that has gotten better in the recent years, largely driven by some of the major eyeball ISPs.

> All said and done, we got it completed, but it far most costly than we first thought, and IPv4 costs did not go away 
> after we were completed.  Sure it’s possible to reduce those costs with an aggressive program, but it is not as 
> simple as many think.

Agreed… IPv4 costs for you, as an eyeball provider, can’t really go away until you can start doing one or more of the 
following:
        1.      Raising your overall rates and providing a discount to IPv6-only customers
        2.      Adding a surcharge to your billing for customers still requiring IPv4 services
        3.      Turning off or reducing availability of IPv4 native services and moving more towards IPv4AAS
        4.      Turning off IPv4 services outright

All of those basically depend on moving a few key laggard content providers forward.

> > As you noted John, its the plethora of software, support systems, tooling, and most important in many environments - 
> > legacy customer management and provisioning systems that can be the limiting factor.  I recall looking back when 
> > leading IPv6 turn-up, those were the biger problems to solve for.  Often such systems are extremely expensive to 
> > touch and working on them required prioritization against direct revenue generating projects (opportunity cost) .  
> > Replacing routers was just a money problem.
>
> Why do those systems have to be updated in order to deliver the service to the customer in both protocols?
>
> Addressed in above comment.  

Eyeball… Now think about it in the context of a content provider… Especially one that is behind a CDN where it’s 
literally simply flipping the switch on the CDN
that says front end our stuff with both protocols.

> When it comes to turning on IPv6 and reducing your dependency on IPv4, your mileage may vary (in terms of real costs, 
> complexities and deployment).   

Agreed… Much more variable for eyeballs, but still some variability for content providers, too. Nonetheless, I think it 
is generally quite simple for content providers today while there are still some issues that remain unsolved on the 
eyeball side of things.

The good news is that once a few laggard content providers add IPv6 capabilities, eyeball providers can start treating 
IPv4 as (mostly) optional in a lot of areas, so that those which have deployed IPv6 to their customers in a meaningful 
way can start to shrink their IPv4 costs.

Owen