Re: COVID-19 vs. our Networks

["Livingood, Jason" <Jason_Livingood () comcast com>]

On this subject, this is worth a read:

https://transition.fcc.gov/pshs/docs/emergency-information/Pandemic_Comms_Impact_Study_%28December%202007%29.pdf
Department of Homeland Security
Pandemic Influenza Impact on Communications Networks Study
Dec 2007

JL

From: NANOG <nanog-bounces () nanog org> on behalf of Tom Beecher <beecher () beecher cc>
Date: Monday, March 16, 2020 at 2:10 PM
To: Mike Bolitho <mikebolitho () gmail com>
Cc: NANOG <nanog () nanog org>
Subject: Re: COVID-19 vs. our Networks

Mike-

The TSP program provides for priority treatment for only 2 things : provisioning of new capacity, and restoration of 
capacity. It provides no accommodations for intermittent degradation events upstream.

Source :

DHC Office of Emergency Communications, TSP Program Office, TSP Vendor Handbook.

https://www.cisa.gov/sites/default/files/publications/OEC%20Service%20Vendor%20Handbook%20for%20TSP%2010-23-2017%20FINAL%20508C.pdf<https://urldefense.com/v3/__https:/www.cisa.gov/sites/default/files/publications/OEC*20Service*20Vendor*20Handbook*20for*20TSP*2010-23-2017*20FINAL*20508C.pdf__;JSUlJSUlJSU!!CQl3mcHX2A!XxhMP9AfXmBNjiociWHdq1-r4dwGmRnCRuDMe2bkws8JFS7ZeBDDcSORQWi3sQehSRlv9eI9$>

On Mon, Mar 16, 2020 at 10:42 AM Mike Bolitho <mikebolitho () gmail com<mailto:mikebolitho () gmail com>> wrote:
I think there's a bit of a misunderstanding of what I'm trying to say here. We have dual private lines from two Tier I 
providers. These interconnect all major hospitals and our data centers. We also have a third metro connection that 
connects things regionally. We have DIA on top of that. I think people are vastly underestimating just how much $aaS 
there is within the medical field. TeleDoc, translation services, remote radiologists, the way prescriptions get 
filled, how staffing works, third party providers basically hoteling within our facilities, critical staff VPNed in 
because the government has locked things down, etc. Then there's things that we don't use but I'm sure other providers 
do, GoToMeeting, O365, VaaS, etc. There's no practical way to engineer your WAN to facilitate dozens of connections to 
these services.

This extends beyond just hospitals as well. Fire departments, police departments, water treatment etc. Regardless of 
whether or not those entities planned well (I think we did), the government should and will step in if critical 
services are degraded. And for what it's worth, Stephen, I know how things are built within the ISP world. I spent four 
years there. That doesn't change the fact that we're possibly heading into uncharted waters when it comes to 
utilization and the impact that will have on $aaS products that are interwoven into every single vertical, including 
entities that fall under TSP, critical national security and emergency preparedness functions, including those areas 
related to safety, maintenance of law and order, and public health. It's easy for all you guys to sit here and armchair 
quarterback other people's planning but when things really start to degrade, all bets are off. If you don't believe 
that, just look at the news. States are literally shutting down private businesses (restaurants, bars, night clubs, 
private schools) and banning people from associating in groups of larger than 50.
The opinions expressed here are my own and do not represent my employer or their views.

- Mike Bolitho

On Sun, Mar 15, 2020 at 6:12 PM Stephen Fulton <sf () lists esoteric ca<mailto:sf () lists esoteric ca>> wrote:
In $dayjob I constantly see the lack of understanding of the difference
between what the Internet is and what a path engineered private circuit
is (eg. pseudowire, wave, whatever).  The latest fight is over SD-WAN
and those who think it will replace MPLS entirely and they won't need
those expensive routers anymore.  But I digress.

Mark's comment and others like it are the correct approach Mike.  If
your private WAN is most critical, then invest in and manage user
complaints about poor Internet service.  ISP's, IXP's and CDN's are not
going to twist themselves into knots to solve your problems, even if
someone calls it an emergency.  Sorry.

Stephen


On 2020-03-15 02:01, Mark Tinka wrote:
> On 14/Mar/20 19:14, Mike Bolitho wrote:
>
> >     /
> >     /
> >
> > I work for a hospital, we ran into some issues last week due to
> > congestion that was totally outside of our control that was off of our
> > WAN (Thanks Call Of Duty). Now, the issue we ran into was not mission
> > critical at the time but it was still disruptive. As more and more
> > people are driven home during this time, more and more people will be
> > using bandwidth intensive streaming and online gaming products. If
> > more and more TSP coded entities are running into issues, ISPs, IXPs,
> > and CDNs will be forced to act.
>
> Hmmh, if that level of priority is required, I'd probably build my own
> network, and not rely on public infrastructure like the Internet.
>
> Mark.