Northern California fires and telecomm outages

[Mike <mike-nanog () tiedyenetworks com>]

Hello fellow operators,


    The recent northern California fires wiped us out along with many 
other service providers and services including cellular, landline, 
e-911, cable and the like and I feel it's of operational concern the 
fragility of the telecommunications networks that link us together and 
the experiences had during this most recent event. I hope this topic is 
welcome and I apologize in advance for length.


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

    I am the founder and owner of an Internet Service Provider business 
in Mendocino County, California and I also operate a CLEC (Competitive 
Local Exchange Carrier, a type of telecommunications company akin to ATT 
or Frontier), also in Mendocino County, California. The recent Northern 
California fires were again a direct and upfront demonstration of the 
potential for unanticipated widespread public disaster and again 
demonstrated how vulnerable we all are. Especially, the rising 
dependence on broadband service, and an expectation by end users that 
these services which are critical to them will continue functioning 
during emergencies as they may have no other access. There was a 
complete and total breakdown in telecommunication services due to fire 
which wiped out a primary telecommunications cable, and with it, e-911, 
land lines, cellular, cable and other telecommunication services. I hope 
that there will be a more complete public investigation and report on 
this latest incident, but in the mean time I would like to just offer 
what I know and to suggest where we could go from here.

    In the early morning of October 9, the Northern California fires 
were in effect and in Redwood Valley California, a fire was burning that 
destroyed a AT&T telecommunications cable that runs between Ukiah and 
Willits. This single cable is the primary carrier of nearly all 
telecommunications services north of Ukiah. Although I cannot comment on 
the totality of the outage due to this specific fire and line cut, it's 
widely reported elsewhere that services were affected all the way up to 
Arcata at least. For Mendocino County, I can report that this 
telecommunication outage resulted in loss of e-911, cellular, landline, 
cable and most all other broadband services, and caused a complete 
outage of our service too. This resulted in a virtual dark age, where 
reliable and up to date information concerning the emergency situation 
was unable to be communicated, people could not find their families or 
know whether they got out in time because nothing was working and no way 
to talk. We were reduced to using paper and posting notes at the Library 
or other emergency shelter locations or other places where people 
gathered in order to communicate. People in these areas also had to use 
cash to buy supplies. Nothing worked.

    We attempted to engage emergency restoration processes on our own 
for our own customers and services. We determined that we could rapidly 
deploy high capacity microwave connecting a remote location where we 
also already operate microwave, to our main point of presence at an ATT 
building in Ukiah, and provide a sufficient level of service to all of 
our customers that would be a huge benefit during this crisis. As a CLEC 
with an existing presence and power arrangement and so forth within this 
facility, it seemed natural that we should be able to go ahead and get a 
simple OK to go do the work. But, we encountered lethargy and 
insensitivity to the widespread outage and emergency conditions we are 
trying to address. We notified ATT of our need and only days later, 
without followup response, it was suggested that we somehow don't have 
the legal authority to install microwave (despite the fact that our 544 
page contract with ATT that allows our presence within their facilities 
in the first place DOES SAY 'microwave' is allowed), and further they 
suggest we'd have to give up all of our existing contractual rights and 
that if it could happen, it would be only with months of delay and 
endless expense and bureaucratic nonsense that is only designed to slow 
down and de-incentivize any such initiative. The equipment we were 
proposing, is small, low power, lightweight, license free gear that 
could easily fit complete in a car and be deployed within hours without 
prior notice or coordination. The electrical and structural requirements 
are next to nothing, and present absolutely no risk to anyone or the 
facility. We could have had service restored and further, could have 
plugged a critical gap by creating an active backup telecommunications 
path which could be used in future emergencies. I am frustrated with the 
lack of cooperation to restore services during this latest incident and 
I think it speaks volumes about the mentality of the incumbent 
providers. They can't get it together, but here I am standing by with 
the solution at least for my own customers and why, they just don't want 
to allow me to do it either.

    Mendocino County (and northern counties also connected on this same 
telecommunication cable) have suffered similar telecommunications 
outages before. Prior to this event, a cable just north of Hopland was 
cut by two individuals thinking they were stealing copper, resulting in 
widespread outages lasting upwards of 14 hours. And prior to that, a car 
crash on the coast hit the one pole where the cable system comes in from 
the east, wiping out all services along the coast for an extended period 
of time. And before that, a fire in redwood valley just north of coyote 
valley casino along the roadway burned out the cable again as in this 
instance. And before that, another fire north of Willits burned the 
cable again. And the time before that, a backhoe south of Hopland dug up 
the cable and cut it, again resulting in far reaching outages affecting 
hundreds of thousands of people.

    These repeated outages are not being taken seriously and nothing in 
substance is being done to fortify the telecommunications network 
against the next disaster, despite the fact that disaster can and will 
occur again. We have repeatedly drawn attention to these vulnerabilities 
but the public perception and awareness of the risks wanes quickly once 
the crisis is over. I think the incumbent providers have had ample 
opportunity to address the underlaying causes of their repeated 
widespread outages, and the empty promises and assurances they have made 
after past prior disasters should be proof that no actual steps will be 
taken, that they either do not truly understand or care about the effect 
that loss of telecommunications services has and the risk it presents to 
the public.

    The basic fundamental problem facing the telecommunications network 
in Mendocino County, is the fact that it is structured without any sort 
of backup communication links. A primary communications cable extends 
north along highway 101 at Hopland and then goes into Ukiah. This cable 
then splits out at Ukiah and goes north up 101 thru Willits, while 
another segment goes west thru Boonville to Mendocino and then up and 
down along the coast. A cut at any of several points results in the 
complete isolation of communities past the cut as what occurred during 
the recent fire. A basic solution to such a situation then, is to have 
additional cables to increase the connections between areas, so that if 
a cable is cut and disrupts communication flowing one way, that the 
communications can flow over another path from another direction. But 
the incumbent providers have failed so far to do anything and are likely 
to continue to ignore this lack of redundancy.

    I believe that there are some realistic solutions available, and 
based on my business experience in Mendocino County, I have developed 
the beginnings of a plan of action. The short of it is, there needs to 
be a redundant network of geographically diverse, very high capacity 
telecommunication links that connect the core of the County, in order to 
overcome the existing problem of there being only 1 connection today. We 
would need new fiber to connect the major cities from alternate 
directions, and we further would need new connectivity to other sources 
of out of region telecommunication services besides AT&T (Such as 
Level3, who has their own independent cable route into the county). It 
would further require service providers and emergency responders to all 
lease connectivity on this redundant system for full effect, the net 
result however, could be a telecommunications system that stands up and 
continues functioning and with emergency first responders as well as the 
general public able to communicate during emergencies.

    Although I am busy with business and ensuring my customers recover 
from this event, I certainly am happy to add additional background or 
technical information if anyone is interested. Going forward, I also 
could also use help in identifying funding sources and other contacts 
who could help with some of the heavy lifting that is going to be needed 
here and so any pointers would certainly be appreciated.

Thank you.



Mike Ireton
President
Your Town Online, Inc DBA WillitsOnline
Rural Broadband Now, LLC