RE: Verizon FiOS - is BGP an option?

["Ralph E. Whitmore, III" <ralphw () interworld net>]

What I think most people are objecting to is that most of the issue with maintain service is not related to technical 
capabilities, but related to the cost of providing these support services impacting the profit margins of the large 
monopolistic carriers.  Verizon with their FIOS offerings, at least in my area is CO based and all optical, so all I 
have to provide is power to my own FIOS terminal which is easy to do, floated on batteries (not the POS that VZ 
provided but a real battery bank) and I stay online. We have been through 10 outages at least 10 hours in duration and 
one 3 day outage all courtesy of SCE who can't figure out how to replace the 57 year old wires that keep breaking from 
corrosion here in So. Calif.  As a subscriber, I paid for the copper Wire that VZ installed, I paid for the Copper that 
Edison installed, I pay to maintain  both of these services on a 7x24x365 basis. And each and every month, SCE and VZ 
take  a mandatory deduction out of my bill specifically to replace the copper every  50 years (the design life of the 
infrastructure).  Edison squanders that money and just patches the infrastructure with no regard for the customers, and 
VZ replaces the copper with FIOS so they don't have to allow any competition from anyone else and demos the copper when 
they are done so no one else can use it.  All of these companies fail to understand that they were granted a license  
and handed the keys to provide a public service and we expect them to perform that job rain or shine 7x24x365.  
Hurricanes (while not a problem where I am) are a known problem in many parts of the company and it is their job to 
maintain service despite the hurricanes.  These fiber huts/RSU's were installed to minimize VZ's (insert your favorite 
carrier here) cost of maintenance for their network . This way, they can increase their profits by laying off more 
workers and hiring more subcontractors.  So be it, that is their business model.  What people/PUC/ Regulatory bodies 
fail to follow up on is that just because they are allowed to install Fiber Huts/RSU's the customers should expect the 
same level of service and redundancy that is provided by a brick and mortar CO built to the ATT/Bellcore standards for 
stability and reliability.  I am all for the carriers pushing the edge closer to the customers, but it should not be 
allowed to occur at a substandard level.  They certainly aren't offering a discount for substandard service received by 
some.  My customers get 99.999% reliability from my infrastructure, I expect the carriers to do at least as well, 
obviously that doesn't happen.

All my Roadside cabinets have a DC plant that is engineered to hold the facility for  at least 18 hours based on the 
equipment in the box. All my facilities have an external Transfer switch and a generator plug.  A single 5kw generator, 
with one cable and padlock,  with one tank of fuel (generally propane or diesel)(about 5 gallons) will run the hut for 
8-10 hours, fully recharge the DC plant inside buying you another 18 hours on battery (therefore 28 additional hours 
before fuel is needed) and to boot have electronic monitoring to tell me if there is another AC failure of the 
generator during that 8-10 hours.   One contractor, god forbid we wouldn't do this with actual employees of the 
company, with a pickup truck and a small trailer should be able to support between 12-20 Fiber Huts /RSU's in a single 
shift. If your site is bigger than a 5KW site then you should have generator backup onsite in the first place.  If I 
can do this Verizon/ATT/Centurylink should have no trouble supporting their facilities in a similar fashion.  I am a 
little dinky company with 12 employees, but Verizon( your carrier name here) If you want to use Wisper Watt 25KW 
generators, more power to you, they will just pass the costs on to us anyways in the next rate hike.  The argument 
about lack of fuel availability is total bunk as well as I can't believe that any carrier doesn't already have  1000's 
of gallons onsite already to support their CO's, trucks, ect. if you need more you can order a 5000gallon tank like 
everyone else in the real world does, you can probably have it delivered tomorrow afternoon if you place the order 
tomorrow morning before noon.  Lack of power at an Fiber hut/RSU is negligence at best on a carriers part, not an 
unsolvable technical problem

My uncle who retired from Bell Atlantic/VZ after 30 years was in a rural area of Virginia and now works as a contractor 
for emergency restoration  after storms.  He was called in to work on Storm  restoration  he along with 20 other crews, 
were told report to the staging yard and wait for an assignment. It took three days to get to the staging area, They 
waited 4 days in the yard  before being assigned work and was then told that they couldn't work more than 12 hours in a 
day, because VZ would only pay for  a max of 4 hours OT for any personnel  in a 24 hour period.  They worked for 4 days 
and were then sent home as they were no longer needed.  To the best of my recollection  people were without service for 
weeks if not months. The safety wackos in the world will say well we don't want to tax our employees and more than 12 
hours isn't safe. Ok, I will not argue the validity of their argument, personally I disagree with it,  but this is an 
emergency response not a shareholders meeting, but sending the crews home weeks before all service is restored doesn't 
resonate well.   IMHO Restoration was not about serving the people who were out of service, but rather spending at 
little as they could get away with without regard for the people who pay for the infrastructure and pay their bills.  
While we are at it,  my uncle installed the RSU he is serviced from at his house before he retired from Bell 
Atlantic/VZ, they engineered only 4 hours of battery backup for their site near his house, the fiber hut/RSU is approx. 
8'x8'x5' and it has only 4 -105AH batteries installed in it, there are three vacant shelves for more batteries and one 
totally vacant compartment left in the cabinet for battery expansion.  Given the rural nature of his service RSU, the 
nearest stocking yard for VZ is better than an hour's drive from his RSU.  Given the Union contract gives the workers 2 
hours to show up at the yard from the time they are contacted ( which could take hours to do in the first place if his 
RSU is down) there is almost no way that you can callout a tech and make it to the site before the battery back is 
exhausted and Dominion Power is highly un-likely to arrive onsite in less than 12 hours from the time of call. So this 
site is doomed to go down from the start.

A single house, my house, your house, despite what we think are not critical infrastructure to the carriers, but the 
fiber huts that service them certainly is and needs to be supported better than they are now.

Back to the original topic, BGP works great over FIOS if you have a business account with static IP's and a Cooperative 
Colo.  I have IPv4 and IPv6 running at my house and office (we microwave the Home's business FIOS to the office) 
through a GRE tunnel.  It's the cheapest 35x35Mb connection around for about $200.00 with 32 static IP's around TWC 
wants about $2k for the same service.  It took a little bit of fiddling to get the  MTU right across the tunnel (ip tcp 
adjust-mss is your friend)  I could upgrade the service for another $100.00 to 150Mb x  65Mb, but the wireless link 
won't support that without a large expenditure and for 12 people it's not needed.


Boy that was a lot, glad I had time on a Sunday night to write this after fueling my generator.

Ralph


-----Original Message-----
From: Frank Bulk [mailto:frnkblk () iname com] 
Sent: Sunday, August 05, 2012 7:41 PM
To: 'William Herrin'; Andy Koch
Cc: nanog () nanog org
Subject: RE: Verizon FiOS - is BGP an option?

I think the term "critical" is being used in different senses in this discussion.  Are people's lives critical?  Yes, 
but the regulations for wired and wireless infrastructure don't require service providers to expend any and all costs 
to maintain connectivity.  And we don't have ambulances and fire trucks at every corner and hospitals in every 
subdivision.  Despite the almost incalculable value of life, there are still limitations on all the services provided 
to residences.  Would I like to have the same uptime at my home as we have in the CO? or data center?  Sure, but 
collectively we aren't willing, nay, able, to pay that price.

Frank

-----Original Message-----
From: William Herrin [mailto:bill () herrin us]
Sent: Saturday, August 04, 2012 11:15 PM
To: Andy Koch
Cc: nanog () nanog org
Subject: Re: Verizon FiOS - is BGP an option?

<snip>

On Sat, Aug 4, 2012 at 10:26 PM, Nathan Eisenberg <nathan () atlasnetworks us> wrote:
> > Residences aren't critical infrastructure, no matter how angry the 
> > owners
get.
> 911 access isn't a critical service?  Fire and security panels aren't
critical services?
> If basic life safety and property protection aren't critical services, 
> I'm
not sure what is.

Whether each individual's residence contains critical infrastructure is a decision best left up to that individual. By 
necessity that makes the upstream aggregation components critical infrastructure. No different than it was for POTS 20 
years ago.

The Internet isn't just a toy any more. It's the primary communications channel in to many folks homes and well on its 
way to becoming the primary channel period.

Regards,
Bill Herrin


--
William D. Herrin ................ herrin () dirtside com  bill () herrin us
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004