Re: Zayo woes

[Billy Earley via NANOG <nanog () nanog org>]

Navigating the Zayo maze has become my five-month odyssey, attempting to breathe life into a new wave circuit. Yet, at 
every turn, the elusive Letter of Authorization (LOA) emerges as a formidable opponent, adorned with syntax problems or 
entangled in the chaos of already occupied ports.

Securing new LOAs has evolved into a drawn-out saga, reminiscent of a never-ending loop that defies both reason and 
resolution. Weeks are spent coaxing these documents into existence, only to find myself caught in the clutches of the 
same cycle, ad infinitum.

Communication with Zayo has been challenging. Phone calls always go unanswered. Persistent radio-silence despite emails 
promising regular updates. Zayo's website boldly touts "waves on demand" with a 24-hour turn-up, a tantalizing promise 
that seems more mirage than reality in my current struggle.

Wish me luck as I continue my quest through the Zayo wilderness. May the waves finally align in my favour some day 
soon. If anyone at Zayo is reading this - I would love to hear from you off-list.



> On Sep 21, 2023, at 01:15, Richard Holbo <holbor () sonss net> wrote:
>
> Laughing out Loud, really, good views all...
> Having been through this a few times.. and being one of those who is now the one of the hated C level guys..
> Much truth is spoken here.  EBITA and size are the issues IMHO in our current system.  
> Having been the owner of a few "smallish" retail ISPs in the west.. I can say with certainty that size is an issue.  
> in the world of today a small/medium ISP can do OK, but can never access the funds or resources necessary to actually 
> be a long term survivor.  To that end we look at sources of money that require us to sell a majority interest in our 
> company to make it to the next level, that and exhaustion from staying awake nights wondering how to make next 
> payroll because we spent a couple million on new gear. 
> This money seeking then comes with an interview system where we court the guys with money and they court us.. The 
> outcome is always iffy.  You can partner with money that has a good plan or one that sucks (done both).
> Even if you get good money, you will suffer from culture issues.  Small ISP's tend to focus on the people and the 
> customers, the larger you get the more important the money becomes (EBITA) which for a lot of the employees is, well, 
> hard.
> But staying small in general = extinction so  you do what you do to keep employees working (at least that's what I 
> tell myself).
> Good views all, and I totally agree with @Matthew Petach miracles statement.. I'm no longer that guy, but I like to 
> think I was in the past, and I've got a bunch of them working for me now, so I try really hard to appreciate it and 
> to recognize that that is what is happening, I fight for the money to do it as right as possible.. but I do depend on 
> .... you miracle workers.
>
> /thanks
> /rh
>
> On Wed, Sep 20, 2023 at 9:30 AM Matthew Petach <mpetach () netflight com <mailto:mpetach () netflight com>> wrote:
> > On Tue, Sep 19, 2023 at 12:21 PM Mike Hammett <nanog () ics-il net <mailto:nanog () ics-il net>> wrote:
> > > Well sure, and I would like to think (probably mistakenly) that just no one important enough (to the money people) 
> > > made the money people that these other things are *REQUIRED* to make the deal work.
> > >
> > > Obviously, people lower on the ladder say it all of the time, but the important enough money people probably don't 
> > > consider those people important enough to listen to.
> >
> >
> > Not quite.
> >
> > It's more of what Mark said:
> >
> > "  I blame this on the success of how well we have built the Internet with whatever box and tool we have, as network 
> > engineers."
> >
> > I have worked time and time again with absolute miracle workers in the networking field.
> > They say over and over again "to make this work, we need $X M to get the right hardware", even directly to the CFO.
> >
> > They get handed a roll of duct tape, some baling wire, a used access point and a $25 gift card from Office Depot, 
> > and they turn it into a functional BGP-speaking backbone, because that's what they're good at.
> >
> > The CFO and the rest of the executives that said "no" to the request for $X M to make the integration work properly 
> > pat themselves on the back, saying "see, we knew they didn't really NEED that money to make it work."
> >
> > A year down the line, customers are posting to NANOG wondering why things are going to hell in a handbasket at ISP 
> > A, as the BGP-speaking access point with some duct tape, baling wire, and SFPs purchased from Office Depot that ties 
> > the two networks together starts failing.
> >
> > As network engineers, we collectively set ourselves up for this by being so damn good at pulling miracles out of our 
> > backside to keep things running.
> > We've effectively been training our executives that if they habitually turn down our requests for resources, we'll 
> > still find some way of making things work.
> >
> > We pride ourselves on being able to keep a dozen spinning plates going like a circus performer, without letting any 
> > of them crash to the floor.
> >
> > It's a hard thing to do, but one lesson I've taught junior network engineers of all ages is that sometimes, you have 
> > to step back, and watch a plate smash into the floor, *even if you could have rescued it*, if it seems like that's 
> > the only way your executive team will understand that if requests for necessary resources are denied, there will be 
> > operational impacts.
> >
> > Now, it's not something you should do lightly, and not something to do without first working with the executives to 
> > understand why the resource request is being denied.
> > If you are working at a startup, and the money is running out, and the company is one step ahead of the creditors, 
> > probably not the time to put the foot down and intentionally let things crash and burn. 
> >
> > But if the company is doing well, has the money, and the executives just want the numbers to look good for wall 
> > street analysts, then it's time to pause the miracle working, and help them understand that they cannot simply 
> > expect you to pull a miracle out of your backside every time, just so they can look good.
> >
> > If we continue to pull off miracles after telling executives that additional resources are required, it's no wonder 
> > they don't take the requests as seriously as they should.  ^_^;
> >
> > Matt
> >
> >
> >  
> > > From: "Mark Tinka" <mark@tinka.africa>
> > > To: nanog () nanog org <mailto:nanog () nanog org>
> > > Sent: Tuesday, September 19, 2023 10:28:26 AM
> > > Subject: Re: Zayo woes
> > >
> > >
> > >
> > > On 9/19/23 16:48, Mike Hammett wrote:
> > > As someone that has been planning to be in the acquiring seat for a while (but yet to do one), I've consistently 
> > > passed to the money people that there's the purchase price and then there's the % on top of that for equipment, 
> > > contractors, etc. to integrate, improve, optimize future cashflow, etc. those acquisitions with the rest of what we 
> > > have.
> > >
> > > I blame this on the success of how well we have built the Internet with whatever box and tool we have, as network 
> > > engineers. 
> > >
> > > The money people assume that all routers are the same, all vendors are the same, all software is the same, and all 
> > > features are easily deployable. And that all that is possible if you can simply do a better job finding the 
> > > cheapest box compared to your competition.
> > >
> > > In general, I don't fancy nuance when designing for the majority. But with acquisition and integration, nuance is 
> > > critical, and nuance quickly shows that the acquisition was either underestimated, or not worth doing at all.
> > >
> > > Mark.