Re: Conflicts and fiber cuts

[Laura Smith via NANOG <nanog () nanog org>]

On Sunday, March 6th, 2022 at 23:40, Sean Donelan <sean () donelan com> wrote:

> Historically, the largest telecommunication outages have been due > to operator error


Yeah, tell me about it.

In the very recent past there was a certain Tier 1 operator who decided to move a core router between X & Y within the 
same datacentre.  As you might imagine, this impacted a good few hundred fibres.

Physical move happened without issue, router powered up fine, but a large subset of fibres refused to come up.

Cue escalation, lots of testing, lots of talking to vendors etc.

Eventually 72 hours later the problem was solved.

Cause ?  SFP signals too high at new location, attenuators fitted, everything came up.