Re: Facebook post-mortems...

[Mark Tinka <mark@tinka.africa>]

On 10/6/21 06:51, Hank Nussbacher wrote:

> - "During one of these routine maintenance jobs, a command was issued 
> with the intention to assess the availability of global backbone 
> capacity, which unintentionally took down all the connections in our 
> backbone network"
>
> Can anyone guess as to what command FB issued that would cause them to 
> withdraw all those prefixes?

Hard to say, as it seems that the command was innocent enough, perhaps 
running a batch of other sub-commands to check port status, bandwidth 
utilization, MPLS-TE values, e.t.c. However, sounds like another 
unforeseen bug in the command ran other things, or the cascade process 
of how the sub-commands were ran caused unforeseen problems.

We shall guess this one forever, as I doubt Facebook will go into that 
much detail.

What I can tell you is that all the major content providers spend a lot 
of time, money and effort in automating both capacity planning, as well 
as capacity auditing. It's a bit more complex for them, because their 
variables aren't just links and utilization, but also locations, fibre 
availability, fibre pricing, capacity lease pricing, the presence of 
carrier-neutral data centres, the presence of exchange points, current 
vendor equipment models and pricing, projection of future fibre and 
capacity pricing, e.t.c.

It's a totally different world from normal ISP-land.


> - "it was not possible to access our data centers through our normal 
> means because their networks were down, and second, the total loss of 
> DNS broke many of the internal tools we’d normally use to investigate 
> and resolve outages like this.  Our primary and out-of-band network 
> access was down..."
>
> Does this mean that FB acknowledges that the loss of DNS broke their 
> OOB access?

I need to put my thinking cap on, but not sure whether running DNS in 
the IGP would have been better in this instance.

We run our Anycast DNS network in our IGP, mainly to always guarantee 
latency-based routing, but also to ensure that the failure of a 
higher-level protocol like BGP does not disconnect internal access that 
is needed for troubleshooting and repair. Given the IGP is a much more 
lower-level routing protocol, it's more likely (not impossible) that it 
would not go down with BGP.

In the past, we have, indeed, had BGP issues that allowed us to maintain 
DNS access internally as the IGP was unaffected.

The final statement from that report is interesting:

    "From here on out, our job is to strengthen our testing,
    drills, and overall resilience to make sure events like this
    happen as rarely as possible."

... which, in my rudimentary translation, means that:

    "There are no guarantees that our automation software will not
    poop cows again, but we hope that when that does happen, we
    shall be able to send our guys out to site much more quickly."

... which, to be fair, is totally understandable. These automation 
tools, especially in large networks such as BigContent, are 
significantly more fragile the more complex they get, and the more batch 
tasks they need to perform on various parts of a network of this size 
and scope. It's a pity these automation tools are all homegrown, and 
can't be bought "pre-packaged and pre-approved to never fail" from IT 
Software Store down the road. But it's the only way for networks of this 
capacity to operate, and the risk they always sit with for being that large.

Mark.