nanog mailing list archives

Re: massive facebook outage presently


From: Matthew Petach <mpetach () netflight com>
Date: Mon, 4 Oct 2021 09:51:57 -0700

On Mon, Oct 4, 2021 at 9:47 AM Jason Kuehl <jason.w.kuehl () gmail com> wrote:

Looks like they run there own nameservers and I see the soa records are
even missing.

On Mon, Oct 4, 2021, 12:23 PM Mel Beckman <mel () beckman org> wrote:

Here’s a screenshot:



 -mel beckman

On Oct 4, 2021, at 9:06 AM, Eric Kuhnke <eric.kuhnke () gmail com> wrote:


https://downdetector.com/status/facebook/

Normally not worth mentioning random $service having an outage here, but
this will undoubtedly generate a large volume of customer service calls.

Appears to be failure in DNS resolution.


If you check your BGP routing tables, you'll probably
find that it's not so much the SOA records that are
missing, as it is the prefixes to reach the DNS servers
entirely.

I suspect the DNS entries on the servers themselves
may look fine from inside facebook, leading to a slower
diagnostic and repair, as it's only from the outside world
the missing routing entries in the global table make the
problem so painfully visible.

Having the DNS team frantically checking their servers
may slow the resolution down, if it is indeed a BGP failure
rather than a DNS server failure situation, as it seems to
appear at the moment.  ^_^;

Matt

Current thread: