nanog mailing list archives
Re: massive facebook outage presently
From: PJ Capelli via NANOG <nanog () nanog org>
Date: Mon, 04 Oct 2021 20:16:41 +0000
Seems unlikely that FB internal controls would allow such a backdoor ... "Never to get lost, is not living" - Rebecca Solnit Sent with ProtonMail Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Monday, October 4th, 2021 at 4:12 PM, Baldur Norddahl <baldur.norddahl () gmail com> wrote:
On Mon, 4 Oct 2021 at 21:58, Michael Thomas <mike () mtcc com> wrote:
On 10/4/21 11:48 AM, Luke Guillory wrote:
I believe the original change was 'automatic' (as in configuration done via a web interface). However, now that connection to the outside world is down, remote access to those tools don't exist anymore, so the emergency procedure is to gain physical access to the peering routers and do all the configuration locally.
Assuming that this is what actually happened, what should fb have done different (beyond the obvious of not screwing up the immediate issue)? This seems like it's a single point of failure. Should all of the BGP speakers have been dual homed or something like that? Or should they not have been mixing ops and production networks? Sorry if this sounds dumb.
Facebook is a huge network. It is doubtful that what is going on is this simple. So I will make no guesses to what Facebook is or should be doing.
However the traditional way for us small timers is to have a backdoor using someone else's network. Nowadays this could be a simple 4/5G router with a VPN, to a terminal server that allows the operator to configure the equipment through the monitor port even when the config is completely destroyed.
Regards,
Baldur
Attachment:
signature.asc
Description: OpenPGP digital signature
Current thread:
- RE: massive facebook outage presently, (continued)
- RE: massive facebook outage presently Tony Wicks (Oct 04)
- Re: massive facebook outage presently Jason Kuehl (Oct 04)
- Re: massive facebook outage presently Hank Nussbacher (Oct 04)
- Re: massive facebook outage presently David Andrzejewski (Oct 05)
- RE: massive facebook outage presently Luke Guillory (Oct 04)
- Re: massive facebook outage presently Michael Thomas (Oct 04)
- Re: massive facebook outage presently Baldur Norddahl (Oct 04)
- Re: massive facebook outage presently bzs (Oct 04)
- Re: massive facebook outage presently Niels Bakker (Oct 04)
- Re: massive facebook outage presently Doug McIntyre (Oct 04)
- Re: massive facebook outage presently PJ Capelli via NANOG (Oct 05)
- Re: massive facebook outage presently Baldur Norddahl (Oct 04)
- Re: massive facebook outage presently Blake Dunlap (Oct 04)
- Re: massive facebook outage presently Mel Beckman (Oct 04)
- Re: massive facebook outage presently Mark Tinka (Oct 04)
- Re: massive facebook outage presently Łukasz Bromirski (Oct 04)
- Re: massive facebook outage presently Mark Tinka (Oct 04)
- Re: massive facebook outage presently bzs (Oct 04)
- RE: massive facebook outage presently Tony Wicks (Oct 04)
- Re: massive facebook outage presently Mark Tinka (Oct 04)
- Re: massive facebook outage presently Karl Auer (Oct 04)