Re: massive facebook outage presently

[Mel Beckman <mel () beckman org>]

I’m not the only one who finds this timing suspicious, Starting with the publishers of 60 Minutes themselves :-)

CBS:

The outage comes the morning after "60 Minutes" aired an interview with a whistleblower who said Facebook is aware of 
how it amplifies hate, misinformation and unrest but claimed the company hides what it knows.

https://www.cbsnews.com/news/facebook-instagram-whatsapp-down-2021-10-04/

https://abcnews.go.com/Technology/facebook-instagram-users-us/story?id=80397437

https://www.cnbc.com/2021/10/04/facebook-shares-drop-5percent-after-site-outage-and-whistleblower-interview.html

https://www.insidenova.com/headlines/facebook-instagram-down-after-60-minutes-whistleblower-report/article_29977530-2531-11ec-b5ae-73311c46edfb.html

https://adage.com/article/digital-marketing-ad-tech-news/what-facebook-telling-advertisers-about-60-minutes-whistleblower/2370346




 -mel beckman

On Oct 4, 2021, at 1:36 PM, Blake Dunlap <ikiris () gmail com> wrote:


If there isn't an undernetwork capable of being backdoored with the proper keys (I'd be shocked if there isn't - the 
big players have very good infra and DR people), I suspect there will be one soonish.

It doesnt do much good to have DR plans and keys otherwise if you can't even get to the locks without getting on a 
plane.

Regardless of how people may feel about the company, I just feel bad for their sres right now and am drinking one in 
their honor. I just want to know what an October meltdown gets called in the pm.

On Mon, Oct 4, 2021, 15:24 Baldur Norddahl <baldur.norddahl () gmail com<mailto:baldur.norddahl () gmail com>> wrote:
Not in such a primitive fashion no. But they could definitely have a secondary network that will continue to work even 
if something goes wrong with the primary.

On Mon, 4 Oct 2021 at 22:16, PJ Capelli <pjcapelli () pm me<mailto:pjcapelli () pm me>> wrote:
Seems unlikely that FB internal controls would allow such a backdoor ...

"Never to get lost, is not living" - Rebecca Solnit

Sent with ProtonMail<https://protonmail.com/> Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday, October 4th, 2021 at 4:12 PM, Baldur Norddahl <baldur.norddahl () gmail com<mailto:baldur.norddahl () gmail 
com>> wrote:


On Mon, 4 Oct 2021 at 21:58, Michael Thomas <mike () mtcc com<mailto:mike () mtcc com>> wrote:


On 10/4/21 11:48 AM, Luke Guillory wrote:


I believe the original change was 'automatic' (as in configuration done via a web interface). However, now that 
connection to the outside world is down, remote access to those tools don't exist anymore, so the emergency procedure 
is to gain physical access to the peering routers and do all the configuration locally.

Assuming that this is what actually happened, what should fb have done different (beyond the obvious of not screwing up 
the immediate issue)? This seems like it's a single point of failure. Should all of the BGP speakers have been dual 
homed or something like that? Or should they not have been mixing ops and production networks? Sorry if this sounds 
dumb.

Facebook is a huge network. It is doubtful that what is going on is this simple. So I will make no guesses to what 
Facebook is or should be doing.

However the traditional way for us small timers is to have a backdoor using someone else's network. Nowadays this could 
be a simple 4/5G router with a VPN, to a terminal server that allows the operator to configure the equipment through 
the monitor port even when the config is completely destroyed.

Regards,

Baldur