• Hi guest! As you can see, the new Wizard Forums has been revived, and we are glad to have you visiting our site! However, it would be really helpful, both to you and us, if you registered on our website! Registering allows you to see all posts, and make posts yourself, which would be great if you could share your knowledge and opinions with us! You could also make posts to ask questions!

Facebook + Instagram + WhatsApp are down

Irish Bard

Neophyte
Joined
Sep 16, 2021
Messages
49
Next question - how long does the outage need to be before there's a genuine impact?


What would it take to make that damn nest of vipers lose victims?
 

Scottish_Pride

Meme-y Tree Nymph
Joined
Apr 17, 2021
Messages
410
Awards
8
I wonder how productive lots of people are gonna suddenly become now that their main source of goofing off at the office is down. Or just goofing off in general, like it'd be hilarious if a bunch of people come out of this having read all of War&Peace, learned a whole new language or some shit like that to fill the void.
 

SkullTraill

Glorious Light of Knowledge and Power
Staff member
Custodian
Librarian
Joined
Apr 12, 2021
Messages
479
Awards
12
For anyone technical, here's a very concise summary of what happened:

Some configuration change lead to the removal of the BGP route to facebook's backbone router with all of their DNS and stuff.

Every single facebook server was therefore disconnected from the internet. And I mean every single one. To the point that offsite experts could not even enter the site's secure area because the ID/security systems were inaccessible so even the doors to the secure areas had to be "manually overriden"

Please, Log in or Register to view URLs content!
Please, Log in or Register to view URLs content!
Please, Log in or Register to view URLs content!
 

Incognitus

Lead Transcriber
Joined
May 30, 2021
Messages
73
Awards
5
It's amazing how such a large billion dollar company can be so amateurish. All of that stuff was avoidable. Some people are losing their jobs, I'm sure.
 

Scottish_Pride

Meme-y Tree Nymph
Joined
Apr 17, 2021
Messages
410
Awards
8
It's amazing how such a large billion dollar company can be so amateurish. All of that stuff was avoidable. Some people are losing their jobs, I'm sure.
Reminds me of the Suez Canal incident, where the captain drew a dick in the ocean with the course of the ship beforehand.

Like any big corporation, though, they've possibly cut a corner or two with their setup in order to save a quick buck. At least I wouldn't be surprised if that were the case.
 

Mider2009

Zealot
Joined
Aug 29, 2021
Messages
119
Awards
3
Next question - how long does the outage need to be before there's a genuine impact?


What would it take to make that damn nest of vipers lose victims?
They wouldn’t allow that IMO, as a public company they would lose billions

I bet they closed things down on purpose ever since they had that whistleblower go public.
 

Incognitus

Lead Transcriber
Joined
May 30, 2021
Messages
73
Awards
5
I've been a professional System/network admin since the mid-1990's. I'm very familiar with IT incompetence in large organizations.

I doubt they're sharing all the details, but yes it makes sense. It was a very avoidable, but completely realistic issue. I don't think the average Internet use has any idea how tenuous everything is. IT departments never get the funding they need, and the larger the organization, the more incompetent people they have.

I just recently moved our companies data center, which involved making BGP changes. Cisco gear (IOS) has commands such as "reload in #", which will do an automatic router (switch, firewall, etc) reboot if the reload isn't stopped. This is a great way to reset the router to last known good config if what you enter cuts you off or does something else you're not expecting. Higher end Cisco routers have even more config management that allow automatic rollback of config commands if you don't perform some action (commit the config, etc). Most other router brands have a similar thing.

And that's just one example of how to easily avoid a multi-hour outage. The first would be to make sure the config you're entering is actually correct!

So no conspiracy. Just incompetence. This isn't even unheard of. Incorrect BGP config happens quite often. It just doesn't always hit the news like a Facebook outage.
 
Top