[talk] FB outage

[Jim Thompson]

Rumor has it that even the card/badge access systems are down.
https://twitter.com/sheeraf/status/1445099150316503057 <https://twitter.com/sheeraf/status/1445099150316503057>

So getting someone in-front of a console is … challenging.

> On Oct 4, 2021, at 3:28 PM, Warner Losh <imp at bsdimp.com> wrote:
> 
> The rumors from my side. I've not verified them all (some are in the twitter feed). We'll see how reliable this is...
> 
> But, Facebook does all their own tooling, and it's all down.
> Even internal chat. So they are talking directly via cell phones (system to lookup cell numbers, also down)
> The only way that can provision the bgp routes is via the console of the routers at this point...
> ... but the internal system to get credentials is also down.
> They also need to get someone in front of the console to bring things back online...
> Their tools to coordinate and meet to come up with a plan, also down....
> 
> Single point of failure in the extreme.
> 
> 
> On Mon, Oct 4, 2021 at 3:20 PM Michael W. Lucas <mwlucas at michaelwlucas.com <mailto:mwlucas at michaelwlucas.com>> wrote:
> 
> <blink> <blink>
> 
> That's it. Twitter is the next to go.
> 
> ==ml
> 
> 
> On Mon, Oct 04, 2021 at 04:15:39PM -0400, George Rosamond wrote:
> > One of the cooler things we can all learn from the Facebook/Instagram/WA 
> > outage is some technical innovations most of you probably missed.
> > 
> > https://nitter.eu/briankrebs/status/1445077617426718725 <https://www.google.com/url?q=https://nitter.eu/briankrebs/status/1445077617426718725&source=gmail-imap&ust=1633987767000000&usg=AOvVaw2L-CBPvos_qEOOF3z861Wt>
> > 
> > Apparently, there are no RFCs for it, but DNS Global Routing Tables is 
> > the place to be. Move over blockchain.
> > 
> > g
> > 
> > _______________________________________________
> > talk mailing list
> > talk at lists.nycbug.org <mailto:talk at lists.nycbug.org>
> > http://lists.nycbug.org:8080/mailman/listinfo/talk <https://www.google.com/url?q=http://lists.nycbug.org:8080/mailman/listinfo/talk&source=gmail-imap&ust=1633987767000000&usg=AOvVaw0m3zHyt7BS3Bgj3MP_B1en>
> 
> -- 
> Michael W. Lucas        https://mwl.io/ <https://www.google.com/url?q=https://mwl.io/&source=gmail-imap&ust=1633987767000000&usg=AOvVaw2LRnwaP3D1G-qW7qhAZpeb>
> author of: Absolute OpenBSD, SSH Mastery, git commit murder,
>  Absolute FreeBSD, Immortal Clay, Prohibition Orcs, etc, etc, etc...
> ### New books: TLS Mastery, the Networknomicon, $ git sync murder ###
> 
> _______________________________________________
> talk mailing list
> talk at lists.nycbug.org <mailto:talk at lists.nycbug.org>
> http://lists.nycbug.org:8080/mailman/listinfo/talk <https://www.google.com/url?q=http://lists.nycbug.org:8080/mailman/listinfo/talk&source=gmail-imap&ust=1633987767000000&usg=AOvVaw0m3zHyt7BS3Bgj3MP_B1en>
> _______________________________________________
> talk mailing list
> talk at lists.nycbug.org
> https://www.google.com/url?q=http://lists.nycbug.org:8080/mailman/listinfo/talk&source=gmail-imap&ust=1633987767000000&usg=AOvVaw0m3zHyt7BS3Bgj3MP_B1en

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nycbug.org/pipermail/talk/attachments/20211004/02ab1569/attachment.htm>