[talk] FB outage

[steve.b at osfda.org]

Akami, Google, DoD -they all have had serious bgp outages due to config 
mistakes.

IF that's what it was...

On 10/4/21 7:35 PM, steve.b at osfda.org wrote:
>
> CTO recently left.
>
> If it's bgp, happens to all of them...
>
> On 10/4/21 5:28 PM, Warner Losh wrote:
>> The rumors from my side. I've not verified them all (some are in the 
>> twitter feed). We'll see how reliable this is...
>>
>> But, Facebook does all their own tooling, and it's all down.
>> Even internal chat. So they are talking directly via cell phones 
>> (system to lookup cell numbers, also down)
>> The only way that can provision the bgp routes is via the console of 
>> the routers at this point...
>> ... but the internal system to get credentials is also down.
>> They also need to get someone in front of the console to bring things 
>> back online...
>> Their tools to coordinate and meet to come up with a plan, also down....
>>
>> Single point of failure in the extreme.
>>
>>
>> On Mon, Oct 4, 2021 at 3:20 PM Michael W. Lucas 
>> <mwlucas at michaelwlucas.com <mailto:mwlucas at michaelwlucas.com>> wrote:
>>
>>
>>     <blink> <blink>
>>
>>     That's it. Twitter is the next to go.
>>
>>     ==ml
>>
>>
>>     On Mon, Oct 04, 2021 at 04:15:39PM -0400, George Rosamond wrote:
>>     > One of the cooler things we can all learn from the
>>     Facebook/Instagram/WA
>>     > outage is some technical innovations most of you probably missed.
>>     >
>>     > https://nitter.eu/briankrebs/status/1445077617426718725
>>     <https://nitter.eu/briankrebs/status/1445077617426718725>
>>     >
>>     > Apparently, there are no RFCs for it, but DNS Global Routing
>>     Tables is
>>     > the place to be. Move over blockchain.
>>     >
>>     > g
>>     >
>>     > _______________________________________________
>>     > talk mailing list
>>     > talk at lists.nycbug.org <mailto:talk at lists.nycbug.org>
>>     > http://lists.nycbug.org:8080/mailman/listinfo/talk
>>     <http://lists.nycbug.org:8080/mailman/listinfo/talk>
>>
>>     -- 
>>     Michael W. Lucas https://mwl.io/ <https://mwl.io/>
>>     author of: Absolute OpenBSD, SSH Mastery, git commit murder,
>>      Absolute FreeBSD, Immortal Clay, Prohibition Orcs, etc, etc, etc...
>>     ### New books: TLS Mastery, the Networknomicon, $ git sync murder ###
>>
>>     _______________________________________________
>>     talk mailing list
>>     talk at lists.nycbug.org <mailto:talk at lists.nycbug.org>
>>     http://lists.nycbug.org:8080/mailman/listinfo/talk
>>     <http://lists.nycbug.org:8080/mailman/listinfo/talk>
>>
>>
>> _______________________________________________
>> talk mailing list
>> talk at lists.nycbug.org
>> http://lists.nycbug.org:8080/mailman/listinfo/talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nycbug.org/pipermail/talk/attachments/20211004/9d923cf7/attachment.htm>