From george at ceetonetechnology.com  Mon Oct  4 16:15:39 2021
From: george at ceetonetechnology.com (George Rosamond)
Date: Mon, 4 Oct 2021 16:15:39 -0400
Subject: [talk] FB outage
Message-ID: <53100307-c10a-8ee7-3286-2d9604f89061@ceetonetechnology.com>

One of the cooler things we can all learn from the Facebook/Instagram/WA 
outage is some technical innovations most of you probably missed.

https://nitter.eu/briankrebs/status/1445077617426718725

Apparently, there are no RFCs for it, but DNS Global Routing Tables is 
the place to be. Move over blockchain.

g



From mwlucas at michaelwlucas.com  Mon Oct  4 17:17:59 2021
From: mwlucas at michaelwlucas.com (Michael W. Lucas)
Date: Mon, 4 Oct 2021 17:17:59 -0400
Subject: [talk] FB outage
In-Reply-To: <53100307-c10a-8ee7-3286-2d9604f89061@ceetonetechnology.com>
References: <53100307-c10a-8ee7-3286-2d9604f89061@ceetonetechnology.com>
Message-ID: <YVtvh+SZHd4xbsP2@mail.mwl.io>


<blink> <blink>

That's it. Twitter is the next to go.

==ml


On Mon, Oct 04, 2021 at 04:15:39PM -0400, George Rosamond wrote:
> One of the cooler things we can all learn from the Facebook/Instagram/WA 
> outage is some technical innovations most of you probably missed.
> 
> https://nitter.eu/briankrebs/status/1445077617426718725
> 
> Apparently, there are no RFCs for it, but DNS Global Routing Tables is 
> the place to be. Move over blockchain.
> 
> g
> 
> _______________________________________________
> talk mailing list
> talk at lists.nycbug.org
> http://lists.nycbug.org:8080/mailman/listinfo/talk

-- 
Michael W. Lucas 	https://mwl.io/
author of: Absolute OpenBSD, SSH Mastery, git commit murder,
 Absolute FreeBSD, Immortal Clay, Prohibition Orcs, etc, etc, etc...
### New books: TLS Mastery, the Networknomicon, $ git sync murder ###



From imp at bsdimp.com  Mon Oct  4 17:28:13 2021
From: imp at bsdimp.com (Warner Losh)
Date: Mon, 4 Oct 2021 15:28:13 -0600
Subject: [talk] FB outage
In-Reply-To: <YVtvh+SZHd4xbsP2@mail.mwl.io>
References: <53100307-c10a-8ee7-3286-2d9604f89061@ceetonetechnology.com>
 <YVtvh+SZHd4xbsP2@mail.mwl.io>
Message-ID: <CANCZdfp6SFsSE61R6MX=bxHyFjgg6dYzwS7SuOyFfJJegOg7Xw@mail.gmail.com>

The rumors from my side. I've not verified them all (some are in the
twitter feed). We'll see how reliable this is...

But, Facebook does all their own tooling, and it's all down.
Even internal chat. So they are talking directly via cell phones (system to
lookup cell numbers, also down)
The only way that can provision the bgp routes is via the console of the
routers at this point...
... but the internal system to get credentials is also down.
They also need to get someone in front of the console to bring things back
online...
Their tools to coordinate and meet to come up with a plan, also down....

Single point of failure in the extreme.


On Mon, Oct 4, 2021 at 3:20 PM Michael W. Lucas <mwlucas at michaelwlucas.com>
wrote:

>
> <blink> <blink>
>
> That's it. Twitter is the next to go.
>
> ==ml
>
>
> On Mon, Oct 04, 2021 at 04:15:39PM -0400, George Rosamond wrote:
> > One of the cooler things we can all learn from the Facebook/Instagram/WA
> > outage is some technical innovations most of you probably missed.
> >
> > https://nitter.eu/briankrebs/status/1445077617426718725
> >
> > Apparently, there are no RFCs for it, but DNS Global Routing Tables is
> > the place to be. Move over blockchain.
> >
> > g
> >
> > _______________________________________________
> > talk mailing list
> > talk at lists.nycbug.org
> > http://lists.nycbug.org:8080/mailman/listinfo/talk
>
> --
> Michael W. Lucas        https://mwl.io/
> author of: Absolute OpenBSD, SSH Mastery, git commit murder,
>  Absolute FreeBSD, Immortal Clay, Prohibition Orcs, etc, etc, etc...
> ### New books: TLS Mastery, the Networknomicon, $ git sync murder ###
>
> _______________________________________________
> talk mailing list
> talk at lists.nycbug.org
> http://lists.nycbug.org:8080/mailman/listinfo/talk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.nycbug.org:8443/pipermail/talk/attachments/20211004/c1bde62e/attachment.htm>

From jim at netgate.com  Mon Oct  4 17:35:48 2021
From: jim at netgate.com (Jim Thompson)
Date: Mon, 4 Oct 2021 15:35:48 -0600
Subject: [talk] FB outage
In-Reply-To: <CANCZdfp6SFsSE61R6MX=bxHyFjgg6dYzwS7SuOyFfJJegOg7Xw@mail.gmail.com>
References: <53100307-c10a-8ee7-3286-2d9604f89061@ceetonetechnology.com>
 <YVtvh+SZHd4xbsP2@mail.mwl.io>
 <CANCZdfp6SFsSE61R6MX=bxHyFjgg6dYzwS7SuOyFfJJegOg7Xw@mail.gmail.com>
Message-ID: <29668ECA-820D-4271-819E-47F63B5CE474@netgate.com>

Rumor has it that even the card/badge access systems are down.
https://twitter.com/sheeraf/status/1445099150316503057 <https://twitter.com/sheeraf/status/1445099150316503057>

So getting someone in-front of a console is ? challenging.

> On Oct 4, 2021, at 3:28 PM, Warner Losh <imp at bsdimp.com> wrote:
> 
> The rumors from my side. I've not verified them all (some are in the twitter feed). We'll see how reliable this is...
> 
> But, Facebook does all their own tooling, and it's all down.
> Even internal chat. So they are talking directly via cell phones (system to lookup cell numbers, also down)
> The only way that can provision the bgp routes is via the console of the routers at this point...
> ... but the internal system to get credentials is also down.
> They also need to get someone in front of the console to bring things back online...
> Their tools to coordinate and meet to come up with a plan, also down....
> 
> Single point of failure in the extreme.
> 
> 
> On Mon, Oct 4, 2021 at 3:20 PM Michael W. Lucas <mwlucas at michaelwlucas.com <mailto:mwlucas at michaelwlucas.com>> wrote:
> 
> <blink> <blink>
> 
> That's it. Twitter is the next to go.
> 
> ==ml
> 
> 
> On Mon, Oct 04, 2021 at 04:15:39PM -0400, George Rosamond wrote:
> > One of the cooler things we can all learn from the Facebook/Instagram/WA 
> > outage is some technical innovations most of you probably missed.
> > 
> > https://nitter.eu/briankrebs/status/1445077617426718725 <https://www.google.com/url?q=https://nitter.eu/briankrebs/status/1445077617426718725&source=gmail-imap&ust=1633987767000000&usg=AOvVaw2L-CBPvos_qEOOF3z861Wt>
> > 
> > Apparently, there are no RFCs for it, but DNS Global Routing Tables is 
> > the place to be. Move over blockchain.
> > 
> > g
> > 
> > _______________________________________________
> > talk mailing list
> > talk at lists.nycbug.org <mailto:talk at lists.nycbug.org>
> > http://lists.nycbug.org:8080/mailman/listinfo/talk <https://www.google.com/url?q=http://lists.nycbug.org:8080/mailman/listinfo/talk&source=gmail-imap&ust=1633987767000000&usg=AOvVaw0m3zHyt7BS3Bgj3MP_B1en>
> 
> -- 
> Michael W. Lucas        https://mwl.io/ <https://www.google.com/url?q=https://mwl.io/&source=gmail-imap&ust=1633987767000000&usg=AOvVaw2LRnwaP3D1G-qW7qhAZpeb>
> author of: Absolute OpenBSD, SSH Mastery, git commit murder,
>  Absolute FreeBSD, Immortal Clay, Prohibition Orcs, etc, etc, etc...
> ### New books: TLS Mastery, the Networknomicon, $ git sync murder ###
> 
> _______________________________________________
> talk mailing list
> talk at lists.nycbug.org <mailto:talk at lists.nycbug.org>
> http://lists.nycbug.org:8080/mailman/listinfo/talk <https://www.google.com/url?q=http://lists.nycbug.org:8080/mailman/listinfo/talk&source=gmail-imap&ust=1633987767000000&usg=AOvVaw0m3zHyt7BS3Bgj3MP_B1en>
> _______________________________________________
> talk mailing list
> talk at lists.nycbug.org
> https://www.google.com/url?q=http://lists.nycbug.org:8080/mailman/listinfo/talk&source=gmail-imap&ust=1633987767000000&usg=AOvVaw0m3zHyt7BS3Bgj3MP_B1en

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.nycbug.org:8443/pipermail/talk/attachments/20211004/02ab1569/attachment.htm>

From pete at nomadlogic.org  Mon Oct  4 17:38:35 2021
From: pete at nomadlogic.org (Pete Wright)
Date: Mon, 4 Oct 2021 14:38:35 -0700
Subject: [talk] FB outage
In-Reply-To: <29668ECA-820D-4271-819E-47F63B5CE474@netgate.com>
References: <53100307-c10a-8ee7-3286-2d9604f89061@ceetonetechnology.com>
 <YVtvh+SZHd4xbsP2@mail.mwl.io>
 <CANCZdfp6SFsSE61R6MX=bxHyFjgg6dYzwS7SuOyFfJJegOg7Xw@mail.gmail.com>
 <29668ECA-820D-4271-819E-47F63B5CE474@netgate.com>
Message-ID: <87f8c354-2671-a12f-d0b8-0675e4b5c018@nomadlogic.org>



On 10/4/21 2:35 PM, Jim Thompson wrote:
> Rumor has it that even the card/badge access systems are down.
> https://twitter.com/sheeraf/status/1445099150316503057 
> <https://twitter.com/sheeraf/status/1445099150316503057>
>
> So getting someone in-front of a console is ? challenging.

it'll be interesting to see how many people need to add *this* to their 
tabletop exercises this year!

-p

-- 
Pete Wright
pete at nomadlogic.org
@nomadlogicLA



From jondrews at fastmail.com  Mon Oct  4 19:31:20 2021
From: jondrews at fastmail.com (Jonathan Drews)
Date: Mon, 04 Oct 2021 17:31:20 -0600
Subject: [talk] FB outage
In-Reply-To: <53100307-c10a-8ee7-3286-2d9604f89061@ceetonetechnology.com>
References: <53100307-c10a-8ee7-3286-2d9604f89061@ceetonetechnology.com>
Message-ID: <3248365f-b394-4c28-86a2-27d6515abcd8@www.fastmail.com>

"Maybe Zuck got mad that he kept getting roasted for running a junk company 
and he rage quit the internet lol"
A quote from the link you provided George.


On Mon, Oct 4, 2021, at 14:15, George Rosamond wrote:
> One of the cooler things we can all learn from the Facebook/Instagram/WA 
> outage is some technical innovations most of you probably missed.
> 
> https://nitter.eu/briankrebs/status/1445077617426718725
> 
> Apparently, there are no RFCs for it, but DNS Global Routing Tables is 
> the place to be. Move over blockchain.
> 
> g
> 
> _______________________________________________
> talk mailing list
> talk at lists.nycbug.org
> http://lists.nycbug.org:8080/mailman/listinfo/talk
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.nycbug.org:8443/pipermail/talk/attachments/20211004/8b1e6035/attachment.htm>

From steve.b at osfda.org  Mon Oct  4 19:35:30 2021
From: steve.b at osfda.org (steve.b at osfda.org)
Date: Mon, 4 Oct 2021 19:35:30 -0400
Subject: [talk] FB outage
In-Reply-To: <CANCZdfp6SFsSE61R6MX=bxHyFjgg6dYzwS7SuOyFfJJegOg7Xw@mail.gmail.com>
References: <53100307-c10a-8ee7-3286-2d9604f89061@ceetonetechnology.com>
 <YVtvh+SZHd4xbsP2@mail.mwl.io>
 <CANCZdfp6SFsSE61R6MX=bxHyFjgg6dYzwS7SuOyFfJJegOg7Xw@mail.gmail.com>
Message-ID: <324664c1-f961-0d46-5538-47e854a4ad8b@osfda.org>

CTO recently left.

If it's bgp, happens to all of them...

On 10/4/21 5:28 PM, Warner Losh wrote:
> The rumors from my side. I've not verified them all (some are in the 
> twitter feed). We'll see how reliable this is...
>
> But, Facebook does all their own tooling, and it's all down.
> Even internal chat. So they are talking directly via cell phones 
> (system to lookup cell numbers, also down)
> The only way that can provision the bgp routes is via the console of 
> the routers at this point...
> ... but the internal system to get credentials?is also down.
> They also need to get someone in front of the console to bring things 
> back online...
> Their tools to coordinate and meet to come up with a plan, also down....
>
> Single point of failure in the extreme.
>
>
> On Mon, Oct 4, 2021 at 3:20 PM Michael W. Lucas 
> <mwlucas at michaelwlucas.com <mailto:mwlucas at michaelwlucas.com>> wrote:
>
>
>     <blink> <blink>
>
>     That's it. Twitter is the next to go.
>
>     ==ml
>
>
>     On Mon, Oct 04, 2021 at 04:15:39PM -0400, George Rosamond wrote:
>     > One of the cooler things we can all learn from the
>     Facebook/Instagram/WA
>     > outage is some technical innovations most of you probably missed.
>     >
>     > https://nitter.eu/briankrebs/status/1445077617426718725
>     <https://nitter.eu/briankrebs/status/1445077617426718725>
>     >
>     > Apparently, there are no RFCs for it, but DNS Global Routing
>     Tables is
>     > the place to be. Move over blockchain.
>     >
>     > g
>     >
>     > _______________________________________________
>     > talk mailing list
>     > talk at lists.nycbug.org <mailto:talk at lists.nycbug.org>
>     > http://lists.nycbug.org:8080/mailman/listinfo/talk
>     <http://lists.nycbug.org:8080/mailman/listinfo/talk>
>
>     -- 
>     Michael W. Lucas https://mwl.io/ <https://mwl.io/>
>     author of: Absolute OpenBSD, SSH Mastery, git commit murder,
>     ?Absolute FreeBSD, Immortal Clay, Prohibition Orcs, etc, etc, etc...
>     ### New books: TLS Mastery, the Networknomicon, $ git sync murder ###
>
>     _______________________________________________
>     talk mailing list
>     talk at lists.nycbug.org <mailto:talk at lists.nycbug.org>
>     http://lists.nycbug.org:8080/mailman/listinfo/talk
>     <http://lists.nycbug.org:8080/mailman/listinfo/talk>
>
>
> _______________________________________________
> talk mailing list
> talk at lists.nycbug.org
> http://lists.nycbug.org:8080/mailman/listinfo/talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.nycbug.org:8443/pipermail/talk/attachments/20211004/133f87d3/attachment.htm>

From george at ceetonetechnology.com  Mon Oct  4 19:44:31 2021
From: george at ceetonetechnology.com (George Rosamond)
Date: Mon, 4 Oct 2021 19:44:31 -0400
Subject: [talk] FB outage
In-Reply-To: <324664c1-f961-0d46-5538-47e854a4ad8b@osfda.org>
References: <53100307-c10a-8ee7-3286-2d9604f89061@ceetonetechnology.com>
 <YVtvh+SZHd4xbsP2@mail.mwl.io>
 <CANCZdfp6SFsSE61R6MX=bxHyFjgg6dYzwS7SuOyFfJJegOg7Xw@mail.gmail.com>
 <324664c1-f961-0d46-5538-47e854a4ad8b@osfda.org>
Message-ID: <3876bc93-e0b0-3c7f-b429-a0bc4dc43103@ceetonetechnology.com>

On 10/4/21 19:35, steve.b at osfda.org wrote:
> CTO recently left.
> 
> If it's bgp, happens to all of them...

DNS was the first diagnois, but I tested the .onion site over Tor (ie 
"dark web") and once it wasn't accessible it became clear it wasn't DNS.

I have no idea about the procedures and processes on editing BGP routing 
tables at a FB or wherever, but it seems a little crazy that you can 
have a single-point of failure like one wily/confused/hung-over Yankee 
fan could fat finger with an impact like that. Not necessarily 
literally, but you get my point...

Jan noted in a tweet that over 12000 domains used the Facebook DNS as auth.

Am I the only one who was distracted by reading and chatting about it 
all day?

g



From pvarga at pvrg.net  Mon Oct  4 19:46:37 2021
From: pvarga at pvrg.net (Peter Varga)
Date: Mon, 04 Oct 2021 16:46:37 -0700
Subject: [talk] FB outage
In-Reply-To: <324664c1-f961-0d46-5538-47e854a4ad8b@osfda.org>
References: <53100307-c10a-8ee7-3286-2d9604f89061@ceetonetechnology.com>
 <YVtvh+SZHd4xbsP2@mail.mwl.io>
 <CANCZdfp6SFsSE61R6MX=bxHyFjgg6dYzwS7SuOyFfJJegOg7Xw@mail.gmail.com>
 <324664c1-f961-0d46-5538-47e854a4ad8b@osfda.org>
Message-ID: <822c7b00-5d73-4a76-a390-f370f873e5fa@www.fastmail.com>

It is bgp also

On Mon, Oct 4, 2021, at 16:35, steve.b at osfda.org wrote:
> CTO recently left.
> 
> If it's bgp, happens to all of them...
> 
> On 10/4/21 5:28 PM, Warner Losh wrote:
>> The rumors from my side. I've not verified them all (some are in the twitter feed). We'll see how reliable this is... 
>> 
>> But, Facebook does all their own tooling, and it's all down. 
>> Even internal chat. So they are talking directly via cell phones (system to lookup cell numbers, also down)
>> The only way that can provision the bgp routes is via the console of the routers at this point...
>> ... but the internal system to get credentials is also down.
>> They also need to get someone in front of the console to bring things back online...
>> Their tools to coordinate and meet to come up with a plan, also down....
>> 
>> Single point of failure in the extreme.
>> 
>> 
>> On Mon, Oct 4, 2021 at 3:20 PM Michael W. Lucas <mwlucas at michaelwlucas.com> wrote:
>>> 
>>> <blink> <blink>
>>> 
>>> That's it. Twitter is the next to go.
>>> 
>>> ==ml
>>> 
>>> 
>>> On Mon, Oct 04, 2021 at 04:15:39PM -0400, George Rosamond wrote:
>>> > One of the cooler things we can all learn from the Facebook/Instagram/WA 
>>> > outage is some technical innovations most of you probably missed.
>>> > 
>>> > https://nitter.eu/briankrebs/status/1445077617426718725
>>> > 
>>> > Apparently, there are no RFCs for it, but DNS Global Routing Tables is 
>>> > the place to be. Move over blockchain.
>>> > 
>>> > g
>>> > 
>>> > _______________________________________________
>>> > talk mailing list
>>> > talk at lists.nycbug.org
>>> > http://lists.nycbug.org:8080/mailman/listinfo/talk
>>> 
>>> -- 
>>> Michael W. Lucas        https://mwl.io/
>>> author of: Absolute OpenBSD, SSH Mastery, git commit murder,
>>>  Absolute FreeBSD, Immortal Clay, Prohibition Orcs, etc, etc, etc...
>>> ### New books: TLS Mastery, the Networknomicon, $ git sync murder ###
>>> 
>>> _______________________________________________
>>> talk mailing list
>>> talk at lists.nycbug.org
>>> http://lists.nycbug.org:8080/mailman/listinfo/talk
>> 
>> _______________________________________________
>> talk mailing list
>> talk at lists.nycbug.org
>> http://lists.nycbug.org:8080/mailman/listinfo/talk
>> 
> _______________________________________________
> talk mailing list
> talk at lists.nycbug.org
> http://lists.nycbug.org:8080/mailman/listinfo/talk
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.nycbug.org:8443/pipermail/talk/attachments/20211004/11d73954/attachment.htm>

From steve.b at osfda.org  Mon Oct  4 19:52:11 2021
From: steve.b at osfda.org (steve.b at osfda.org)
Date: Mon, 4 Oct 2021 19:52:11 -0400
Subject: [talk] FB outage
In-Reply-To: <324664c1-f961-0d46-5538-47e854a4ad8b@osfda.org>
References: <53100307-c10a-8ee7-3286-2d9604f89061@ceetonetechnology.com>
 <YVtvh+SZHd4xbsP2@mail.mwl.io>
 <CANCZdfp6SFsSE61R6MX=bxHyFjgg6dYzwS7SuOyFfJJegOg7Xw@mail.gmail.com>
 <324664c1-f961-0d46-5538-47e854a4ad8b@osfda.org>
Message-ID: <7fe1def8-1c07-2815-2efc-dbae226cb281@osfda.org>

Akami, Google, DoD -they all have had serious bgp outages due to config 
mistakes.

IF that's what it was...

On 10/4/21 7:35 PM, steve.b at osfda.org wrote:
>
> CTO recently left.
>
> If it's bgp, happens to all of them...
>
> On 10/4/21 5:28 PM, Warner Losh wrote:
>> The rumors from my side. I've not verified them all (some are in the 
>> twitter feed). We'll see how reliable this is...
>>
>> But, Facebook does all their own tooling, and it's all down.
>> Even internal chat. So they are talking directly via cell phones 
>> (system to lookup cell numbers, also down)
>> The only way that can provision the bgp routes is via the console of 
>> the routers at this point...
>> ... but the internal system to get credentials?is also down.
>> They also need to get someone in front of the console to bring things 
>> back online...
>> Their tools to coordinate and meet to come up with a plan, also down....
>>
>> Single point of failure in the extreme.
>>
>>
>> On Mon, Oct 4, 2021 at 3:20 PM Michael W. Lucas 
>> <mwlucas at michaelwlucas.com <mailto:mwlucas at michaelwlucas.com>> wrote:
>>
>>
>>     <blink> <blink>
>>
>>     That's it. Twitter is the next to go.
>>
>>     ==ml
>>
>>
>>     On Mon, Oct 04, 2021 at 04:15:39PM -0400, George Rosamond wrote:
>>     > One of the cooler things we can all learn from the
>>     Facebook/Instagram/WA
>>     > outage is some technical innovations most of you probably missed.
>>     >
>>     > https://nitter.eu/briankrebs/status/1445077617426718725
>>     <https://nitter.eu/briankrebs/status/1445077617426718725>
>>     >
>>     > Apparently, there are no RFCs for it, but DNS Global Routing
>>     Tables is
>>     > the place to be. Move over blockchain.
>>     >
>>     > g
>>     >
>>     > _______________________________________________
>>     > talk mailing list
>>     > talk at lists.nycbug.org <mailto:talk at lists.nycbug.org>
>>     > http://lists.nycbug.org:8080/mailman/listinfo/talk
>>     <http://lists.nycbug.org:8080/mailman/listinfo/talk>
>>
>>     -- 
>>     Michael W. Lucas https://mwl.io/ <https://mwl.io/>
>>     author of: Absolute OpenBSD, SSH Mastery, git commit murder,
>>     ?Absolute FreeBSD, Immortal Clay, Prohibition Orcs, etc, etc, etc...
>>     ### New books: TLS Mastery, the Networknomicon, $ git sync murder ###
>>
>>     _______________________________________________
>>     talk mailing list
>>     talk at lists.nycbug.org <mailto:talk at lists.nycbug.org>
>>     http://lists.nycbug.org:8080/mailman/listinfo/talk
>>     <http://lists.nycbug.org:8080/mailman/listinfo/talk>
>>
>>
>> _______________________________________________
>> talk mailing list
>> talk at lists.nycbug.org
>> http://lists.nycbug.org:8080/mailman/listinfo/talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.nycbug.org:8443/pipermail/talk/attachments/20211004/9d923cf7/attachment.htm>

From george at ceetonetechnology.com  Mon Oct  4 19:59:20 2021
From: george at ceetonetechnology.com (George Rosamond)
Date: Mon, 4 Oct 2021 19:59:20 -0400
Subject: [talk] FB outage
In-Reply-To: <7fe1def8-1c07-2815-2efc-dbae226cb281@osfda.org>
References: <53100307-c10a-8ee7-3286-2d9604f89061@ceetonetechnology.com>
 <YVtvh+SZHd4xbsP2@mail.mwl.io>
 <CANCZdfp6SFsSE61R6MX=bxHyFjgg6dYzwS7SuOyFfJJegOg7Xw@mail.gmail.com>
 <324664c1-f961-0d46-5538-47e854a4ad8b@osfda.org>
 <7fe1def8-1c07-2815-2efc-dbae226cb281@osfda.org>
Message-ID: <36dfde8c-4ff1-78df-170c-fb65d2216dde@ceetonetechnology.com>

On 10/4/21 19:52, steve.b at osfda.org wrote:
> Akami, Google, DoD -they all have had serious bgp outages due to config 
> mistakes.

Yes, we all probably remember Pakistan Telecom...

https://www.wired.com/2008/02/pakistans-accid/

Then Alex P and a friend turned it into free tickets to DefCon:
https://www.wired.com/2008/08/revealed-the-in/

> 
> IF that's what it was...

Sure, we really have no idea since there's been nothing "official" that 
is totally trustable and believable from a company that is clearly neither.

But the reality is that the BGP issues are verifiable...

g



From ike at blackskyresearch.net  Mon Oct  4 20:54:40 2021
From: ike at blackskyresearch.net (Isaac (.ike) Levy)
Date: Mon, 4 Oct 2021 20:54:40 -0400
Subject: [talk] FB outage
In-Reply-To: <36dfde8c-4ff1-78df-170c-fb65d2216dde@ceetonetechnology.com>
References: <53100307-c10a-8ee7-3286-2d9604f89061@ceetonetechnology.com>
 <YVtvh+SZHd4xbsP2@mail.mwl.io>
 <CANCZdfp6SFsSE61R6MX=bxHyFjgg6dYzwS7SuOyFfJJegOg7Xw@mail.gmail.com>
 <324664c1-f961-0d46-5538-47e854a4ad8b@osfda.org>
 <7fe1def8-1c07-2815-2efc-dbae226cb281@osfda.org>
 <36dfde8c-4ff1-78df-170c-fb65d2216dde@ceetonetechnology.com>
Message-ID: <A051E3C0-4902-4FEE-879D-76A2AC4C1F00@blackskyresearch.net>


> On Oct 4, 2021, at 7:59 PM, George Rosamond <george at ceetonetechnology.com> wrote:
> 
> On 10/4/21 19:52, steve.b at osfda.org wrote:
>> Akami, Google, DoD -they all have had serious bgp outages due to config mistakes.
> 
> Yes, we all probably remember Pakistan Telecom...
> 
> https://www.wired.com/2008/02/pakistans-accid/
> 
> Then Alex P and a friend turned it into free tickets to DefCon:
> https://www.wired.com/2008/08/revealed-the-in/
> 
>> IF that's what it was...
> 
> Sure, we really have no idea since there's been nothing "official" that is totally trustable and believable from a company that is clearly neither.
> 
> But the reality is that the BGP issues are verifiable...
> 
> g

But what is FaceBook?

Rocket-
.ike



From raulcuza at gmail.com  Mon Oct  4 21:09:06 2021
From: raulcuza at gmail.com (Raul Cuza)
Date: Mon, 4 Oct 2021 21:09:06 -0400
Subject: [talk] FB outage
In-Reply-To: <A051E3C0-4902-4FEE-879D-76A2AC4C1F00@blackskyresearch.net>
References: <53100307-c10a-8ee7-3286-2d9604f89061@ceetonetechnology.com>
 <YVtvh+SZHd4xbsP2@mail.mwl.io>
 <CANCZdfp6SFsSE61R6MX=bxHyFjgg6dYzwS7SuOyFfJJegOg7Xw@mail.gmail.com>
 <324664c1-f961-0d46-5538-47e854a4ad8b@osfda.org>
 <7fe1def8-1c07-2815-2efc-dbae226cb281@osfda.org>
 <36dfde8c-4ff1-78df-170c-fb65d2216dde@ceetonetechnology.com>
 <A051E3C0-4902-4FEE-879D-76A2AC4C1F00@blackskyresearch.net>
Message-ID: <CAOnfn2ykurot7nS0y4E8ZcscsZZd-VfdnxJs=tCnrH0vBTJh6A@mail.gmail.com>

On Mon, Oct 4, 2021 at 20:55 Isaac (.ike) Levy <ike at blackskyresearch.net>
wrote:

>
> > On Oct 4, 2021, at 7:59 PM, George Rosamond <
> george at ceetonetechnology.com> wrote:
> >
> > On 10/4/21 19:52, steve.b at osfda.org wrote:
> >> Akami, Google, DoD -they all have had serious bgp outages due to config
> mistakes.
> >
> > Yes, we all probably remember Pakistan Telecom...
> >
> > https://www.wired.com/2008/02/pakistans-accid/
> >
> > Then Alex P and a friend turned it into free tickets to DefCon:
> > https://www.wired.com/2008/08/revealed-the-in/
> >
> >> IF that's what it was...
> >
> > Sure, we really have no idea since there's been nothing "official" that
> is totally trustable and believable from a company that is clearly neither.
> >
> > But the reality is that the BGP issues are verifiable...
> >
> > g
>
> But what is FaceBook?
>
> Rocket-
> .ike


What if nobody noticed?

Maybe in BGP v6 we can build in a stochastic feature that turns off the
Internet for an hour. If the number of people who die from that feature is
low enough, it would make the human part of the system more resilient.

- r

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.nycbug.org:8443/pipermail/talk/attachments/20211004/040e8123/attachment.htm>

From greg at ltcc.com  Mon Oct  4 22:46:05 2021
From: greg at ltcc.com (Greg Robinson)
Date: Mon, 4 Oct 2021 19:46:05 -0700
Subject: [talk] FB outage
In-Reply-To: <53100307-c10a-8ee7-3286-2d9604f89061@ceetonetechnology.com>
References: <53100307-c10a-8ee7-3286-2d9604f89061@ceetonetechnology.com>
Message-ID: <004501d7b993$25ebcb20$71c36160$@ltcc.com>

Frankly, it's a welcome event.

Best,
Greg
Lucrosol
415.922.3675 (PST)
?You get rid of the thought of competition. You are to create, not to
compete for what is already created. You do not have to take anything away
from any?one?
Wallace D. Wattles


-----Original Message-----
From: talk [mailto:talk-bounces at lists.nycbug.org] On Behalf Of George
Rosamond
Sent: Monday, October 04, 2021 1:16 PM
To: talk <talk at lists.nycbug.org>
Subject: [talk] FB outage

One of the cooler things we can all learn from the Facebook/Instagram/WA
outage is some technical innovations most of you probably missed.

https://nitter.eu/briankrebs/status/1445077617426718725

Apparently, there are no RFCs for it, but DNS Global Routing Tables is the
place to be. Move over blockchain.

g

_______________________________________________
talk mailing list
talk at lists.nycbug.org
http://lists.nycbug.org:8080/mailman/listinfo/talk



From jondrews at fastmail.com  Mon Oct 18 20:14:55 2021
From: jondrews at fastmail.com (Jonathan Drews)
Date: Mon, 18 Oct 2021 18:14:55 -0600
Subject: [talk] Semi-BUG Presentation on EMACS
Message-ID: <YW4N/+v7YlGA9nKv@leo.easyfashionclothing.com>

Hi Folks:

SEMI-Bug will host a presentation on EMACS tomorrow, Tuesday, October,
19th, at 1900 Detroit, MI time.
The Jitsi URL is:

https://meet.jit.si/SEMI-BugEMACSPresentation

--
Kind regards,

Jonathan