RE: massive facebook outage presently

[Jean St-Laurent via NANOG <nanog () nanog org>]

I don't understand how this would have helped yesterday.

> From what is public so far, they really paint themselves in a corner with no way out. A classic, but at epic scale.

They will learn and improve for sure, but I don't understand how "firmware default to your own network" would have help 
here.

Can you elaborate a bit please?

Jean

-----Original Message-----
From: NANOG <nanog-bounces+jean=ddostest.me () nanog org> On Behalf Of Glenn Kelley
Sent: October 4, 2021 8:18 PM
To: nanog () nanog org
Subject: Re: massive facebook outage presently

This is why you should have Routers that are Firmware Defaulted to your own network.  ALWAYS

Be it Calix or even a Mikrotik which you have setup with Netboot - having these default to your own setup is REALLY a 
game changer.

Without it - you are rolling trucks or at minimum taking heavy call volumes.


Glenn Kelley
Chief cook and Bottle Washing Watcher @ Connectivity.Engineer

On 10/4/2021 3:56 PM, Matt Hoppes wrote:
> Yes,
> We've seen that.
>
> On 10/4/21 4:33 PM, Eric Kuhnke wrote:
> > I am starting to see reports that in ISPs with very large numbers of 
> > residential users, customers are starting to press the factory-reset 
> > buttons on their home routers/modems/whatever, in an attempt to make 
> > Facebook work. This is resulting in much heavier than normal first 
> > tier support volumes. The longer it stays down the worse this is 
> > going to get.
> >
> >
> >
> > On Mon, Oct 4, 2021 at 3:30 PM Jay Hennigan <jay () west net 
> > <mailto:jay () west net>> wrote:
> >
> >     On 10/4/21 12:11, bzs () theworld com <mailto:bzs () theworld com> wrote:
> >      >
> >      > Although I believe it's generally true that if a company appears
> >      > prominently in the news it's liable to be attacked I assume 
> > because
> >      > the miscreants sit around thinking "hmm, who shall we attack 
> > today oh
> >      > look at that shiny headline!" I'd hate to ascribe any altruistic
> >      > motivation w/o some evidence like even a credible twitter post 
> > (maybe
> >      > they posted that on FB? :-)
> >
> >     I personally believe that the outage was caused by human error 
> > and not
> >     something malicious. Time will tell.
> >
> >     However, if you missed the 60 Minutes piece, it was a former 
> > employee
> >     who spoke out with some rather powerful observations. I don't think
> >     that
> >     this type of worldwide outage was caused by an outside bad actor. 
> > It is
> >     certainly within the realm of possibility that it was an inside job.
> >
> >     In other news:
> >
> > https://twitter.com/disclosetv/status/1445100931947892736?s=20
> >
> >     --     Jay Hennigan - jay () west net <mailto:jay () west net>
> >     Network Engineering - CCIE #7880
> >     503 897-8550 - WB6RDV