Re: facebook outage

["Patrick W. Gilmore" <patrick () ianai net>]

On Oct 4, 2021, at 5:30 PM, Bill Woodcock <woody () pch net> wrote:
> On Oct 4, 2021, at 11:21 PM, Bill Woodcock <woody () pch net> wrote:
> > On Oct 4, 2021, at 11:10 PM, Bill Woodcock <woody () pch net> wrote:
> > > They’re starting to pick themselves back up off the floor in the last two or three minutes.  A few answers getting 
> > > out.  I imagine it’ll take a while before things stabilize, though.
> >
> > aaaand we’re back:
> >
> > WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9
>
> So that was, what…  15:50 UTC to 21:05 UTC, more or less…  five hours and fifteen minutes.
>
> That’s a lot of hair burnt all the way to the scalp, and some third-degree burns beyond that.
>
> Maybe they’ll get one or two independent secondary authoritatives, so this doesn’t happen again.  :-)

If by “independent” you mean “3rd party” (e.g. DynDNS), not sure what an external secondary would have done here. While 
their BGP was misbehaving, the app would not work even if you had a static DNS entry.

And while using external / 3rd party secondaries is likely a good idea for many companies, almost none of the largest 
do this. These companies view it as a control issue. Giving someone outside your own employees the ability to change a 
DNS name is, frankly, giving another company the ability to take you down.

Taking a sample of FB, cisco, Amazon, NF, Dell, Akamai, Google, MS, CF, only 2 use 3rd party resolvers.
* NF uses only awsdns, so same problem, just moved to another company they do not control.
* Amazon uses Ultra & Dyn. (Anyone else amused amazon.com has no authorities on Route 53? At least not from my vantage 
point.)

That said, plenty of what people may call “big” companies do use 3rd parties, e.g. IBM, PayPal, Juniper.

You want to use a 3rd party DNS, go for it. There are lots of reasons to do it. But it is not a panacea, and there are 
reasons not to.

-- 
TTFN,
patrick