Re: United Airlines is Down (!) due to network connectivity problems

[Sean <sean76 () gmail com>]

I've been in UA's datacenter and while I'm no expert on their setup I can
say with some confidence that it's most likely NOT related to anything else
going on.  I don't want to violate any NDA I may or may not have signed but
I think I can safely say its all one big private network.  Whatever's
happening on the internet or with NYSE has got nothing to do with what is
more than likely in this case a big fat BGP clusterfudge like most of these
things are.  I don't have any inside info or anything just a slightly more
educated guess.

On Wed, Jul 8, 2015 at 2:31 PM, Patrick W. Gilmore <patrick () ianai net>
wrote:

> I’m with Ferg-dog.
>
> I can’t tell you the number of times someone (yes, including me) has
> designed, purchased, and installed a system with multiple backups,
> failovers, redundancies, etc., and some vital piece fails in a weird way
> which sends the whole thing into a tailspin.
>
> Taking UA as an example, since we have the most information (FSVO “most”),
> namely it was a “bad router”. Let’s assume they had multiple routers
> configured with VRRP, BGP, OSPF, and an alphabet soup of other ways to
> detect and route-around failures. Now further assume one of those routers
> has a software or hardware bug which doesn’t take the router out of
> service, but leaves it up, replying to pings, answer SNMP polls, speaking
> BGP or OSPF, sending VRRP hellos, etc., etc. - but also eats half of all
> packets going _through_ the router. That can happen, I’ve seen it first
> hand.
>
> All those redundant systems do nothing, since the “bad router” is doing
> everything a good router would do. The systems designed to catch such
> problems all think things are fine, but they are not. Is it an attack? No,
> it’s bad luck.
>
> Now some will claim - and perhaps rightfully - that UA should have systems
> which monitor for exactly this type of failure as well. Perhaps they should
> have, or perhaps the problem was nothing like what I explained. Either way,
> the point still stands that a company can have had multiple redundancies in
> place, but still experienced a failure mode which caused exactly the
> problem described.
>
>
> At this point, we move on to: “All three simultaneously?!? NO WAY!!” To
> which I would point out they were not simultaneous. UA was back up before
> NYSE went down. But even if they were simultaneous, sometimes stuff
> happens. The human mind is very good at seeing connections, even when there
> are none. Absent other evidence, I’m going to believe the companies’ public
> statements that this was not a hack. Perhaps I am being naive, but as I
> said, absent other evidence, it is a perfectly plausible explanation.
>
> --
> TTFN,
> patrick
>
>
> > On Jul 08, 2015, at 14:56 , Jay Ashworth <jra () baylink com> wrote:
> >
> > UA, WSJ /and/ NYSE all in the same day?
> >
> > Once is an accident;  twice is a coincidence...
> >
> > Three times is enemy action.
> >
> > On July 8, 2015 1:18:47 PM EDT, Paul Ferguson <fergdawgster () mykolab com>
> wrote:
> > > -----BEGIN PGP SIGNED MESSAGE-----
> > > Hash: SHA256
> > >
> > > Given that the Internet is held together with paper clips, bailing
> > > twine, and bubblegum, I'd prefer to take theses organizations' initial
> > > word for the fact that there is nothing obviously malicious in these
> > > outages.
> > >
> > > The mainstream press, on the other hand, seems to want it to be a hack
> > > or data breach or... something other than a "glitch". :-)
> > >
> > > - - ferg
> > >
> > >
> > > On 7/8/2015 10:15 AM, Mel Beckman wrote:
> > >
> > > > It's important to not form an opinion too early, especially anyone
> > > > involved with forensic analysis of these systems. This is a
> > > > classic fault in amateur investigation: an early opinion will lead
> > > > you into confirmation bias, irrationally accepting data agreeing
> > > > with your opinions and rejecting that disproving it.
> > > >
> > > > -mel beckman
> > > >
> > > > > On Jul 8, 2015, at 10:07 AM, Paul Ferguson
> > > > > <fergdawgster () mykolab com> wrote:
> > > > NYSE: "The issue we are experiencing is an internal technical issue
> > > > and is not the result of a cyber breach."
> > > >
> > > > https://twitter.com/NYSE/status/618818929906085888
> > > >
> > > > United Air statement CNBC: “An issue with a router degraded network
> > > > connectivity for various applications. We fixed the router."
> > > >
> > > > https://twitter.com/barronstechblog/status/618816643821633536
> > > >
> > > > - ferg
> > >
> > >
> > > - --
> > > Paul Ferguson
> > > PGP Public Key ID: 0x54DC85B2
> > > Key fingerprint: 19EC 2945 FEE8 D6C8 58A1 CE53 2896 AC75 54DC 85B2
> > > -----BEGIN PGP SIGNATURE-----
> > > Version: GnuPG v2
> > >
> > > iF4EAREIAAYFAlWdW3cACgkQKJasdVTchbLr/wD/aBNnLFv+MU+QI1ja7dd9LiSN
> > > Zkum4lSIutxFn1NmaYoBAIgO/Ig7FxD4vRzQK8bUturn4YGw9FXMT+EzVTKhIbVG
> > > =/yYp
> > > -----END PGP SIGNATURE-----
> >
> > --
> > Sent from my Android phone with K-9 Mail. Please excuse my brevity.