Pwned so many times – but saved by the incident response plan

[Audrey McNeil <audrey () riskbasedsecurity com>]

http://www.theregister.co.uk/2015/06/17/it_security_incident_response_plan/

Companies that are more proficient with technology are more likely to
believe that their security is "very effective". Is this a form of contempt
born of familiarity, or a true understanding of the risks? The bigger the
company, the harder they fall, and no organisation – not even the US state
department – has proven impenetrable.

Survey after survey is conducted and each time it seems that the"early
adopters" of whatever tomorrow's next big thing is are the companies who
think they have got this security thing taped. Companies who felt ahead of
the curve because they were all cloudified were riding high on the
knowledge that public cloud providers have better security than most.

Until they realised you're only as secure as your stupidest mistake.

Internet of Things and wearable adopters don't seem to even be thinking
that far. Lots of widgets are being deployed with security threats that
will never be patched and companies are gleefully putting these on the same
networks as their primary data. In IT, the hard lessons learned face-first
by so many other companies in the past never seem to be retained.

It's easy to think that large companies with huge resources have security
down. Small business admins are constantly told by their better resourced
brethren about how little they know, how being an enterprise admin playing
with enterprise toys means that the SMB admin's skillset is old, rusty and
will never amount to anything.

I disagree.

Unhealthy security

An off the record discussion with a source of mine within the US health
care insurance industry revealed the shocking fact his very large employer
had thousands of servers with no anti=malware, intrusion detection or other
defenses installed. These aren't any old servers either - we're talking
database farms filled with the most sensitive of customer data. The entire
organisation relies on eggshell security: crunchy defences on the very
outer edge, but the inside is soft.

Worse, being in the industry they talk to others in the industry, and it
turns out that this is fairly standard. It is alleged that some of these
companies even cover it up in order to pass audits. This, BTW, looks like
this remains the case after the ridiculous hacks against Premera. Which, if
you'd followed the Anthem hacks, is no big surprise. And that's before we
get into revenge hacking by health care sysadmins.

After some digging around, I found a common theme to the "why" of this: the
CIOs were under immense pressure to cut costs, and they didn’t feel the
risk of getting Sony-ed was all that high. Besides, their companies
themselves have insurance in case of a breach, so really, why should they
care? The consequences just aren't high enough.

Fortunately, this is changing.

Paranoia is proper

I've been hacked. After 20 years in IT, my systems have probably been
penetrated dozens of times. It happens a few times a year now, so I've
honestly lost count.

By admitting this, I am of course setting myself up for a right pantsing in
the comments section for being wholly inadequate to live. Sysadmin machismo
and brogrammer culture demand that all technologists be infinitely capable,
infinitely knowledgeable and under no circumstances ever admit fault.

Well screw that. I got pwned. Right pwned. Several times. Some of them were
clearly my fault (I done goofed) and some of them were the result of
completely asymmetric resources deployed against me. I've seen stuff turn
up on my personal systems that qualifies as state-level (seriously, they
hid the malware in the video bios!)

I've also had attacks against systems I thought were "secure" that were so
spectacularly beyond my level of capability that months later I still don't
have the foggiest idea in hell how they got in. Those systems had no known
exploits, the attackers danced past every security feature, from fail2ban
to layers of intrusion detection. To this day, I'm completely baffled.

I have come to accept being pwned as a matter of course. I don't have the
resources of the US State Department. I don't have the resources of a
health care company, Sony, or a cloud services provider. They can't – or
don't, or won't – put the time and effort into fending off black hat
hackers, so in what universe do I get off thinking I'm going to?

And that's before we talk about the bit where using public cloud computing
is essentially handing your data over to governments that are decidedly
hostile to even their own constitutional guarantees regarding individual
civil liberties.

Those breaches however, aren't "fatal". Why? Paranoia. Defence in depth. A
realisation quite some time ago that eggshell computing is a really, really
bad plan.

Separation anxiety

We must assume that any system on our network can be – and in the fullness
of time will be - compromised. Data and applications need to be segmented.
Different applications need to be separated. Pwning one system cannot be
allowed to grant an attacker access to all other systems, nor can it allow
them the ability to nuke the only copy of vital data.

As I write this, I am dealing with a pwned webserver on a client site.
Reasonably well built server, but some putz walked through some ancient and
unmaintained script in the client's custom-built (and publicly facing)
web-based middleware. They thoroughly owned the CentOS VM running that
workload (using an off-the-shelf package that, by the looks of it, was
designed to compromise Debian,) and used it to try to send spam.

Which must have been frustrating for the attackers, as there's a lovely
little hardware appliance monitoring every packet out of the host that
system is attached to, and it is squelching any and all email traffic. Oh,
and the website files are loaded read only. And it doesn't have access to
deeper layers of the network because the routers squelch any attempts (and
then freak out and email when they detect it.)

I've taken down the compromised script, shaken my finger at the devs and
the VM is being cloned for some later forensics. Somewhere around the point
where I finish writing this article, I'll revert to the "known good"
snapshot, run updates and we're back to clean living.

Sysadmin enough to admit you need help

I learned a long time ago that I'm not the IT messiah. I was once pretty
decent at providing enterprise-level IT on a practically impossible budget,
but the older I get the less stamina I have to play that game. IT is bigger
today than it was 20 years ago. No one human being can fit all that needs
be known about our industry into their brain.

So I simply gave up. Every now and again I need an adult. I am not going to
build a better firewall than F5. Even if it is all I did all year long,
they are smarter, more experienced and there are simply more of them than
there are of me. Hell, technology is moving so fast today that I can barely
keep up with rebuilding my anti-spam server's applications and definitions
up to date. Every 6 months or so ClamAV makes some change that borks
everything and I have to rebuild it.

Security appliances are required. Professional security services are
required. External audits, additional pairs of eyes and above all else
engineering your network expecting to be compromised are required.

You aren't as good a sysadmin as you think. I don't care if you fly to work
every day on your personal helicopter that you handcrafted from mahogany
while getting 50,000 steps a day on your Fitbit and bench pressing a 747
full of lesser sysadmins, I promise you that you aren't hot shit enough
defend your home network against a well-resourced attack, let alone an
enterprise IT network.

Accept it. Deal with it. And as with all things in IT, plan for failure.
Including your own.

Prevention will not stop all security threats. Detection will miss some
incidents. You, and I, and all of us need to be spending time on mitigating
the inevitable, and preparing incidence response plans for when breaches
happen anyways.

_______________________________________________
Dataloss Mailing List (dataloss () datalossdb org)
Archived at http://seclists.org/dataloss/
Unsubscribe at http://lists.osvdb.org/mailman/listinfo/dataloss
For inquiries regarding use or licensing of data, e-mail
        sales () riskbasedsecurity com 

Supporters:

Risk Based Security (http://www.riskbasedsecurity.com/)
YourCISO is an affordable SaaS solution that provides a comprehensive information security program that ensures focus 
on the right security.  If you need security help or want to provide real risk reduction for your clients contact us!