6 lessons learned about the scariest security threats

[Audrey McNeil <audrey () riskbasedsecurity com>]

http://www.networkworld.com/news/2014/022414-6-lessons-learned-about-the-279082.html?source=nww_rss

Advanced persistent threats have garnered a lot of attention of late,
deservedly so. APTs are arguably the most dangerous security concern for
business organizations today, given their targeted nature.

An APT attack is typically launched by a professional organization based in
a different country than the victim organization, thereby complicating law
enforcement. These hacking organizations are often broken into specialized
teams that work together to infiltrate corporate networks and systems and
extract as much valuable information as possible. Illegally hacking other
companies is their day job. And most are very good at it.

By all expert opinion, APTs have compromised the information infrastructure
of any relevant company. The question isn't whether you've been compromised
by an APT, but whether you've noticed it.

I've been helping companies fight and prevent APTs for nearly a decade. In
that time I've amassed my share of war stories from the IT security
trenches. Here are some of the better real-life tales, not just for the
chase, but for the lessons learned.

APT war story No. 1: APT eyes are watching you

I once spent more than a year responding to an APT attack at a
multinational company that was involved in everything from high-tech
satellites and guns to refrigerators and education. When I got the call,
the client had already been hit by two other APT attacks, which isn't
unusual. Most companies that discover an APT usually figure out it's been
there for years. One client I worked with has been compromised by three
different APT teams over the past eight years -- not surprising in the
least.

The multinational was finally serious about combatting the attack. The
entire IT team was called together to respond; a large, single-purpose task
force was created; all the relevant experts were brought in. It was decided
that many months in the future all passwords would be reset.

You may wonder why the delay in resetting passwords. Password resets should
always be pushed out far into the future in these situations because
there's no use changing all the passwords to kick out an APT if you can't
guarantee you can prevent the baddies from breaking right back in. Identify
the most relevant weaknesses, fix them, then change the passwords. That's
the best defense.

As in most companies I work with, everyone involved was sworn to secrecy.
Code words were established, so the team could discuss various aspects of
the project in (possibly monitored) emails without alerting intruders or
employees not yet engaged.

In this instance, the big password reset day was scheduled to coincide with
the company's annual baseball game, which had been instituted to increase
employee morale. Because of this, the project was dubbed "company baseball
game," with the name of the company changed here to protect its identity.
> From that point forward, no one mentioned APT or password reset. Everything
was about the baseball game.

The company's systems were completely compromised, so new laptops and
wireless routers were purchased. All project-related work was to be
performed on these laptops over a secured wireless network to prevent any
accidental leakage of information about the project, regardless of
code-word use.

One facet of the project was to tackle the overabundance of domain
administrators at the company. There were far too many -- all told, more
than 1,000. We set up camp in one of the many executive conference rooms we
used over the course of the project and began discussing what to do.

We couldn't decide which domain administrators were truly needed and which
we could disable, so we decided to disable them all on "company baseball
game" day, and force those who really needed domain admin access to
reaffirm their need. We drafted a domain admin access request form on one
of the project laptops and called it a day. We would send out the forms
just before "company baseball game" day so that each person who needed a
domain admin account could get one in time to be prepared.

The next morning around 7:30 a.m., I entered that same executive conference
room. The project manager was already there. He looked up at me, his eyes a
bit wider than usual for the early hour, and said, "Here's our first two
domain admin requests," as he flipped them to me.

What did he mean domain admin requests? The form wasn't out of draft stage
and wasn't scheduled to go out for months. But there they were, two
filled-out "domain admin access request" forms. They had some small, but
very noticeable mistakes, so it was obvious they were not from our original
draft. Each was filled in by team members belonging to a foreign
subsidiary, who currently had domain admin access. The reason they were
requesting the reinstated domain admin access? Because the current access
was to be cut on baseball game day.

To this day, I still can't believe it. I was holding two forms that
shouldn't have existed. The only draft was on a laptop on an air-gapped
network. Our precious secret project code was blown. Astonishment passed
from team member to team member along with the forms as we gave them the
news.

After much investigation, we figured out that the APT, led by insiders, had
infiltrated all the conference rooms using the data display projectors and
executive videoconference systems. They were watching and digesting all our
supposedly secret meetings. Their only mistake was in not understanding
that the form didn't really exist yet and was not due to be sent out for
months. Thank goodness for language barriers.

Lesson: If your conference equipment is networked and has the ability to
record voice or video, make sure you disable them before conducting
meetings.

APT war story No. 2: Not all APTs are as advanced as experts think

This is the story of an APT team that had taken total control of a
company's network. They were actively creating connections all around the
network, day or night, by the time I got called in. They were beyond caring
whether they had been discovered.

APTs are almost certain to dump all password hashes and use pass the hash
(PtH) tools to take over the rest of an organization's network. In this
instance, the customer decided it was time to disable those weak LAN
Manager (LM) password hashes that Microsoft had been recommending to
disable for at least 10 years, and trying to disable by default at least
since 2008. This particular APT was using the captured LM password hashes
to do the dirty work.

I told the customer the proposal would not work because, by default, at
least two types of Windows password hashes exist in Microsoft
authentication databases: LM and NT hashes. The attackers had downloaded
both types, and the PtH tool they were working with could use either. I
even showed the client how the attacker's tool had the syntax built in to
switch between LM and NT hashes, a very common feature of PtH attack tools.
Worse, even if you disable the storing of LM hashes, they are still created
in memory when someone logs on. It sounds crazy, but that's how Windows
works.

The customer would not be dissuaded. Despite my protestations of wasted
effort, it disabled the LM hashes and reset the passwords. Now the local
and Active Directory databases contained no usable LM password hashes. You
know how well that worked?

Well, it worked -- because the APT team never used another password hash to
perform its attack. Truth be told, they just moved on to other methods (see
below), but the PtH attacks stopped. It turned out that the APT team didn't
even know its own tools. You could imagine the discussion they must have
had internally when all the LM hashes disappeared, including shrugged
shoulders and a brainstorm of new strategies.

Lesson: "Advanced" may be included in the name of APT, but not all APT
attackers are all that advanced. Plus, sometimes the expert is wrong. I
wasn't wrong technically, but that didn't prevent the outcome the client
was looking for to be the same. It humbled me.

APT war story No. 3: The medicine may be the poison

As a full-time Microsoft security consultant, I'm frequently asked to work
on APT engagements led by other companies; I'm a resource, not the project
leader. There's one security consulting company I've worked with enough to
know many of its staff members and consultants informally, if not
personally. We understand what our roles are -- depending on who gets there
first, makes friends with the CIO, and assumes leadership. Our partnerships
have always been friendly, though competitive. After all, it's better to be
a leader than a follower.

This security consulting firm is well known for fighting APTs and even
sells detection software to help. Frequently, on engagements, it succeeds
in selling its software and getting it installed on every computer in the
environment. I was very used to seeing its service running in Windows Task
Manager.

In this particular story, the security consulting firm arrived first, saved
the day, and moved on. It also succeeded in installing its software
throughout the organization and hadn't been onsite in nearly a year. As far
as anyone knew, the customer had been APT-free since the initial remedy. At
least no one had detected any signs.

I'm a big fan of honeypots. A honeypot is software or a device that exists
simply to be attacked. It can be an unused computer, router, or server.
Honeypots can mimic anything, and they are great for detecting previously
undetectable adversaries, so I recommend them often. This can be a
decommissioned computer to which no person or service should be connecting.
When a hacker or malware does connect, the honeypot sends an alert that can
trigger an immediate incident response.

In this instance, I spent a few days helping the client deploy some
honeypots. Most customers ask me how we are going attract hackers to the
honeypots. I always laugh and answer the same way: "Don't worry, they will
come." Indeed, every honeypot I've ever set up has detected nefarious
activity within a day or two. These new honeypots were no different.

We detected network logon attempts coming from multiple workstations, none
of which had a legitimate reason to be logging on. We pulled a few of these
workstations and forensically examined their hard drives. We found that the
APT had placed a remote-access Trojan on each of them. The Trojan's name?
The same as the anti-APT detection software. The bad guys had someone
replace the legitimate anti-APT software with a Trojan, and it turns out
they did it on nearly every computer.

This explained a few things, like why no APT had been detected. But the
bigger question was how did it get installed in the first place. It turned
out the customer's "gold build" had been compromised in its build
environment, and this Trojan was part of the build.

Lessons: First, verify the integrity of your builds; prevent unauthorized
modification or invent some way to detect it. Second, honeypots are a great
way to detect malicious activity. Third, always look for and investigate
strange network connections from unexpected places.

APT war story No. 4: All your PKI base belong to us

APT attacks on Public Key Infrastructure (PKI) servers used to be somewhat
rare. In fact, until two years ago, I never personally ran across an APT
where PKI servers had been involved. Now, it's fairly common. But the most
relevant story is the one where the PKI turned into physical access in a
sensitive area.

This particular customer used its internal PKI servers to create employee
smartcards. These smartcards were used to not only log on to computers but
to physically access company buildings and other infrastructure.

The customer's root CA (certification authority) server was a virtual
instance sitting, disabled, on a VMware host server. The bad guys had found
it, copied it offsite, cracked the local (weak) administrator password, and
generated their own trusted subordinate CA. They used this CA to issue
themselves PKI access to everything they could.

What surprised me most was the video my client showed me of two unknown men
posing as employees. Using the fake smartcards they created, they had
parked their cars inside the secured company parking lot, walked into the
building through the employee entrance, and onto a floor that stored highly
sensitive data.

My customer couldn't tell me what happened after that or what was taken,
but I knew they were not happy. There was a very serious mood in the room.
I was invited to help them create a new PKI and to migrate the company into
the better-secured PKI environment.

Lesson: Protect your PKI CA servers. Offline CAs should be just that:
offline! They should not be disabled or sitting on the network with their
network cards disabled, but off the network, stored in a safe, and not so
easy to compromise. CA private keys should be protected by a Hardware
Storage Module appliance, and all related passwords should be very long (15
characters or more) and complex. Plus, it can't hurt to look for and
monitor if other unauthorized CAs get added as trusted CAs.

APT war story No. 5: Don't forget the accounts you're not supposed to touch

As mentioned above, most APT recovery events involve resetting passwords.
If you're going to reset passwords, reset all accounts -- though it's
easier said than done. All my customers start out doe-eyed, ready to reset
all passwords, but when they discover how much it will disrupt the
business, they quickly scale back their goals. It's far easier to get fired
for causing a significant business interruption than it is for not getting
all the hackers out.

This particular customer was ready and incredibly thorough. The plan was
not only to reset all user and service accounts, but computer accounts as
well. Almost no companies do this, especially when it comes to resetting
service and computer accounts. Heck, I'm giddy if they reset all elevated
user accounts, because it's hard to get that little bit done thoroughly.
Laugh only if you haven't been through this drill.

Password reset day came and went. There were significant service
disruptions, some of which were painful enough that we had to tell the CEO.
By the end of the week, however, we had reset all the passwords.

Within a few days, the APT owned everything again, picking up all email,
controlling all the elevated accounts, including IT security accounts. It
was like the password reset never happened. We were perplexed. As best we
knew, we had removed the easy holes, educated employees, and couldn't see
any evidence of Trojan backdoors.

Alas, there's a built-in Windows account called krbtgt that is used for
Kerberos authentication. You shouldn't touch it, remove it, or as far as we
previously knew, change its password. It really shouldn't be a user account
that shows up in user account management tools, and this APT team knew it.

As I've learned on successive engagements, krbtgt is a go-to technique.
After an APT crew compromises an environment, they add the krbtgt account
to other elevated groups. Because customers usually leave it alone, even
during a password reset, it can be exploited as a go-to backdoor account.
Great idea -- if you're a malicious hacker.

My customer reset the passwords of its krbtgt accounts and everything else
(again). As far as I know, it has not had another detected problem. Be
aware that resetting krbtgt accounts will absolutely cause authentication
problems. It's a pain. But if you have to do this, you too will get through
it.

Lesson: if you're going to reset all accounts, make sure you know what
"all" means.

APT war story No. 6: Information overload is spurring APT innovation, too

My last story isn't about a single client, and it shows the evolution of
APT over the years. Early APT practitioners would immediately collect
everything they could as soon as they broke in. They would siphon out all
old emails and install bots to get every new email sent. Many times they
would install Trojans to monitor the network and databases, and if new
content was created, they would copy it.

In other words, many companies have online backup services they aren't
paying for.

Those were the old days. In the world where terabyte databases are no
longer even close to surprising, APT has a problem. When they get complete
access to a network and learn where all the information is stored, they
have to be more selective. Whereas they used to grab everything, what we
see now are very discrete selections. The more advanced APTs these days
build their own search engines, sometimes with their own APIs or borrowing
the APIs of other well-known search engines, to search for specific data.
They may still only leave with gigabytes of data a day, but what they have
is highly selective.

Lesson: APT has the same issues finding and managing data just like you do.
Don't let them index your data better than you do.

_______________________________________________
Dataloss Mailing List (dataloss () datalossdb org)
Archived at http://seclists.org/dataloss/
Unsubscribe at http://lists.osvdb.org/mailman/listinfo/dataloss
For inquiries regarding use or licensing of data, e-mail
        sales () riskbasedsecurity com 

Supporters:

Risk Based Security (http://www.riskbasedsecurity.com/)
YourCISO is an affordable SaaS solution that provides a comprehensive information security program that ensures focus 
on the right security.  If you need security help or want to provide real risk reduction for your clients contact us!