No Security is Bad Security:

[jjasen1 () UMBC EDU (John \)]

<<
Aleph;

Where I work, the policy has often been that security is not relevant, or
a pain in the neck, or the paranoid delusions of systems admin-types.

Well, last thursday, we discovered a rather annoying intrusion, and I
wrote a brief, almost 'manager-level' paper on hwo bad things get when
they go wrong ...

Perhaps other people on Bugtraq will have use of it?
> >

What Happened:
-------------

8:00am, Thursday, January 28th, I recieved email that one of my servers a)
had been compromised, and b) had been used in the compromise of a remote
university site, who was now very displeased with our existence.

I immediately logged into the offending machine, and investigated what
evidence the cracker had left behind. The first thing discovered was
trojan'ed copies of rshd, telnetd, and ftpd, in a supposedly hidden ...
directory. Much to my annoyance, I also found out that /usr/bin/ls was
trojan'ed, at least not to list ... and '. ' files. Switching to
/usr/ucb/ls, which the cracker missed, a rootkit trojan script was
discovered, which replaced several executables in /usr/bin and /usr/sbin.
I believe that the network services were manually trojan'ed.

The logs looked 'too clean', causing me to suspect that they had been
sanitised in some fashion.

As an offhand guess, we think that ftpd was compromised, in early January,
but lack concrete evidence.

My general opinion is that we most likely were dealing with what a friend
of mine calls a 'script kiddie.' However, he did a few things that struck
me as somewhat abnormal for a standard kiddie [namely the apparent manual
replacement of the rshd, et al], and I felt it prudent to continue to the
next step: the machine was sentenced to death -- unplugged from the
network, backed up, formatted and reinstalled. While we were at it, we
sentenced all the client machines to the same death, as they were
supported through NFS of binaries and rdist'ed account information.

Mistakes Made in Incidence Response:
-----------------------------------

1) Don't log in as root on a machine that most likely has been
compromised. Bsd things can happen.

2) Don't go around blithely executing binaries. (I feel rather stupid
about that)

3) Do *immediately* take the machine offline, and mount the disks on
another system for analysis.

Mistakes Made in Security Implementation:
----------------------------------------

1) we have no firewall nor tcpd running, so there is no effective access
control or access logging. We have incredibly primitive router filtering,
which eliminates only the most basic of IP-spoofing attacks.

2) we did not audit our services effectively, nor secure more reliable
alternatives to services we needed to offer. [examples: telnetd, rshd,
ftpd, httpd/cgi-bin, pop, imap]

3) we did not keep up with the security mailing lists, versioning of
important freeware services [sendmail, apache, ssh, bind, pop, imapd], nor
did we maintain operating system patches.

4) we did not audit our workstations or compute servers for unnecessary
SUID/SGID programs.

5) we did not engage in security sweeps/random audits with portscanners,
publicly available tools [SAINT], or commercial utilities [ISS].

6) we did not purchase or implement any Intrusion Detection Software.
[IDS]

7) we did not run tripwire, or any other cryptographic checksum program
against the executables on the server.

8) we did not log activity to a secure loghost.

9) we did not manually analyse the log files, nor did we run scripts to
correlate accesses against the log files.

10) we combined multiple services on one machine, and sometimes
known-insecure services on NIS or rdist-passwd clients. [web/ftp/mail on a
general use server]

11) we did not have a trusted archive of all the tools we
compiled/installed on the server, nor on the clients

Direct Costs Incurred:
---------------------

1) Over twelve hours of system unavailability.

2) Over 48 man-hours of work to restore services.

Indirect Costs Incurred:
-----------------------

1) secondary DNS server unavailable.

2) two days where researchers and staff cannot perform work.

3) open-ended period of time compiling and installing tools that we missed
the first time.

Where Our Security Implementation Hurt/Hindered:
-----------------------------------------------

Not providing more than the most rudimentiary access control denied us the
possibility of delaying, or even denying the compromise.

Not providing for a secure loghost resulted in the destruction of the
relevant log files, making analysis of the incident difficult,
apprehension of the felon unlikely and conviction impossible.

Not auditing services and not removing/replacing/securing especially
odious ones left systems openly available to the Internet[tm] in a fragile
state.

Not keeping up with security information and/or patch revisions left
a number of exploits 'lying around' on our system -- one or more of which
were used by the cracker.

Not auditing SUID/SGID programs leaves a wonderful toolkit of mayhem, with
which a malicious user can hurt you. (How many Solaris admins have
ufsrestore rsxr-xr-x?)

Not engaging in random security sweeps and roving audits denied us the
opportunity to notice something amiss, specifically in services offered on
our machines.

Not having trusted logfiles to analyse cost us the possibility of noticing
portscans or 'security sweeps' from the cracker, before they struck.

Not using tripwire cost us a lot, in that a) we had to rebuild every last
GNU program from source, and b) we did not have it available as a means of
detecting 'wrongness' on a production system.

Not divorcing services meant that we were running web/ftp on machines that
had the complete account/password information for the lab, 'just ripe for
the picking'. In addition, the compromise rendered the entire lab
unavailable as we rebuilt, rather than just one or two functions.

Not having a secure archive of tools (or even the systems in general!)
meant that we had to rebuild everything from scratch.

Lessons Learned:
---------------

When you think 'security,' think 'defense in depth.' The French
demonstrated very neatly  that putting all their resources into the
Maginot Line was not very bright, and we should make every effort *not* to
recreate the Maginot Line.

Security is *not* cost-intensive, if you build it in the first time, or
add it in as you upgrade your environment, especially as you value it
against the total loss of your environment.

Find a way to control outside access. Either throttle it through a
firewall, run it through router filters, or use tcpd. (in decending order
of preference)

Scan. Scan. Scan. Scan your machines for vulnerabilities, scan your
network for machines that shouldn't be there, and services that shouldn't
be running.

Create/purchase and use Intrusion Detection Software (IDS) across common
avenues of attack [phf exploit in cgi-bin, port scans, nfs traffic from
across the network, executing common buffer overflow exploits, etc -- this
could get exciting if you run a scan and the IDS sets off your pager ....
9000 times.]

Use a checksum sanity checking program, such a tripwire, on your binaries;
and run it periodically as insurance against anything going wrong.

Keep secure, trusted backups of important things -- cut a tape after
you've freshly installed/configured a server; and/or periodically cut a CD
of your favourite compiled tools.

Audit the services that you are offering, shut off the annoying, upgrade
the rest, and find ways of tightening security on them. (apop vs. pop, or
perhaps pop through an ssh tunnel?)

Try to keep track of security -- Bugtraq, at www.netspace.org, is a very
good resource.

Keep up to date on your patches. It is truly embarrassing to get nailed by
an exploit that was patched last year.

Divorce global services (httpd/ftp) from internal ones (mail/file);
important services from not so important, and *don't* run them on the same
machine!

CGI-BIN scripts are generally annoying, and generally hard to keep secure.
In my opinion, they should not be allowed to live. However, if you must
let them stay, consider: using perl with the 'taint' and 'warn' flags;
consider cgi_wrappers; consider 'mod_perl' for Apache; php; or sperl, for
so called 'secure PERL.'

Get a secure loghost, and run analysation algorythms against your
logfiles, perhaps: tabulating accesses to your network from external IP
number/domain name; by date; by port number; etc -- and look for
abnormalities. [This can even be scripted and crontabbed to death]

Investigate other security software: kerberos, AFS, etc.

By all means, attempt to break into your site. [NOTE: make sure you have
everything signed, notarised, dotted and witnessed before doing so!] Probe
remotely, discover what an 'outsider' can discover, and research how to
shut them off.

And, whatever you do, don't get complacent.

--
-- Dragons slain; Demons banished; Castles stormed--all in one low price! --
-- John E. Jasen    //  DNRC Ambassador to Earth  \\    jjasen1 () umbc edu  --
-- These views have been approved by the Y2K Panic Facilitation Committee --