Re: Penetration testing via shrinkware

[Dominique Brezinski <dom_brezinski () securecomputing com>]

Just so everyone knows my bias up front: I worked for ISS at one point as
part of the X-Force (the research group that develops exploit checks for
the scanner and ID signatures for RealSecure amongst other things) and I
currently work for Secure Computing in the architecture/assessment branch
of Professional Services. I do work very closely with our penetration team,
and occasionally I actually do some penetration testing too. 

At 07:22 PM 9/2/98 -0400, Stout, Bill wrote:
> What are the opinions on the thoroughness of shrinkwrap software
> penetration testing?  Is today's shrinkware more capable for penetration
> testing (a single machine) than a human?

Absolutely not. There is a constraint in developing shrinkwrap
vulnerability scanners that the code can not really change the
configuration of the target machine or have the potential to corrupt or
disable running services. Most of the checks in commercial software
vulnerability scanners don't actually exploit the problems, rather they
*identify* them. There are some vulnerabilities that *must* be exploited to
even know they are there, and in a few cases the commercial software
scanners will cross the line, so to speak, because it is the only way to
identify the problem. What commercial software scanners don't do is
actually exploit the machine and systematically rummage through it looking
for information that can be used to penetrate other machines (or networks).
There are certain isolated instances where the scanners make use of some
secondary information, but it is not nearly at the level a human can/will.
SATAN did some of this transitive trust following, but again it isn't
anywhere near what a human can do. Think of it this way: software scanners
tend to do a great job at finding the first layer of vulnerabilities, but a
human penetration team can possibly find N (N being affected by many things
including time, money, and they security of the tested system). That is why
human penetration teams often use software scanners in the initial phase,
because they do a good job of pointing out open doors. It is then up to the
pen team to go through them. If the penetration team you hire isn't going
through those doors, then you are getting ripped off and you should just
license the scanner yourself!

> I'll take one example of a tool, Internet Security Scanner.  It can do a
> complete external scan of the currently known vulnerabilities of a
> machine or subnet.  ISS is very consciensious of keeping up to date with
> vulnerabilities.

No matter what, the commercial vulnerability scanners will always be behind
the underground community. A good penetration team will have some
communication channel with the underground, and hopefully will be aware of
exploits that are not in the commercial scanners yet. Actually, penetration
teams are often one of the places the commercial scanner companies get
there exploit knowledge from. Again, the better penetration teams do their
own vulnerability research, and therefore are ahead of the curve in some
regards. Vulnerability research is part of my job. I am expected to add
value to our services above and beyond just running commercial software
scanners and cleaning up the report (though that is exactly what a lot of
"penetration teams" do).

> Some counter-points I have are:
>  o The human needs to do data collection about the target through
> whois, nslookup, search engines, anonymous or spoofed phone calls, etc.
>  o The human element still needs to select the targets, the connection
> path (dial-up, X.25, Internet, hops via private links, etc), the social
> engineering, the password crackers, etc.
>  o The human also needs to define the D.O.S. threshold of the target,
> and limits on brute force efforts.  
>  o The tests won't detect sniffers installed at the target's ISP.

Even with the best expert system or artificial intelligence technologies, a
software based system can not hack as well as a human (though with a few
million and a couple years, we could probably get pretty close ;).
Penetration testing pulls in a lot of diverse knowledge and experiences. It
is really simple sometimes, but other times it can be extremely complex. It
is not uncommon for our team to write new tools or attacks on the spot,
because we were able to recognize a vulnerability that consists of series
of issues with some previously unknown aspect to it. Believe me, if we
could write a tool that did everything we can, I would be writing it right
now so we could click a few buttons and all go to coffee - permanently ;)

> Say someone wants to do penetration testing and security auditing for a
> company, and use various types of shrinkware to do it.  Any comments?

Shrinkwrap vulnerability scanners *should* be *one* of the tools used in a
penetration test, but not the only thing used. If the auditor or
penetration team does not bring more to the table than a shrinkware tool
kit, then license the same software, run it yourself, and save a ton of
cash (actually, see if you can get their consulting rate minus 10% or
something ;).

IMHO,

Dominique Brezinski CISSP                   (612)628-5378
Secure Computing        http://www.securecomputing.com