ddj: beyond the badnessometer

[nash at solace.net (Nash)]

On Thu, Jul 13, 2006 at 07:56:16AM -0400, Gary McGraw wrote:
> Is penetration testing good or bad?
> http://ddj.com/dept/security/189500001


Test coverage is an issue that penetration testers have to deal with,
without a doubt. Pen-tests can never test every possible attack
vector, which means that pen-tests can not always falsify a security
assertion.

Ok. But... 

First, pen-testers are highly active. The really good ones spend alot
of time in the hacker community keeping up with the latest attack
types, methods, and tools. Hence, the relevance of the test coverage
you get from a skilled pen-tester is actually quite good. In addition,
the tests run are similar to real attacks you're likely to see in the
wild. Also, pen-testsing is often intelligent, focused, and highly
motivated. After all, how would you like to have to go back to your
customer with a blank report? And, the recommendations you get can be
quite good because pen-testers tend to think about the entire
deployment environment, instead of just the code. So, they can help
you use technologies you already have to fix problems instead of
having to write lots and lots of new code to fix them. All of these
make pen-testing a valuable exercise for software environments to go
through.

Second, every software application in deployment has an associated
level of acceptable risk. In many cases, the level of acceptable risk
is low enough that penetration testing provides all the verificaton
capabilities needed. In some cases, the level of acceptable risk is
really low and even pen-testing is overkill. I do mostly code review
work these days, but I find that pen-testing has more general
applicability to my customers. There are exceptions, but not that
many.

Third, pen-tests also have real business advantages that don't
directly address risk mitigation. Pen-test reports are typically more
"down to earth." That is, they can be read more easily and the attacks
can be demonstrated more easily to business leaders, executives, and
other stakeholders. In my experience, recommendations from both
pen-tests and code reviews are commonly ignored. But, a good pen-test
gets the executive blood flowing in a way that code-oriented security
evaluations just don't.

Fourth, assertion falsification isn't always what you're after. Being
able to falsify the statement, "this app is secure enough," is a
common objective, but it's not really that useful for most businesses.
What exactly is secure enough? How do you define it? How do you
measure it?  How much accuracy do you need? How do you get more
accuracy, if you want it? How much do you trust your expert's opinion?

Sometimes, it's better to simply demonstrate a positive assertion,
such as:

    - This application is not subject to known, automatic attacks.
    - This application demonstrates the same security profile in all
      supported deployment environments.
    - This application demonstrates different security profiles,
      depending upon the deployment environment.
    - The latest MS patch does not affect the testable security
      profile of this application.

These are all assertions that pen-testing is arguably pretty good for
demonstrating. In some cases it might even be better than code
anlaysis--e.g., the effects of new environments or upgrades to
low-level libraries, virtual machines, operating systems.

Finally, my freind Sam pointed out that only during some kind of
pen-testing can you really identify what the actual attack surface of
an application looks like in its final deployment environment. This is
especially relevant in today's world because applications are now made
as much through integration of existing, off-the-shelf components as
through new development. A "new" application might only have a few
thousand lines of original code, but might be resting on top of a
software stack that has millions.  Whether it's J2EE, .NET, or LAMP,
all those environments are only really practical to test using some
form of pen-test.

Every security assessment methodology has its limits. Pen-testing has
limited falsification capabilities.  Code review, various kinds of
code analysis, unit testing, whatever else. These methods can all have
practical financial limitations and information accesibility problems. 

Of course, all these are good approaches and a wise security manager
will employ as wide a variety of assessment methods as he can afford
so that they compliment each other. But, affordability is a real
concern for most busineses and pen-testing is pretty affordable.

In the end, no assessment methodology produces results that are as
good as having a skilled Security Developer on your team during the
application design stage. Getting a security architecture in place
that matches your risk tolerance and functional requirements is the
single best way to prevent intrusions, bar none.


    nash e. foster
    Stratum Security, LLC


-- 

"the lyf so short, the craft so long to lerne."
                    - Geoffrey Chaucer