The Static Analysis Market and You

[Dave Aitel <dave () immunityinc com>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

So OWASP was dominated by lots of talk from and about static code
analysis tools. I wandered around with a friend of mine at the various
booths (CodeSecure [1], Fortify[2], IBM AppScan[3], Ounce Labs) and
tried them all while listening to their sales pitches. My friend works
for a financial institution that was looking to integrate static
analysis into their code development process. Like many people, she
thought the marketing sounded good. Keep in mind, a lot of the
sponsors for OWASP were static analysis tool vendors, and the
"Industry Panel" was heavily in favor of static analysis tools (until
you talked to them off-stage).

Here's my thoughts:

1. The technology's capabilities does not match the marketing pitch -
ideally for my friend, the tools would find all the exploitable
vulnerabilities in your code and then you would fix them, re-run it,
and get a clean bill of health.

All the tools provide you an interface that purports to fit into this
workflow. None of them, however, work like that. One of the major
problems with the technology is that you have to be a super genius
code auditor to decide if the vulnerabilities are real or not.

Also annoyingly the false positive rate is enormous even when run
against the tiny test programs they are using to demo the tools with.
So you end up with a ten page list of "bugs" that you may or may not
be able to understand enough to fix. All the tools provide nice code
browsers and a graph of data flow to help you with this process, but
in practice it's not enough.

2. IBM has integrated their static analysis with AppScan's scanning
and with their own dynamic analysis to try to help triage
vulnerabilities. There aren't a lot of other companies with all these
technologies ready to go, so I'd expect HP to pick up one of the
static code analysis tools on the cheap to match this - Fortify would
be a good fit. IBM's coverage is not as broad as the other vendors
though, and they are mostly sticking to a J2EE application sweet spot.
(.Net is coming soon, etc).

3. Ounce has exposed a really nice .Net API so if you ARE a code
auditor as a professional you can do filtering and hard-core analysis
using their internal knowledge base and data structures.

4. CodeSecure did a pretty good job on the PHP app I looked at, and
was easy to use, but as far as I can tell is a bit of a newer player
than some of the others. Not that it matters when the field is this
crowded.


The Nist Survey
____________________________________________________________________________________________________________________

Anyways, market stuff aside, NIST did a survey[5] (and presented at
OWASP) of all the solutions they could get to play, and discovered
that they basically don't work (not their words). They said not to use
their survey to make decisions like that, but let me run down the
conclusions as I saw them based entirely on the 1 hr OWASP presentation:

1. Running the tools against 6 programs (3 Java, 3 C) generated 47000
distinct "vulns" - in some cases (tinyhttpd) generating about 80%
false positive rate. Imho anything over 5% makes the process unusable
by developers. They estimated 1 man year to do false/true positive
determination on the entire set - which is a LOT of time.

2. In many cases skilled engineers were unable to determine the false
positive/true positive of a particular vulnerability warning (or were
"wrong" about it). Imagine how well your developer team will do!

3. No 0days were found worth reporting  to the open source projects. (!!!)

Conclusions

___________________________________________________________________________________

Those are not good signs for the technology field as a whole. One
possibility is that more research dollars will flood into the space
and the technology will get better and live up to its marketing.
Another possibility is that no matter how much you spend, pure static
analysis can't do the things you want it to do (the IBM and to some
extent Fortify bet).

Which is it?

- -dave

[1] http://www.armorize.com/corpweb/en/products/codesecure
[2] http://www.fortify.com/
[3]
http://www.sdtimes.com/IBM_IMPLEMENTS_STATIC_ANALYSIS_IN_APPSCAN_UPDATE/About_ALM_and_SECURITY_and_IBM/32889
[4] http://www.ouncelabs.com/
[5] https://samate.nist.gov/index.php/SATE
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFI9LJetehAhL0gheoRAgVzAJ4qTwc4sJ1VEj1AGrDVyLf0QDz6gQCeIXsj
cmdI/sBsk225x5V4qIl5jPk=
=Y/wJ
-----END PGP SIGNATURE-----

_______________________________________________
Dailydave mailing list
Dailydave () lists immunitysec com
http://lists.immunitysec.com/mailman/listinfo/dailydave