Re: Solutions, Results, and Comments - Was [ISA Server and SQL Injection]

[Jeremiah Grossman <jeremiah () whitehatsec com>]

On Wednesday, February 23, 2005, at 09:36  PM, Jeff Williams wrote:

> There are essentially four ways of finding flaws in an application:
>    - Vulnerability scanning with signatures
>    - Manual penetration testing
>    - Static analysis of source or binary
>    - Manual code review


In your description, aside from automation,  what is the difference 
between bullet 1 and bullet 2? I've view the testing process as 3 
fundamental methods. Automated "scanners" can be applied to each 
method, but essentially they are performing the same process we'd do by 
hand.

- black-box testing (functional testing)
- static binary analysis
- source code review


> Any application review that relies on a single technique is going to 
> be less cost-effective than one that uses a combination. And when I 
> say "cost effective" I mean more serious flaws found for the $. The 
> 80-20 rule is nice, but you've got to find the right 80%.  Finding 
> 4000 minor flaws and missing a major obvious hole doesn't help 
> anything.

"Find the right 80%". This is really good way to look at it.

> Just so I'm totally clear here -- no matter how much you're planning 
> to spend on finding vulnerabilities in your application, you're better 
> off including some scanning, some code review, some static analysis, 
> and some penetration testing.  Skilled analysts with a full toolbox of 
> techniques is what produces the best bang for the buck.

Well said and I think people are leaning that direction. The challenge 
customers face is how much of what method to use and when. This answer 
is not readily apparent and certainly not agreed upon in the industry. 
We've seen people on the list comment on how they do (or have done) 
secure software in the past. A process that made sense to them.  It 
always appears to be a combination... but the way the process is 
implemented is always a bit different. This an area within the industry 
I see lacking good information. Assisting customers to find the right 
balance.

Also, I haven't come across much of anyone doing scans on binaries, but 
I am in the web app sec world)

> Anyone claiming that a single technique can find nearly all (if not 
> all)vulnerabilities or costs less than another technique just doesn't 
> get it.  Savvy consumers will look for reviewers that use best of 
> breed scanning and static analysis tools.  They'll also look for 
> serious pentest and code review expertise with the technologies 
> they're using.

One testing method might actually find all the vulnerabilities (bugs), 
but its not something you want to rely upon. The premise is 
theoretically code can free of bugs, we just have no way to prove it. 
Standard Turing logic. Similarly to the way we diversify our investment 
portfolio to provide safe and consistent results. Its no different with 
security, we should diversify our solutions and go for defense in depth.


Regards,

Jeremiah-