Re: Solutions, Results, and Comments - Was [ISA Server and SQL Injection]

["Jeff Williams" <jeff.williams () aspectsecurity com>]

> Automated "scanners" can be applied to each method, but essentially they 
> are performing the same
> process we'd do by hand.

Scanners *can't* perform "the same process we'd do by hand."  This is 
because a skilled penetration tester can learn how an application works and 
adjust.  Scanners can't.  The vast majority of serious (i.e. high risk) 
problems we've found over the past 7 years could *never* have been 
identified by a scanner, even though many were fairly obvious to a human 
tester.

> Also, I haven't come across much of anyone doing
> scans on binaries, but I am in the web app sec world)

If you are in the web app sec world and you aren't scanning binaries, you're 
wasting your customer's time and money.  Scanning binaries is (in some ways) 
easier than scanning source code because the compiler has done much of the 
difficult parsing work already -- meaning you don't have to replicate the 
customer's build environment.  For example, scanning binaries will allow you 
to find potential SQL injection points (and more importantly, non-SQL 
injection points) in seconds. You'll immediately see whether the web 
application uses dangerous libraries and methods. You're missing out on an 
important source of information.

--Jeff

----- Original Message ----- 
From: "Jeremiah Grossman" <jeremiah () whitehatsec com>
To: "Jeff Williams" <jeff.williams () aspectsecurity com>
Cc: <webappsec () securityfocus com>
Sent: Monday, February 28, 2005 11:28 AM
Subject: Re: Solutions, Results, and Comments - Was [ISA Server and SQL 
Injection]


> On Wednesday, February 23, 2005, at 09:36  PM, Jeff Williams wrote:
>
> > There are essentially four ways of finding flaws in an application:
> >    - Vulnerability scanning with signatures
> >    - Manual penetration testing
> >    - Static analysis of source or binary
> >    - Manual code review
>
>
> In your description, aside from automation,  what is the difference 
> between bullet 1 and bullet 2? I've view the testing process as 3 
> fundamental methods. Automated "scanners" can be applied to each method, 
> but essentially they are performing the same process we'd do by hand.
>
> - black-box testing (functional testing)
> - static binary analysis
> - source code review
>
>
> > Any application review that relies on a single technique is going to be 
> > less cost-effective than one that uses a combination. And when I say 
> > "cost effective" I mean more serious flaws found for the $. The 80-20 
> > rule is nice, but you've got to find the right 80%.  Finding 4000 minor 
> > flaws and missing a major obvious hole doesn't help anything.
>
> "Find the right 80%". This is really good way to look at it.
>
> > Just so I'm totally clear here -- no matter how much you're planning to 
> > spend on finding vulnerabilities in your application, you're better off 
> > including some scanning, some code review, some static analysis, and some 
> > penetration testing.  Skilled analysts with a full toolbox of techniques 
> > is what produces the best bang for the buck.
>
> Well said and I think people are leaning that direction. The challenge 
> customers face is how much of what method to use and when. This answer is 
> not readily apparent and certainly not agreed upon in the industry. We've 
> seen people on the list comment on how they do (or have done) secure 
> software in the past. A process that made sense to them.  It always 
> appears to be a combination... but the way the process is implemented is 
> always a bit different. This an area within the industry I see lacking 
> good information. Assisting customers to find the right balance.
>
> Also, I haven't come across much of anyone doing scans on binaries, but I 
> am in the web app sec world)
>
> > Anyone claiming that a single technique can find nearly all (if not 
> > all)vulnerabilities or costs less than another technique just doesn't get 
> > it.  Savvy consumers will look for reviewers that use best of breed 
> > scanning and static analysis tools.  They'll also look for serious 
> > pentest and code review expertise with the technologies they're using.
>
> One testing method might actually find all the vulnerabilities (bugs), but 
> its not something you want to rely upon. The premise is theoretically code 
> can free of bugs, we just have no way to prove it. Standard Turing logic. 
> Similarly to the way we diversify our investment portfolio to provide safe 
> and consistent results. Its no different with security, we should 
> diversify our solutions and go for defense in depth.
>
>
> Regards,
>
> Jeremiah-