RE: Good benchmark application for web security testing tools?

["Mark Curphey" <mark () curphey com>]

I met for a very enjoyable brunch with Dinis and family yesterday and we
basically agreed that Foundstone will sponsor Dinis's development to
complete it as an OWASP project so it will be an free open source OWASP
tool. From what I heard yesterday I think it will rock !  Dinis was hoping
to demo an early version at the OWASP conference this week.

-----Original Message-----
From: Evans, Arian [mailto:Arian.Evans () fishnetsecurity com] 
Sent: Friday, October 07, 2005 10:45 PM
To: postmaster () cnchost com
Cc: Mark Curphey
Subject: RE: Good benchmark application for web security testing tools? 

This sounds like it will be a more effective approach than most of what is
out there now:

> -----Original Message-----
> From: Mark Curphey [mailto:mark () curphey com]
> Sent: Thursday, October 06, 2005 10:27 AM
[...]
> Hacme Bank is now in Rev 2 (re-write including web services and new
[...]
> That said its not a good benchmarking tool for testing these tools, 
> nor is WebGoat.

exactly

> SiteGenerator however will be and is being specifically developed for

Will this be a public domain, open-source application? Who will be making
SiteGenerator available? Foundstone?

[...]
> We won't publish any results of tools themselves but the tool is 
> designed so people can do that against an environment that is like 
> their own and not some canned site (I cant belive anyone would but 
> based on results from a canned site built by a vendor of the product 
> but .I guess some do.)

People can and do use vendor tautologies (in the definition of 'self-proving
frameworks') to validate their multi-$100,000 purchases of webappsec
scanners, WAFs, etc. I see this regularly.

Sad but true.

> I am sure people will share results in public.

If SiteGenerator can be coupled with rigorous definition and methods for
evaluation, this would be good.

If find some time, I'll post some recent webappscanner reviews and point out
why/where they are low quality and/or completely inaccurate.

The problem is getting worse, not better, right now. Awareness of the issues
is growing exponentially but *understanding* hasn't grown with it.

-ae