Further Thoughts about Benchmarking

["Mark Curphey" <mark () curphey com>]

Wow, what a response....I never knew so many people wanted to see it be
done.  

If OWASP is to do it (and I can't think of a better place) I think there are
a few things we need to think about in order to do it properly, make it
fair, repeatable and open. It also needs some resources and part of the
reason for this mail is to see if anyone has any.

1. I initially thought people wanted to see app scanners benchmarked but it
seems that they also want to see App IDS's and other products. That's fine
but obviously we need to develop a benchmark platform for each technology.

2. In order for this to be fair and effective, we will need to build a
benchmarking platform. That would need to be fairly complex and stand up to
public and vendor scrutiny. I know several people that have used WebGoat to
test the scanners for instance and whilst its interesting I think there are
a lot more things you would want to know such as scalability (crawling the
100k and 500K sites), dealing with flash etc. So I think what we would need
to do is to define a set of requirements that we would like to be able to
test for each product line (scanners, IDS etc) and then build a benchmarking
platform that works for that. OWASP already has some code that can be
re-used (the WebGoat scorecard) but that's not a trivial task. 

3. Most (some) vendors prob won't want to be benchmarked. That's fine as I
am sure people will migrate to buying stuff that is a known quantity rather
than an unknown quantity and some have already come forward.  The way most
vendors currently do trial agreements is to not allow benchmarking results
to be shared so the offers of use ours (whilst appreciated) can not be
accepted.

I know we have some common criteria labs people on the list that could help
us with their experience of testing and labs as well so I think we can do
it. 

But.....if OWASP takes it on it could take forever. As you know by the fact
the site is down (no it was not hacked, someone unplugged our servers and
walked off with them (serious)). 

Are there any companies that are considering purchasing that would be
interested in diverting test dollars to the community to build such a
platform or is there anyone with deep pockets that would be interested in
funding a public benchmarking platform? If so for which technologies ?

Please contact myself off the list if you would like to discuss.