WebApp Sec mailing list archives
Further Thoughts about Benchmarking
From: "Mark Curphey" <mark () curphey com>
Date: Wed, 31 Mar 2004 18:27:55 -0500
Wow, what a response....I never knew so many people wanted to see it be done. If OWASP is to do it (and I can't think of a better place) I think there are a few things we need to think about in order to do it properly, make it fair, repeatable and open. It also needs some resources and part of the reason for this mail is to see if anyone has any. 1. I initially thought people wanted to see app scanners benchmarked but it seems that they also want to see App IDS's and other products. That's fine but obviously we need to develop a benchmark platform for each technology. 2. In order for this to be fair and effective, we will need to build a benchmarking platform. That would need to be fairly complex and stand up to public and vendor scrutiny. I know several people that have used WebGoat to test the scanners for instance and whilst its interesting I think there are a lot more things you would want to know such as scalability (crawling the 100k and 500K sites), dealing with flash etc. So I think what we would need to do is to define a set of requirements that we would like to be able to test for each product line (scanners, IDS etc) and then build a benchmarking platform that works for that. OWASP already has some code that can be re-used (the WebGoat scorecard) but that's not a trivial task. 3. Most (some) vendors prob won't want to be benchmarked. That's fine as I am sure people will migrate to buying stuff that is a known quantity rather than an unknown quantity and some have already come forward. The way most vendors currently do trial agreements is to not allow benchmarking results to be shared so the offers of use ours (whilst appreciated) can not be accepted. I know we have some common criteria labs people on the list that could help us with their experience of testing and labs as well so I think we can do it. But.....if OWASP takes it on it could take forever. As you know by the fact the site is down (no it was not hacked, someone unplugged our servers and walked off with them (serious)). Are there any companies that are considering purchasing that would be interested in diverting test dollars to the community to build such a platform or is there anyone with deep pockets that would be interested in funding a public benchmarking platform? If so for which technologies ? Please contact myself off the list if you would like to discuss.
Current thread:
- Further Thoughts about Benchmarking Mark Curphey (Mar 31)