RE: Good benchmark application for web security testing tools?

["Evans, Arian" <Arian.Evans () fishnetsecurity com>]

This sounds like it will be a more effective approach than
most of what is out there now:

> -----Original Message-----
> From: Mark Curphey [mailto:mark () curphey com] 
> Sent: Thursday, October 06, 2005 10:27 AM
[...]
> Hacme Bank is now in Rev 2 (re-write including web services and new
[...]
> That said its not a good benchmarking tool for testing these 
> tools, nor is WebGoat.

exactly

> SiteGenerator however will be and is being specifically developed for

Will this be a public domain, open-source application? Who will
be making SiteGenerator available? Foundstone?

[...]
> We won't publish any results of tools themselves but the tool is 
> designed so people can do that against an environment that is like
> their own and not some canned site (I cant belive anyone would but
> based on results from a canned site built by a vendor of the product
> but .I guess some do.)

People can and do use vendor tautologies (in the definition of 'self-proving
frameworks') to validate their multi-$100,000 purchases of webappsec
scanners, WAFs, etc. I see this regularly.

Sad but true.

> I am sure people will share results in public.

If SiteGenerator can be coupled with rigorous definition and methods
for evaluation, this would be good.

If find some time, I'll post some recent webappscanner reviews and
point out why/where they are low quality and/or completely inaccurate.

The problem is getting worse, not better, right now. Awareness of the
issues is growing exponentially but *understanding* hasn't grown with it.

-ae