RE: What are we trying to "Benchmark" anyway? Report color, length, number of red exclamation points....

["Evans, Arian" <Arian.Evans () fishnetsecurity com>]

Hola Senior Eoin of the small words,

> -----Original Message-----
> From: Eoin Keary [mailto:eoinkeary () gmail com] 
> Sent: Thursday, October 06, 2005 3:48 AM

[...]
> We can evaluate which tools produced the most false positives and even
> worse False negatives (biggest problem IMHO).
> What we got was a comparison between the two tools and also a
> comparison between the tools and a human.

Who says your human "control" knows what they are doing?
(rhetorically speaking, not being rude)
 
> Without the third control (manual test) how do we know if the results
> are accurate for either tools?

This provides an unknown quality of control for testing. Humans are
fallible, non-omniscient, poor at repetitive tasks, and have tired eyes.
 
> In general tools are for "the easy stuff" they do not understand
> workflow or the logic of the app. They can not handle dynamic URL's:
> One app i tested recently with a number of tools got no results at all
> but after testing manually it had many XSS issues. These were
> exploited only by understanding the workflow.

I see this all the time. We probably all do. WebInspect, in your
example, is fairly good at doing the stuff I'm not, like checking
for every file with different extension types.

e.g.--global.asa.old or .bak in directories I wouldn't have thought
to check or didn't have the time to make the 10,000 GET requests.

Connection strings can potentially be just as or more valuable than
a limited/reflected XSS in an authenticated part of a web interface.

> I don't have any "big words" to explain this carry-on but blame the
> marketing staff who know much about very little and very little about

Yep. Sales and marketing. And the CISSP nonsense.

We need science. Scientific analysis of what the issues are, and
what it is we are testing for. I am hoping the NIST SAMATE project
puts us on this path.

I would love to tackle this issue myself, but it's beyond the limited
hours that I have free in life.

-ae