Re: Application Assessment

["Amit Klein (AKsecurity)" <aksecurity () hotpop com>]

On 11 Aug 2005 at 12:57, Jeremiah Grossman wrote:

> Today we understand that its possible to make a test-case website  
> where a scanner could find just about every class of vulnerability.  
> Or, a website where a scanner be incapable identifying anything and  
> yet still be riddle with security issues. THIS is the challenge of  
> scanning real world websites.  And what do we do about this to make  
> the results meaningful? I would submit without directly involving  
> those who live and breath auditing websites and designing scanners,  
> the resulting value of the testing would be limited. And adoption as  
> well.

While I generally prefer to take cover when the titans clash ;-)   there's an angle here 
I'd like to shed light on:

Unlike the car industry (I think), in the web application scanning/security world, it is 
possible (once a test suite/benchmark is known) to patch the scanner to work perfectly for 
a given benchmark, yet at the same time not significantly improving it. 
In the car industry, when you test a car for a head-on crash, side crash, and so forth, you 
pretty much cover the main areas of interest. A car that gets high scores in all the tests 
is probably going to be more secure on the road. Period. In the web application security 
world, where each application is slightly different, the fact that a scanner performed 
better in a given benchmark doesn't mean it'll perform better in the field, especially if 
it was patched to perform well for the given benchmark. So any such attempt should, in my  
mind, consider how to handle this issue. One such avenue is to create a benchmark of  
hundreds of applications. This would make it both more real-life (a scanner that performs    
well on all applications is more likely to perform well in real life, as it is likely that  
the real life application will bear resemblence to one of the apps in the benchmark), and  
harder to patch for...