Re: Of the three expensive vulnerability scanners

[Jeremiah Grossman <jeremiah () whitehatsec com>]

Jim,

You mention an interesting and overlooked challenge with regard to web 
application security scanning. Whenever you scan custom web code, you 
are naturally going to be prone to an annoying false-positive rate. 
Most of us familiar, already understand this problem and why it happens.

But, there is also the OTHER annoying problem of "duplicate" 
vulnerabilities. Where the same issue is reported several times. This 
problem is difficult to resolve in scanning products and hard to 
articulate why it actually occurs in the first place. This issue is 
also not something we're used to in other scanners. I'll do my best at 
describing some reasons why...

The same vulnerable form shows up on multiple pages and is not treated 
uniquely. The problematic parameter name may show up across several 
CGI's (like a template name field). Perhaps the URL always looks 
slightly different to the scanner because it has some dynamically 
changing session values. The URL structure could also be non-standard 
and input points hard to isolate. The scanner product may not group 
vulnerabilities by application, parameter, and class. There are several 
other reasons as well.

As we know, web applications are quite a diverse type of software. 
Effective generic techniques for scanning are difficult to create and 
apply. The good news is all the scanners do seem to be getting 
significantly better over time.


Regards,

Jeremiah-




On Saturday, November 13, 2004, at 01:22  PM, Jim+Lisa Weiler wrote:

> I've run WebInspect (WI, SPIDynamics) and AppScan (AS, Sanctum, now 
> Watchfire) against 2 large ecommerce site code bases, written in 
> IIS/ASP/VB and Apache/IBM Websphere - Websphere Commerce Server/JSP. 
> Both products seem equally comprehensive. AS is faster and Watchfire 
> says they do support better because they are bigger and have a support 
> organization. I found both organizations to be responsive. The 
> Watchfire folks didn't seem to know as much about the world outside 
> their product as the WI folks did. Another dimension that I found very 
> important to evaluate, is how useful the reporting is for planning 
> remediation projects and actually fixing the problems. Both products 
> might show 100 vulnerabilities. This might actually only involve 5 web 
> pages because they count a vulnerability as a single failed test on a 
> single field on 1 page. 5 failed XSS tests on the same field in the 
> same page will count as 5 vulnerabilities. If the XSS tests are in 
> different categories (HTML injection vrs unchecked parameter) they 
> will be in different parts of the reports. AS is very poor at telling 
> you
> 1. what are the different unique web pages (no repeated web pages) 
> with a vulnerability
> 2. for each vulnerability type, what pages does it occur on
>
> Just finding out exactly what different pages were involved took 2 
> hours looking at the AS report screen. They report the same page over 
> and over again in different places.
> WI is somewhat better and I haven't seen a version later than 5 months 
> ago.
>
> The pie charts, executive reports and spreadsheet exports don't help 
> planning the work to fix the problem.
> ----- Original Message ----- From: "Tom Stracener" <strace () gmail com>
> To: <webappsec () securityfocus com>
> Sent: Sunday, October 10, 2004 2:45 PM
> Subject: Re: Of the three expensive vulnerability scanners
>
>
> > In-Reply-To: <20041007153115.28058.qmail () www securityfocus com>
> >
> > Hi! I sought to answer this question for myself a while back, so 
> > hopefully you'll find my own experiences here useful. First, consider
> > the types of applications and the application environment you will be
> > securing. Depending upon the complexity of the web application you're
> > dealing with, your likely to get quick diminishing returns from the 
> > tools you have mentioned. Strong manual testing capabilities are a 
> > must, in my opinion, and sadly a lot of commercial apps fall short 
> > there.
> >
> > When possible, you should contact the vendors and acquire a demo 
> > license in order to get a feel for how a tool actually performs. If 
> > that's not available, then you should sit down with the vendors and 
> > get a hands on session.
> >
> > SPI Dynamics is very demo friendly. You'll find their people polite, 
> > professional, and quick to respond once you download the product. So 
> > if you want to take a look at it, just contact Natalie Hinkle 
> > <nhinkle () spidynamics com> if you have any questions or run into 
> > problems downloading it. Also, if you go this route be sure to 
> > download the SPI Toolkit, which includes some manual pen testing 
> > utilities.
> >
> > With Sanctum, acquiring a demo was more difficult, I had to speak with
> > the salesperson's manager and then wait a few days, only to be 
> > declined. Only after sending an email to their VP Internal Sales 
> > together with my resume did I manged to get a demo. You may have 
> > better results. Jane Foulkes <jfoulkes () sanctuminc com> is a sales 
> > person you can contact over there.
> >
> > Last I checked Scando did not have a demo available at all.
> >
> > I would also strongly encourage you to contact Cenzic and discuss 
> > having a look at their up and coming version of Hailstorm 2.0. Its by 
> > far the most extensible of the available commercial offerings. The 
> > tool provides a nice balance of automated verses manual app 
> > spidering, allows you to record and replay complicated HTTP sessions 
> > (which they call traversals) and then you can apply different types 
> > of security policies as Hailstorm iteratively steps through the web 
> > application. You can also create your own policies and have full 
> > control over the fault injectors which interrogate the app, as well 
> > as types of response conditions you're interested in detecting. This 
> > tool shows an incredible amount of promise, so it would probably be 
> > in your interest to evaluate it. You can contact Mandeep Khera over 
> > there <mandeep () cenzic com> if you're interested finding out more 
> > about it.
> >
> > Also, browse the recent archives of this list because your question
> > has surfaced in various forms and you'll be able to find a variety of
> > useful perspectives.
> >
> > --Tom