Re: Web Application Scanners Comparison

[anantasec <anantasec () googlemail com>]

> The vendors recommend configuration/tuning and it seems pretty
> appropriate given the nature of the tools.

Yes, in some cases you need to configure/tune the applications.
However, it would be a nightmare to configure/tune all the scanners
for all the websites and on the same time don't favor one or another.
The vendors will always find something to pick about.
And in this case there was not much to tune. I mean, I didn't used any
kind of authentication. So, I didn't had to care about login macros,
logout links and so on.
All the three scanners are performing an initial web
server/application information gathering and they usually have enough
information from that to be able to perform an unauthenticated scan.

However, if scanner 1 finds a vulnerability without any tuning and
scanner 2 doesn't, my conclusion is that something is wrong with
scanner 2. It could be bad defaults, it could be poor crawling, it
could be inconsistent scanning, it could be a lot of things.

> > > - What about the application coverage (not only links)? Maybe a tool
> > > didn't find a >vulnerability because it didn't cover this part of the
> > > application. Should it then get -5, since >it's a crawler problem?
> >
> > Yes, it should get a -5 if it didn't found a valid vulnerability. I
> > don't think it's important why it didn't found a vulnerability.
>
> Most people do care.

I also care. That's what a hacker does, it tries to figure out how
things are working.
However, I don't care in the current context, in the context of
evaluating which scanners found what vulnerability. In this comparison
I don't think I should care about that.

During my evaluation, I had to investigate every bug to confirm if
it's a false positive or not. It was a lot of work and I have a pretty
good idea why the scanners didn't found the vulnerabilities.
Most of the time it's not about tuning, it's about poor crawling or
bad JavaScript parsing or inconsistent scanning or just bugs.


> > > If a tool don't cover a part of the application and generates a
> > > false-negative, I don't think it >should count as much as if it cover the
> > > application and also generates a false-negative: >since you focus on
> > > rating the vulnerability finding, you have no idea what you are scoring
> > > > here -- the badness of the crawler/parser or the badness of the attack
> > > engine.
>
> I'm going to have to agree with Romain, especially on this point.
>
> Look, the basic premise is that web application security scanners work
> differently in different hands.  If you know what a breadth vs. depth
> search is... and know other tunables, then there is a totally
> different result.
>
> There is no comparatives for web application security scanners still.
> Web application security scanners are relatively useless in non-expert
> hands.  A seriously old-school, 5+ year experience person is required
> to run these tools to get any value outside of awareness.

I agree with you on this point. A tool is only as good as the person using it.

However, I repeat myself, if a tool has problems identifying a
vulnerability found by other scanners without tuning then my
conclusion is that something is wrong with that tool.
Maybe I'm thinking too simple but that's how I am.

> The purpose of running such a tool should be to get root-cause, which
> works best when source-code assisted with an advanced tool such as
> Dinis Cruz's O2 and realizing where O2 missed certain software
> weaknesses in order to hone into those specific areas with a
> functional fault-injection tool such as a web application security
> scanner and possibly a few semi-manual methods using tools like Burp
> Suite, flasm/flare/swfintruder, Firebug/Firecookie, and/or Sahi along
> with passive tools such as ProxMon, Pantera, ratproxy, Casaba Passive
> Web Security Auditor, and Skavenger.  A lot of this interaction is
> really application-specific, such as if Flash, Ajax, and other RIA or
> Widget technologies are in use, in addition to
> framework/language-specific.

I totally agree with you on this point.

-- 
http://anantasec.blogspot.com

-------------------------------------------------------------------------
Sponsored by: Watchfire 
Methodologies & Tools for Web Application Security Assessment 
With the rapid rise in the number and types of security threats, web application security assessments should be 
considered a crucial phase in the development of any web application. What methodology should be followed? What tools 
can accelerate the assessment process? Download this Whitepaper today! 

https://www.watchfire.com/securearea/whitepapers.aspx?id=70170000000940F
-------------------------------------------------------------------------