Re: Comparison report on web app security scanners

["Zaninotti, Thiago" <thiago () nstalker com>]

Concerning some of the points discussed, I may only add that nothing will 
substitute human capacity to detect so called logical vulnerabilities.

I agree with Grossman and Curphey in way that OWASP SiteGenerator may assist 
vendors on moving toward a better and effective solution on web app 
scanning. This is already helping out WAF vendors. In my point of view, 
vendors must not provide excuses, but ways to customize their solutions to 
assist users on obtaining the best scanning session possible. On some 
applications, obscure coding/scripting may become a dead alley to web 
scanners, however, a good tool will always add value to the final result (it 
should save time and money).

Thiago Zaninotti,Security+,CISSP-ISSAP,CISM
CTO - N-Stalker/ZMT
----- Original Message ----- 
From: "Mark Curphey" <mark () curphey com>
To: "'Jeremiah Grossman'" <jeremiah () whitehatsec com>; 
<webappsec () securityfocus com>
Sent: Wednesday, May 17, 2006 8:49 PM
Subject: RE: Comparison report on web app security scanners


> 100% agree.
>
> On your last point this is exactly why Dinis built OWASP SiteGenerator. 
> The
> architecture is designed in a way you can generate a site with your 
> specific
> type of site and compare / test.
>
> -----Original Message-----
> From: Jeremiah Grossman [mailto:jeremiah () whitehatsec com]
> Sent: Wednesday, May 17, 2006 12:24 PM
> To: webappsec () securityfocus com
> Subject: Re: Comparison report on web app security scanners
>
> I started giving presentations on this subject at Blackhat in 2004.
>
> "Challenges of Automated Web Application Scanning"
> http://www.whitehatsec.com/presentations/challenges_of_scanning.pdf
>
> To summarize....
>
> My company, WhiteHat Security, has been developing web application
> scanning technology for years. We use the technology to make our
> continuous vulnerability assessment service efficient.  After
> assessing thousands of websites, here are two things we've learned
> about scanning websites:
>
> - Its possible to design a website that CAN'T be scanned
> - Its possible to design a website where a scanner WILL find everything
>
> Websites such as these are edge cases. The rest of the worlds
> millions of websites fall somewhere in the middle where they are
> partially scannable. In my experience, scanners can only reliably
> TEST for about half of the possible vulnerabilities in a website.
> These observations have brought us to the following conclusions:
>
> - Scanners find vulnerabilities and save time.
> - Experiences security engineers must find the "logical flaws"
>
> Only by combining the two is it possible to achieve comprehensiveness
> and due diligence.
>
>
> As for a industry wide comparison of web application scanners,
> running them against a handful of staged websites is not going to
> yield compelling results. The only way I can see it done is stacking
> them side-by-side to see what they do support. Then anyone can run a
> demo on their own website to see if the solution works for their
> particular situation.
>
>
>
> Regards,
>
> Jeremiah Grossman
> Founder and CTO
> WhiteHat Security, Inc.
> www.whitehatsec.com
>
>
> On May 16, 2006, at 7:52 AM, Ory Segal wrote:
>
> > Hello,
> >
> > I would like to add several important comments to this thread, in
> > behalf
> > of Watchfire:
> >
> > According to tests done in Watchfire's labs, when using AppScan 6.0
> > SP2
> > + update 553 on the WebGoat application - AppScan will find 85 links,
> > will create 9557 tests and will eventually find 31 issues (211
> > different
> > test variants).
> >
> > People who perform such benchmarks against WebGoat should pay
> > attention
> > to the fact that AppScan needs some configuration in order to run
> > successfully on this application -
> >
> > - Explore method should be set to DFS (Depth First)
> > - Scan should be done in Single threaded mode
> > - Path limit should be disabled (no path limit)
> > - Depth limit should be enabled (otherwise one of the lessons gets
> > into
> > an infinite loop)
> > - HTTP Authentication - use guest/guest
> > - Add the "Screen" parameter to the black-list (untested parameters)
> > - Auto-form filler should be enabled
> >
> > (!!!) IMPORTANT: all of the above configuration items existed in
> > AppScan
> > for a long time, these were not added in order to "cook the
> > product" to
> > work properly on WebGoat.
> >
> > In addition I support what Acunetix mentioned, WebGoat and the
> > FoundStone (Hackme) banking applications are poor examples to be
> > testing
> > on.
> >
> > I am also quoting Mark Curphey (OWASP), regarding OWASP's WebGoat
> > project:
> > "That said its (i.e. Foundstone's HackMe bank) not a good benchmarking
> > tool for testing these tools, nor is WebGoat"  - taken from:
> > http://seclists.org/lists/webappsec/2005/Oct-Dec/0025.html
> >
> > Thank you very much,
> > -Ory Segal
> > Watchfire
> >
> >
> > -----Original Message-----
> > From: Bogdan Calin [mailto:bogdan () acunetix com]
> > Sent: Tuesday, May 16, 2006 17:10
> > To: webappsec () securityfocus com
> > Cc: Holger.Peine () iese fraunhofer de
> > Subject: Re: Comparison report on web app security scanners
> >
> > Hello,
> >
> > A few days ago Dr. Holger Peine published a "Comparison report on web
> > app security scanners".
> >
> > For this report he used two web applications: one of them is
> > WebGoat and
> > the other one is a proprietary application which is not public.
> >
> > I don't know anything about this proprietary application but I would
> > like to say that WebGoat is not a good test case for evaluating web
> > scanners.
> >
> > WebGoat is using server side state variables to track user actions.
> > For
> > example, if you want to test the String SQL injection flaw you first
> > need to navigate to the "String SQL injection" section in order to set
> > the proper state of the application.
> >
> > If the application is not in the proper state, the SQL injection test
> > will not work. The application will just ignore your inputs.
> >
> > An automated scanner cannot guess this application behavior, and
> > unless
> > you optimize your scanner for this particular application it will
> > not be
> > able to scan it properly.
> >
> > When the scanner has finished discovering the site structure, WebGoat
> > will be in some unknown state. All tests will be performed while
> > WebGoat
> > is in this state.
> >
> > This is not a common implementation.
> >
> > Because we are offering free audits, we have audited more than 1,000
> > websites and didn't encountered this kind of implementation.
> >
> > WebGoat is great for learning about web security flaws but I don't
> > think
> > it should be used as a test case for web security scanners.
> >
> > Bogdan Calin
> > Acunetix Ltd. - www.acunetix.com
> > Acunetix Web Vulnerability Scanner
> >
> > ---------------------------------------------------------------------- 
> > --
> > -
> > Sponsored by: Watchfire
> >
> > Watchfire named worldwide market share leader in web application
> > security assessment by leading market research firm. Watchfire's
> > AppScan
> > is the industry's first and leading web application security testing
> > suite, and the only solution to provide comprehensive remediation
> > tasks
> > at every level of the application. See for yourself.
> > Download a Free Trial of AppScan 6.0 today!
> >
> > https://www.watchfire.com/securearea/appscansix.aspx?
> > id=701300000007t9c
> > ---------------------------------------------------------------------- 
> > --
> > --
> >
> >
> >
> > ---------------------------------------------------------------------- 
> > ---
> > Sponsored by: Watchfire
> >
> > Watchfire named worldwide market share leader in web application
> > security
> > assessment by leading market research firm. Watchfire's AppScan is the
> > industry's first and leading web application security testing
> > suite, and
> > the only solution to provide comprehensive remediation tasks at every
> > level of the application. See for yourself.
> > Download a Free Trial of AppScan 6.0 today!
> >
> > https://www.watchfire.com/securearea/appscansix.aspx?
> > id=701300000007t9c
> > ---------------------------------------------------------------------- 
> > ----
>
>
> -------------------------------------------------------------------------
> Sponsored by: Watchfire
>
> Watchfire named worldwide market share leader in web application security
> assessment by leading market research firm. Watchfire's AppScan is the
> industry's first and leading web application security testing suite, and
> the only solution to provide comprehensive remediation tasks at every
> level of the application. See for yourself.
> Download a Free Trial of AppScan 6.0 today!
>
> https://www.watchfire.com/securearea/appscansix.aspx?id=701300000007t9c
> --------------------------------------------------------------------------
>
>
>
> -------------------------------------------------------------------------
> Sponsored by: Watchfire
>
> Watchfire named worldwide market share leader in web application security
> assessment by leading market research firm. Watchfire's AppScan is the
> industry's first and leading web application security testing suite, and
> the only solution to provide comprehensive remediation tasks at every
> level of the application. See for yourself.
> Download a Free Trial of AppScan 6.0 today!
>
> https://www.watchfire.com/securearea/appscansix.aspx?id=701300000007t9c
> --------------------------------------------------------------------------



-------------------------------------------------------------------------
Sponsored by: Watchfire

Watchfire named worldwide market share leader in web application security 
assessment by leading market research firm. Watchfire's AppScan is the 
industry's first and leading web application security testing suite, and 
the only solution to provide comprehensive remediation tasks at every 
level of the application. See for yourself. 
Download a Free Trial of AppScan 6.0 today!

https://www.watchfire.com/securearea/appscansix.aspx?id=701300000007t9c
--------------------------------------------------------------------------