WebApp Sec mailing list archives
Re: Comparison report on web app security scanners
From: Jeremiah Grossman <jeremiah () whitehatsec com>
Date: Wed, 17 May 2006 09:24:00 -0700
I started giving presentations on this subject at Blackhat in 2004. "Challenges of Automated Web Application Scanning" http://www.whitehatsec.com/presentations/challenges_of_scanning.pdf To summarize....My company, WhiteHat Security, has been developing web application scanning technology for years. We use the technology to make our continuous vulnerability assessment service efficient. After assessing thousands of websites, here are two things we've learned about scanning websites:
- Its possible to design a website that CAN'T be scanned - Its possible to design a website where a scanner WILL find everythingWebsites such as these are edge cases. The rest of the worlds millions of websites fall somewhere in the middle where they are partially scannable. In my experience, scanners can only reliably TEST for about half of the possible vulnerabilities in a website. These observations have brought us to the following conclusions:
- Scanners find vulnerabilities and save time. - Experiences security engineers must find the "logical flaws"Only by combining the two is it possible to achieve comprehensiveness and due diligence.
As for a industry wide comparison of web application scanners, running them against a handful of staged websites is not going to yield compelling results. The only way I can see it done is stacking them side-by-side to see what they do support. Then anyone can run a demo on their own website to see if the solution works for their particular situation.
Regards, Jeremiah Grossman Founder and CTO WhiteHat Security, Inc. www.whitehatsec.com On May 16, 2006, at 7:52 AM, Ory Segal wrote:
Hello,I would like to add several important comments to this thread, in behalfof Watchfire:According to tests done in Watchfire's labs, when using AppScan 6.0 SP2+ update 553 on the WebGoat application - AppScan will find 85 links,will create 9557 tests and will eventually find 31 issues (211 differenttest variants).People who perform such benchmarks against WebGoat should pay attentionto the fact that AppScan needs some configuration in order to run successfully on this application - - Explore method should be set to DFS (Depth First) - Scan should be done in Single threaded mode - Path limit should be disabled (no path limit)- Depth limit should be enabled (otherwise one of the lessons gets intoan infinite loop) - HTTP Authentication - use guest/guest - Add the "Screen" parameter to the black-list (untested parameters) - Auto-form filler should be enabled(!!!) IMPORTANT: all of the above configuration items existed in AppScan for a long time, these were not added in order to "cook the product" towork properly on WebGoat. In addition I support what Acunetix mentioned, WebGoat and theFoundStone (Hackme) banking applications are poor examples to be testingon. I am also quoting Mark Curphey (OWASP), regarding OWASP's WebGoat project: "That said its (i.e. Foundstone's HackMe bank) not a good benchmarking tool for testing these tools, nor is WebGoat" - taken from: http://seclists.org/lists/webappsec/2005/Oct-Dec/0025.html Thank you very much, -Ory Segal Watchfire -----Original Message----- From: Bogdan Calin [mailto:bogdan () acunetix com] Sent: Tuesday, May 16, 2006 17:10 To: webappsec () securityfocus com Cc: Holger.Peine () iese fraunhofer de Subject: Re: Comparison report on web app security scanners Hello, A few days ago Dr. Holger Peine published a "Comparison report on web app security scanners".For this report he used two web applications: one of them is WebGoat andthe other one is a proprietary application which is not public. I don't know anything about this proprietary application but I would like to say that WebGoat is not a good test case for evaluating web scanners.WebGoat is using server side state variables to track user actions. Forexample, if you want to test the String SQL injection flaw you first need to navigate to the "String SQL injection" section in order to set the proper state of the application. If the application is not in the proper state, the SQL injection test will not work. The application will just ignore your inputs.An automated scanner cannot guess this application behavior, and unless you optimize your scanner for this particular application it will not beable to scan it properly. When the scanner has finished discovering the site structure, WebGoatwill be in some unknown state. All tests will be performed while WebGoatis in this state. This is not a common implementation. Because we are offering free audits, we have audited more than 1,000 websites and didn't encountered this kind of implementation.WebGoat is great for learning about web security flaws but I don't thinkit should be used as a test case for web security scanners. Bogdan Calin Acunetix Ltd. - www.acunetix.com Acunetix Web Vulnerability Scanner---------------------------------------------------------------------- --- Sponsored by: Watchfire Watchfire named worldwide market share leader in web applicationsecurity assessment by leading market research firm. Watchfire's AppScanis the industry's first and leading web application security testingsuite, and the only solution to provide comprehensive remediation tasksat every level of the application. See for yourself. Download a Free Trial of AppScan 6.0 today!https://www.watchfire.com/securearea/appscansix.aspx? id=701300000007t9c ---------------------------------------------------------------------- -------------------------------------------------------------------------- ---Sponsored by: WatchfireWatchfire named worldwide market share leader in web application securityassessment by leading market research firm. Watchfire's AppScan is theindustry's first and leading web application security testing suite, andthe only solution to provide comprehensive remediation tasks at every level of the application. See for yourself. Download a Free Trial of AppScan 6.0 today!https://www.watchfire.com/securearea/appscansix.aspx? id=701300000007t9c ---------------------------------------------------------------------- ----
------------------------------------------------------------------------- Sponsored by: WatchfireWatchfire named worldwide market share leader in web application security assessment by leading market research firm. Watchfire's AppScan is the industry's first and leading web application security testing suite, and the only solution to provide comprehensive remediation tasks at every level of the application. See for yourself. Download a Free Trial of AppScan 6.0 today!
https://www.watchfire.com/securearea/appscansix.aspx?id=701300000007t9c --------------------------------------------------------------------------
Current thread:
- Comparison report on web app security scanners Holger.Peine (May 05)
- <Possible follow-ups>
- Re: Comparison report on web app security scanners Bogdan Calin (May 16)
- RE: Comparison report on web app security scanners Mark Curphey (May 16)
- Re: Comparison report on web app security scanners Dean H. Saxe (May 18)
- Re: Comparison report on web app security scanners Bogdan Calin (May 18)
- RE: Comparison report on web app security scanners Mark Curphey (May 16)
- RE: Comparison report on web app security scanners Holger.Peine (May 16)
- RE: Comparison report on web app security scanners Ory Segal (May 16)
- Re: Comparison report on web app security scanners Jeremiah Grossman (May 17)
- RE: Comparison report on web app security scanners Mark Curphey (May 18)
- Re: Comparison report on web app security scanners Zaninotti, Thiago (May 18)
- Re: Comparison report on web app security scanners Jeremiah Grossman (May 17)
- Re: Comparison report on web app security scanners Eoin (May 17)
- RE: Comparison report on web app security scanners Mark Curphey (May 17)
- RE: Comparison report on web app security scanners Bogdan Calin (May 18)
- Re: Comparison report on web app security scanners solutions_PHP (May 18)
- Re: Comparison report on web app security scanners Bogdan Calin (May 18)
- RE: Comparison report on web app security scanners Mark Curphey (May 19)
- WAF learning ability limitation? matt farey (May 19)
- Re: Comparison report on web app security scanners solutions_PHP (May 19)