Re: Application Assessment

[Jeremiah Grossman <jeremiah () whitehatsec com>]

An independent body (such as NIST) creating a testing criteria may be  
what the doctor ordered. Provided they have the motivation,  
expertise, and industry support to do so. It just isn't the only  
option available to consider. The reality of the situation is  
developing an accepted testing criteria for a web application scanner  
is going to be VERY VERY difficult for a variety of reasons.

Today we understand that its possible to make a test-case website  
where a scanner could find just about every class of vulnerability.  
Or, a website where a scanner be incapable identifying anything and  
yet still be riddle with security issues. THIS is the challenge of  
scanning real world websites.  And what do we do about this to make  
the results meaningful? I would submit without directly involving  
those who live and breath auditing websites and designing scanners,  
the resulting value of the testing would be limited. And adoption as  
well.

In my opinion, if the testing criteria is developed in cooperation  
with competing vendors and industry experts, this would speak to the  
purity of the outcome. No one would let one vendor bias the scale  
towards them. WASC being a collection of vendors, security experts,  
independents, developers, consultants, etc. has release content that  
directly reflects this.

Remember, we're just talking about the criteria itself. The trust  
factor of the testing results is actually from the reputation of the  
tester.  And at the end of the day, who can say which is the workable  
model. Maybe both can be successful. The user support will be the  
deciding factor.


Regards,

Jeremiah-




On Aug 11, 2005, at 12:21 PM, Mark Curphey wrote:

> Criteria for an assessment tool should not be driven (or created)  
> by tools
> vendors but by must be created by an independent body. I know of  
> enough
> banks and large companies who would come together to define the  
> criteria and
> I heard about a NIST project that is relevant. NIST seems like the  
> only
> suitable candidate to me. This kind of credibility and independence  
> IMHO is
> what would produce credible results.
>
> I actually think its relatively easy to come up with a good set of  
> criteria.
> You need to mimic whats out there in the real world. Real world  
> sites range
> from small simple sites to large complex ones. Vulnerabilities  
> range from
> obvious simple ones to non-trivial complex ones. A testing  
> framework needs
> to mimic this and capture results in a repeatable and consistent  
> and fair
> manner. If everyone is testing against the same benchmark then  
> results are
> comparative anyways. Some tools will be better at some things that  
> others.
> That's valuable in itself.
>
> If the car industry created safety standards then we would have  
> rubber band
> seat belts ;-)
>
> -----Original Message-----
> From: Jeremiah Grossman [mailto:jeremiah () whitehatsec com]
> Sent: Thursday, August 11, 2005 2:28 PM
> To: Mark Curphey
> Cc: webappsec () securityfocus com
> Subject: Re: Application Assessment
>
> Not withstanding the mischaracterization of WASC  
> (www.webappsec.org), which
> is the consortium  I assume you were referring to, web application  
> scanner
> performance reviews would be a good thing for the community. In  
> fact at
> Black Hat I was speaking to a couple of the scanner vendors about  
> doing
> exactly that. The response I got was positive.
>
> The fundamental challenge is developing a fair and balanced  
> criteria in
> which to test the products. Web application scanners of the open  
> source and
> commercial variety have a largely differing features sets, including
> vulnerability identification capabilities. None of them are closely
> comparable and this inevitably skews results since there's no  
> baseline. No
> one wants to be treated unfairly by someone publishing negative and  
> biased
> performance reviews. I believe this is the primary concern on why  
> the trial
> agreements prevent publishing performance results.
>
> This isn't to say the initiative can't be done and the vendors  
> don't want it
> to be done, but the interested product participants would first  
> have to work
> together to develop a fair testing criteria.
> Without they're buy-in its never going to work.  WASC has a strong  
> working
> relationship with many web application security  industry experts and
> vendors to make this possible.
>
> In the web application firewall (WAF) world, WASC has begun this
> process:
>
>   "Web Application Firewall Evaluation Criteria" Project (WAFEC)  
> Led by Ivan
> Ristic, author of "Apache Security" and Mod_Security
> http://www.webappsec.org/projects/waf_evaluation/
>
> Top vendors and experts are working together to develop the industry
> standard testing criteria for evaluating the quality of web  
> application
> firewall solutions. The expectation is that anyone would be able to  
> use the
> criteria to evaluate a WAF product in a consistent manner. Whether  
> is be a
> customer, professional reviewer, vendors, consultant, etc.
>
> I expect the web application scanner guys to follow suit, its  
> really just a
> matter of when.
>
>
> Regards,
>
> Jeremiah Grossman-
>
>
> ---------------------------------------------------------------------
> The Web Security Mailing List
> http://www.webappsec.org/lists/websecurity/
>
> The Web Security Mailing List Archives
> http://www.webappsec.org/lists/websecurity/archive/
>
>
>
>
>
>
>
>
>
>
> On Aug 11, 2005, at 8:52 AM, Mark Curphey wrote:
>
>
> > Seems like it would be pretty valuable to publish an independent (not
> > by the vendors or the vendors consortium) review of performance the
> > web app scanners.  Last time I looked the trial agreements prevented
> > publication of comparisons and results. I know of a few magazines  
> > that
> > would be happy to publish the results and I would volunteer to
> > organize the testing.
> >
> > -----Original Message-----
> > From: Ory Segal [mailto:osegal () watchfire com]
> > Sent: Thursday, August 11, 2005 6:16 AM
> > To: goenw
> > Cc: pen-test () securityfocus com; Webappsec
> > Subject: RE: Application Assessment
> >
> >  Hi,
> >
> > You should also check: http://www.webappsec.org (Web Application
> > Security
> > Consortium)
> >
> > With regards to utilities, you can download the free Watchfire
> > Powertools (HTTP Proxy, HTTP request editor, etc.), here's the link:
> > http://www.watchfire.com/securityzone/download/default.aspx
> >
> > At the same link, you can also download eval versions of Watchfire's
> > AppScan product (An automated application security scanner).
> >
> > You can also find basic and advanced whitepapers on the subject at:
> > http://www.watchfire.com/news/whitepapers.aspx
> >
> > -Ory
> >
> >
> > -----Original Message-----
> > From: Glyn Geoghegan [mailto:glyng () corsaire com]
> > Sent: Thursday, August 11, 2005 4:48 AM
> > To: goenw
> > Cc: pen-test () securityfocus com; Webappsec
> > Subject: Re: Application Assessment
> >
> > On 8 Aug 2005, at 12:53, goenw wrote:
> >
> >
> >
> > > Hi,
> > >
> > > anybody have experience with application assessment ? I am a network
> > > guy, dont know much about the apps PT.
> > > 1. is there any tools that allow me to do the assessment throughly ?
> >
> > If you're talking web-applications, check out www.owasp.org for a
> > wealth of information on the subject.  You may also want to take a
> > look at the webappsec mailing list at www.securityfocus.com.
> >
> > Typically, the kind of tools you'll need are the personal-proxy
> > category, allowing you to intercept and modify communications between
> > the client and server - see Paros Proxy, Odysseus and Burp Proxy, for
> > example.
> >
> > There are fully automated tools, but in my personal experience the
> > manual approach has worked more effectively.
> >
> > Fat client/binary assessment is a slightly different (and arguably
> > more
> > complex) beast, and probably off-topic for this list.
> >
> >
> >
> > > 2. should i have external party conduct this, what are the things i
> > > should expect from them (success criteria) ?
> > > any comments are appriciated.
> >
> > That depends on how confident you are with your abilities, the  
> > drivers
> > for the assessment and a wealth of factors.  Normally, some coding or
> > development background is essential to identify and understand
> > potential vulnerabilities.
> >
> > Check out www.application-testing.com for our guide on the world of
> > Application Security Assessments.
> >
> > --
> > -------------------------------------------------------
> > G l y n   G e o g h e g a n                   BSc, ARCS
> > Principal Consultant                       Corsaire Ltd
> > 3 Tannery House, Tannery Lane
> > Send, Surrey, GU23 7EF, UK      UK: +44 (0)1483 226 000
> > http://www.corsaire.com        Fax: +44 (0)1483 226 001
> > -------------------------------------------------------
> >
> >
> >
> >
> > --------------------------------------------------------------------- 
> > -
> > ------
> > --
> > FREE WHITE PAPER - Wireless LAN Security: What Hackers Know That You
> > Don't
> >
> > Learn the hacker's secrets that compromise wireless LANs. Secure your
> > WLAN by understanding these threats, available hacking tools and
> > proven countermeasures. Defend your WLAN against man-in-the-Middle
> > attacks and session hijacking, denial-of-service, rogue access  
> > points,
> > identity thefts and MAC spoofing. Request your complimentary white
> > paper at:
> >
> > http://www.securityfocus.com/sponsor/AirDefense_pen-test_050801
> > --------------------------------------------------------------------- 
> > -
> > ------
> > ---