Re: InformIT: comparing static analysis tools

["Arian J. Evans" <arian.evans () anachronic com>]

That is a great question. According to Gartner, HA has the stench of
inevitability. And in general, I agree.

There are cases where dynamic and static each have clear strengths.
Pragmatic combination of of the two has promise is solving a broad
spectrum of test-cases. Additionally -HA can help improve each other
by improving context, but developing the underlying technology to make
that happen is non-trivial.

This is my guess as to how things will unfold:

Current HA attempts are at the vuln-mashup phase. Let's call this "correlation".

FP reduction: the next step that folks are working on in HA is
"suppression therapy". e.g- using correlation to filter and suppress
false-positives, increase signal-to-noise in output from both analysis
types.

FN reduction: HA has the promise of heatmapping coverage of both
static and dynamic testing. This would more fully allow the expert
running the solution to see what is and isn't getting covered. This
provides a better notion of False Negatives, and allow targeted tuning
and optimization. Or decide where best to focus expert human review
efforts.

Contextualization: The holy grail of HA would be to automatically have
both types of automation feed and tune each other. Black box would be
significantly enhanced by being feed framework config files, and
getting access to things like function names/parameters and objects
that are not directly exposed. This would really help dynamic on MVC
testing. Likewise, I expect dynamic testing could provide some notion
of design or control-flow back to the static engine to enhance static
authentication and authorization analysis. This would also help solve
for mobile: static could extract calls and functions from mobile
binaries, and dynamic could test the back-end web services they talk
to more effectively with that static context.

Context enhancement via HA, however, is kind of the holy grail of HA.
While it sounds great in theory, the complexity bar is high enough it
may be a long time coming.

As development shifts to more modular code on top of "platforms"
(iphone, xbox, rails, etc.) this is also driving interest in
lightweight solutions that can scan modular bits of code. Given that,
I think there is room for a very simplified, streamlined type of HA to
provide simple SAST that can feed a DAST unit-test type capability.
This is probably more realistic to build than the Ultimate Context
Integration Engine idea mentioned above. The more the world moves
towards coding in this manner, the more a solution like this make
sense. You would miss a lot, but it should be lightweight and actually
work.

For now though - the HA options boil down to mashups, and whether or
not suppression therapy is right for you.

We will see where it goes next...

---
Arian Evans
Software Security Scanning Snob


On Fri, Feb 4, 2011 at 2:21 PM, Prasad N Shenoy <prasad.shenoy () gmail com> wrote:
> Yeah, clear the "cloud" of confusion before talking about the cloud so to
> speak. Not all SaaS offerings available today qualify to be cloud based.
> Well, this thread got morphed into a cloudy discussion. Attempting to get
> back on track, I would say IMHO, it's subjective whether the static analysis
> or dynamic analysis (pen testing/bb testing) technologies have hit the wall
> - depends on who you ask. There is some element of saturation there I
> believe else the industry (term very generously used here)won't be focusing
> on things like Hybrid Analysis. Having said that, what's the future of HA?
> Sent from my iPhone
> On Feb 4, 2011, at 12:27 PM, Ben Laurie <benl () google com> wrote:
>
>
>
> On 4 February 2011 09:22, Chris Wysopal <cwysopal () veracode com> wrote:
> > “Breaking news.  Google says not to use the cloud.  Improving on-premise
> > tools is the future.”
>
> My view is personal. However, in general, whether the cloud is a good place
> for your data depends on your data and the relationship you have with the
> cloud provider. If your boss says "no, you can't push this stuff outside our
> network" then clearly the cloud is not the right answer (or your boss
> doesn't understand the problem).
>
> > Sorry, I couldn’t help myself. J
> >
> >
> >
> > -Chris
> >
> >
> >
> > From: Ben Laurie [mailto:benl () google com]
> > Sent: Friday, February 04, 2011 11:34 AM
> > To: Jim Manico
> > Cc: Chris Wysopal; Secure Code Mailing List
> > Subject: Re: [SC-L] InformIT: comparing static analysis tools
> >
> >
> >
> >
> >
> > On 3 February 2011 16:02, Jim Manico <jim.manico () owasp org> wrote:
> >
> > Chris,
> >
> > I've tried to leverage Veracode in recent engagements. Here is how the
> > conversation went:
> >
> > Jim:
> > "Boss, can I upload all of your code to this cool SaaS service for
> > analysis?"
> >
> > Client:
> > "Uh no, and next time you ask, I'm having you committed".
> >
> > I'm sure you have faced these objections before. How do you work around
> > them?
> >
> >
> >
> > Don't use SaaS, obviously.
> >
> >
> >
> > I'd rather see LLVM's static analysis tools get improved (the framework,
> > btw, is really nice to work with).
> >
> >
> >
> > -Jim Manico
> > http://manico.net
> >
> > On Feb 3, 2011, at 1:54 PM, Chris Wysopal <cwysopal () veracode com> wrote:
> >
> > > Nice article.  In the 5 years Veracode has been selling static analysis
> > > services we have seen the market mature.  In the beginning, organizations
> > > were down in the weeds. "What false positive rate or false negative rate
> > > does the tool/service have over a test suite such as SAMATE."  Then we saw a
> > > move up to looking at the trees.  "Did the tool/service support the Java
> > > frameworks I am using?"  Now we are seeing organizations look at the forest.
> > > "Can I scale static analysis effectively over all my development sites, my
> > > outsourcers, and vendors?"  This is a good sign of a maturing market.
> > >
> > > It is my firm belief that software security has a consumption problem.
> > >  We know what the defects are.  We know how to fix them.  We even have
> > > automation for detecting a lot of them.  The problem is getting the
> > > information and technology to the right person at the right time effectively
> > > and managing an organization-wide program.  This is the next challenge for
> > > static analysis. <bias-alert>I think SaaS based software is more easily
> > > consumed and this isn't any different for software security</bias-alert>
> > >
> > > -Chris
> > >
> > > -----Original Message-----
> > > From: sc-l-bounces () securecoding org
> > > [mailto:sc-l-bounces () securecoding org] On Behalf Of Gary McGraw
> > > Sent: Wednesday, February 02, 2011 9:49 AM
> > > To: Secure Code Mailing List
> > > Subject: [SC-L] InformIT: comparing static analysis tools
> > >
> > > hi sc-l,
> > >
> > > John Steven and I recently collaborated on an article for informIT.  The
> > > article is called "Software [In]security: Comparing Apples, Oranges, and
> > > Aardvarks (or, All Static Analysis Tools Are Not Created Equal)" and is
> > > available here:
> > > http://www.informit.com/articles/article.aspx?p=1680863
> > >
> > > Now that static analysis tools like Fortify and Ounce are hitting the
> > > mainstream there are many potential customers who want to compare them and
> > > pick the best one.  We explain why that's more difficult than it sounds at
> > > first and what to watch out for as you begin to compare tools.  We did this
> > > in order to get out in front of "test suites" that purport to work for tool
> > > comparison.  If you wonder why such suites may not work as advertised, read
> > > the article.
> > >
> > > Your feedback is welcome.
> > >
> > > gem
> > >
> > > company www.cigital.com
> > > podcast www.cigital.com/silverbullet
> > > blog www.cigital.com/justiceleague
> > > book www.swsec.com
> > >
> > > _______________________________________________
> > > Secure Coding mailing list (SC-L) SC-L () securecoding org List
> > > information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
> > > List charter available at - http://www.securecoding.org/list/charter.php
> > > SC-L is hosted and moderated by KRvW Associates, LLC
> > > (http://www.KRvW.com) as a free, non-commercial service to the software
> > > security community.
> > > Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates
> > > _______________________________________________
> > >
> > > _______________________________________________
> > > Secure Coding mailing list (SC-L) SC-L () securecoding org
> > > List information, subscriptions, etc -
> > > http://krvw.com/mailman/listinfo/sc-l
> > > List charter available at - http://www.securecoding.org/list/charter.php
> > > SC-L is hosted and moderated by KRvW Associates, LLC
> > > (http://www.KRvW.com)
> > > as a free, non-commercial service to the software security community.
> > > Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates
> > > _______________________________________________
> >
> > _______________________________________________
> > Secure Coding mailing list (SC-L) SC-L () securecoding org
> > List information, subscriptions, etc -
> > http://krvw.com/mailman/listinfo/sc-l
> > List charter available at - http://www.securecoding.org/list/charter.php
> > SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
> > as a free, non-commercial service to the software security community.
> > Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates
> > _______________________________________________
>
> _______________________________________________
> Secure Coding mailing list (SC-L) SC-L () securecoding org
> List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
> List charter available at - http://www.securecoding.org/list/charter.php
> SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
> as a free, non-commercial service to the software security community.
> Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates
> _______________________________________________
>
> _______________________________________________
> Secure Coding mailing list (SC-L) SC-L () securecoding org
> List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
> List charter available at - http://www.securecoding.org/list/charter.php
> SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
> as a free, non-commercial service to the software security community.
> Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates
> _______________________________________________

_______________________________________________
Secure Coding mailing list (SC-L) SC-L () securecoding org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates
_______________________________________________