IBM Acquires Ounce Labs, Inc.

[arian.evans at anachronic.com (Arian J. Evans)]

Chris -- Good point with Larry's paper. NTO Spider is, by design, a
simplified scanner for unskilled users, and I do not think it was
designed to be an effective tool for deep dynamic analysis of a web
application. It is, however, probably the best scanner on the market
for people who don't have the time or skill to configure dynamic
testing tools for their applications!

Larry Suto's paper reinforces this use-case desktop webapp scanners:

"Each scanner was run in default mode and not tuned in any capacity to
the application. The
importance of this lies in how effective the default mode so that
scalability of scanning is not limited
by manual intervention and setup procedures which can be very time
consuming. Second, in most
cases it is simply unrealistic to spend much time with many applications."


While I definitely look forward to more objective data on this
subject, I do not think Suto's report is a good example of how
consultants or SaaS providers deliver "expert security analysis" when
they use dynamic testing automation IMO. (I know this is not how I
ever did things.)

I think anyone who has experience with deep dynamic testing knows they
need automation tools with custom configuration ability, the ability
to record workflow, a framework to create custom tests, etc. I do not
believe NTO spider offers any of these essential features. I believe
it was explicitly designed for unskilled users' use-cases. (A valid
and important market to be sure.)

Admittedly I could be very wrong here. The NTO guys are sharp folks
and I haven't seen Spider in a while.


> As a group of security practitioners it is amazing to me that we don't have more quantifiable testing and 
> tools/services are just dismissed with anecdotal data.

Completely agreed. I prefer to back up my statements about dynamic
tools with hard, quantifiable data. Deprived of that, I tend to rely
on historical experience, if it is a subject I have enough experience
on within that problem domain.

If you recall I used to do extensive testing and benchmarking of
dynamic testing tools across custom widgets I wrote, and production
enterprise applications, and publish them @OWASP and NIST conferences,
and "HE: Webapps 2nd Ed". Back then the quality of the tools was very
volatile, and changed significantly every release, and from
application to application you would test, so by the time you vetted
all your data, it was almost obsolete. Again the importance of
customizable, manually guided tools, when dealing with new and bespoke
applications.

I tried to tackle static analysis but became overwhelmed by the
challenge of setting up effective labs, and the huge array of static
analysis tools that were available.

Given that I now work on a dynamic testing platform: it would be
completely fair to accuse me of being "non-objective" when discussing
various vendors dynamic testing tools -- and I would have to agree
with you. It won't make my statements any less valid, but I have to
throw that out there to be fair.


Ultimately you hit the need for objective data spot-on. I would be
lying if I didn't say that I would LOVE to see more head-on
benchmarking between static analysis technology vendors like Veracode,
Fortify, Ounce, Coverity, Klockwork, etc. etc.

The problem I had in the past with benchmarks was the huge degree of
customization in each application I would test. While patterns emerge
that are almost always automatable to some degree, the technologies
almost always require hand care-and-feeding to get them to an
effective place. I think this notion of combining the tools with
qualified users is the true potential power of the SaaS solutions that
are coming to market.

I look forward to seeing the release of more objective analysis by
smarter minds than I, and am very impressed with how far things have
come since the simple tests I tried to run over the years.

$0.02. Cheers,

-- 
Arian Evans




On Tue, Aug 4, 2009 at 5:54 PM, Chris Wysopal<cwysopal at veracode.com> wrote:
> I wouldn't say that NTO Spider is a "sort of" dynamic web scanner. It is a top tier scanner that can battle head to 
> head on false negative rate with the big conglomerates' scanners: IBM AppScan and HP WebInspect. ?Larry Suto 
> published an analysis a year ago, that certainly had some flaws (and was rightly criticized), but genuinely showed 
> all three to be in the same league. I haven't seen a better head-to-head analysis conducted by anyone. A little bird 
> whispered to me that we may see a new analysis by someone soon.
>
> As a group of security practitioners it is amazing to me that we don't have more quantifiable testing and 
> tools/services are just dismissed with anecdotal data. ?I am glad NIST SATE '09 will soon be underway and, at least 
> for static analysis tools, we will have unbiased independent testing. I am hoping for a big improvement over last 
> year. ?I especially like the category they are using for some flaws found as "valid but insignificant". Clearly they 
> are improving based on feedback from SATE '08.
>
> Veracode was the first company to offer static and dynamic (web) analysis, and we have been for 2 years (announced 
> Aug 8, 2007). ?We deliver it as a service. If you have a .NET or Java web app, you would cannot find a comparable 
> solution form a single vendor today.
>
> -Chris
>
> -----Original Message-----
> From: sc-l-bounces at securecoding.org [mailto:sc-l-bounces at securecoding.org] On Behalf Of Arian J. Evans
> Sent: Tuesday, July 28, 2009 1:41 PM
> To: Matt Fisher
> Cc: Kenneth Van Wyk; Secure Coding
> Subject: Re: [SC-L] IBM Acquires Ounce Labs, Inc.
>
> Right now, officially, I think that is about it. IBM, Veracode, and
> AoD (in Germany) claims they have this too.
>
> As Mattyson mentioned, Veracode only does static binary analysis (no
> source analysis). They offer "dynamic scanning" but I believe it is
> using NTO Spider IIRC which is a simplified scanner that targets
> unskilled users last I saw it.
>
> At one point I believe Veracode was in discussions with SPI to use WI,
> but since the Veracoders haunt this list I'll let them clarify what
> they use if they want.
>
> So IBM: soon.
>
> Veracode: sort-of.
>
> AoD: on paper
>
> And more to come in short order no doubt. I think we all knew this was
> coming sooner or later. Just a matter of "when".
>
> The big guys have a lot of bucks to throw at this problem if they want
> to, and pull off some really nice integrations. Be interesting to see
> what they do, and how useful the integrations really are to
> organizations.
>
> --
> Arian Evans
>
>
>
>
>
> On Tue, Jul 28, 2009 at 9:29 AM, Matt Fisher<matt at piscis-security.com> wrote:
> > Pretty much. Hp /spi has integrations as well but I don't recall devinspect ever being a big hit. ?Veracode does 
> > both as well as static binary but as asaas model. Watchfire had a RAD integration as well iirc but it clearly must 
> > not haved had the share ounce does.
> >
> > -----Original Message-----
> > From: Prasad Shenoy <prasad.shenoy at gmail.com>
> > Sent: July 28, 2009 12:22 PM
> > To: Kenneth Van Wyk <ken at krvw.com>
> > Cc: Secure Coding <SC-L at securecoding.org>
> > Subject: Re: [SC-L] IBM Acquires Ounce Labs, Inc.
> >
> >
> > Wow indeed. Does that makes IBM the only vendor to offer both Static
> > and Dynamic software security testing/analysis capabilities?
> >
> > Thanks & Regards,
> > Prasad N. Shenoy
> >
> > On Tue, Jul 28, 2009 at 10:19 AM, Kenneth Van Wyk<ken at krvw.com> wrote:
> > > Wow, big acquisition news in the static code analysis space announced today:
> > >
> > > http://news.prnewswire.com/DisplayReleaseContent.aspx?ACCT=104&STORY=/www/story/07-28-2009/0005067166&EDATE=
> > >
> > >
> > > Cheers,
> > >
> > > Ken
> > >
> > > -----
> > > Kenneth R. van Wyk
> > > KRvW Associates, LLC
> > > http://www.KRvW.com
> > >
> > > (This email is digitally signed with a free x.509 certificate from CAcert.
> > > If you're unable to verify the signature, try getting their root CA
> > > certificate at http://www.cacert.org -- for free.)
> > >
> > >
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > Secure Coding mailing list (SC-L) SC-L at securecoding.org
> > > List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
> > > List charter available at - http://www.securecoding.org/list/charter.php
> > > SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
> > > as a free, non-commercial service to the software security community.
> > > _______________________________________________
> > _______________________________________________
> > Secure Coding mailing list (SC-L) SC-L at securecoding.org
> > List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
> > List charter available at - http://www.securecoding.org/list/charter.php
> > SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
> > as a free, non-commercial service to the software security community.
> > _______________________________________________
> >
> > _______________________________________________
> > Secure Coding mailing list (SC-L) SC-L at securecoding.org
> > List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
> > List charter available at - http://www.securecoding.org/list/charter.php
> > SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
> > as a free, non-commercial service to the software security community.
> > _______________________________________________
>
> _______________________________________________
> Secure Coding mailing list (SC-L) SC-L at securecoding.org
> List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
> List charter available at - http://www.securecoding.org/list/charter.php
> SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
> as a free, non-commercial service to the software security community.
> _______________________________________________