BSIMM: Confessions of a Software Security Alchemist (informIT)

[jim at manico.net (Jim Manico)]

That's a bit of dodging the question, I'd like to hear more. You comment 
below implied that it was your consistent use of vendor-based static analyis 
tool that allowed you to figure out top N list of bugs for a specific 
organization. "Leading with static analysis" as your primary analysis driver 
concearns me. Will you elaborate, please?

- Jim

----- Original Message ----- 
From: "Gary McGraw" <gem at cigital.com>
To: "Jim Manico" <jim at manico.net>; "Steven M. Christey" 
<coley at linus.mitre.org>
Cc: "Sammy Migues" <SMigues at cigital.com>; "Dustin Sullivan" 
<dustin.sullivan at informit.com>; "Secure Code Mailing List" 
<SC-L at securecoding.org>
Sent: Thursday, March 19, 2009 9:04 AM
Subject: Re: [SC-L] BSIMM: Confessions of a Software Security Alchemist 
(informIT)


Actually no.  See: http://www.cigital.com/papers/download/j15bsi.pdf
(John Steven,   State of Application Assessment, IEEE S&P)

I am not a tool guy, I am a software security guy.

gem

http://www.cigital.com/~gem


On 3/19/09 2:58 PM, "Jim Manico" <jim at manico.net> wrote:

> Many of the top N lists we encountered were developed through the
> consistent use of static analysis tools.  After looking at millions of
> lines of code (sometimes constantly), a ***real*** top N list of bugs
> emerges for an organization.

You mean a "real list of what a certain vendors static analysis tools find".
If you think that list really measures the risk of an organizations software
security posture - that might ne considered to be insane! =)

- Jim

----- Original Message -----
From: "Gary McGraw" <gem at cigital.com>
To: "Steven M. Christey" <coley at linus.mitre.org>
Cc: "Sammy Migues" <SMigues at cigital.com>; "Dustin Sullivan"
<dustin.sullivan at informit.com>; "Secure Code Mailing List"
<SC-L at securecoding.org>
Sent: Wednesday, March 18, 2009 11:54 AM
Subject: Re: [SC-L] BSIMM: Confessions of a Software Security Alchemist
(informIT)


> Hi Steve,
>
> Many of the top N lists we encountered were developed through the
> consistent use of static analysis tools.  After looking at millions of
> lines of code (sometimes constantly), a ***real*** top N list of bugs
> emerges for an organization.  Eradicating number one is an obvious
> priority.  Training can help.  New number one...lather, rinse, repeat.
>
> Other times (like say in the one case where the study participant did not
> believe in static analysis for religious reasons) things are a bit more
> flip (and thus suffer from the "no data" problem I like to complain
> about).  I do not recall a case when the top N lists were driven by
> customers.
>
> Sorry I missed your talk at the SWA forum.  I'll chalk that one up to NoVa
> traffic.
>
> gem
>
> http://www.cigital.com/~gem
>
>
> On 3/18/09 5:47 PM, "Steven M. Christey" <coley at linus.mitre.org> wrote:
>
>
>
> On Wed, 18 Mar 2009, Gary McGraw wrote:
>
> > Because it is about building a top N list FOR A PARTICULAR ORGANIZATION.
> > You and I have discussed this many times.  The generic top 25 is
> > unlikely to apply to any particular organization.  The notion of using
> > that as a driver for software purchasing is insane.  On the other hand
> > if organization X knows what THEIR top 10 bugs are, that has real value.
>
> Got it, thanks.  I guessed as much.  Did you investigate whether the
> developers' personal top-N lists were consistent with what their customers
> cared about?  How did the developers go about selecting them?
>
> By the way, last week in my OWASP Software Assurance Day talk on the Top
> 25, I had a slide on the role of top-N lists in BSIMM, where I attempted
> to say basically the same thing.  This was after various slides that tried
> to emphasize how the current Top 25 is both incomplete and not necessarily
> fully relevant to a particular organization's needs.  So while the message
> may have been diluted during initial publication, it's being refined
> somewhat.
>
> - Steve
>
>
> _______________________________________________
> Secure Coding mailing list (SC-L) SC-L at securecoding.org
> List information, subscriptions, etc -
> http://krvw.com/mailman/listinfo/sc-l
> List charter available at - http://www.securecoding.org/list/charter.php
> SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
> as a free, non-commercial service to the software security community.
> _______________________________________________