BSIMM: Confessions of a Software Security Alchemist (informIT)

[gem at cigital.com (Gary McGraw)]

Hi Steve,

Many of the top N lists we encountered were developed through the consistent use of static analysis tools.  After 
looking at millions of lines of code (sometimes constantly), a ***real*** top N list of bugs emerges for an 
organization.  Eradicating number one is an obvious priority.  Training can help.  New number one...lather, rinse, 
repeat.

Other times (like say in the one case where the study participant did not believe in static analysis for religious 
reasons) things are a bit more flip (and thus suffer from the "no data" problem I like to complain about).  I do not 
recall a case when the top N lists were driven by customers.

Sorry I missed your talk at the SWA forum.  I'll chalk that one up to NoVa traffic.

gem

http://www.cigital.com/~gem


On 3/18/09 5:47 PM, "Steven M. Christey" <coley at linus.mitre.org> wrote:



On Wed, 18 Mar 2009, Gary McGraw wrote:

> Because it is about building a top N list FOR A PARTICULAR ORGANIZATION.
> You and I have discussed this many times.  The generic top 25 is
> unlikely to apply to any particular organization.  The notion of using
> that as a driver for software purchasing is insane.  On the other hand
> if organization X knows what THEIR top 10 bugs are, that has real value.

Got it, thanks.  I guessed as much.  Did you investigate whether the
developers' personal top-N lists were consistent with what their customers
cared about?  How did the developers go about selecting them?

By the way, last week in my OWASP Software Assurance Day talk on the Top
25, I had a slide on the role of top-N lists in BSIMM, where I attempted
to say basically the same thing.  This was after various slides that tried
to emphasize how the current Top 25 is both incomplete and not necessarily
fully relevant to a particular organization's needs.  So while the message
may have been diluted during initial publication, it's being refined
somewhat.

- Steve