BSIMM update (informIT)

[coley at linus.mitre.org (Steven M. Christey)]

On Wed, 3 Feb 2010, Gary McGraw wrote:

> Popularity contests are not the kind of data we should count on.  But 
> maybe we'll make some progress on that one day.

That's my hope, too, but I'm comfortable with making baby steps along the 
way.

> > Ultimately, I would love to see the kind of linkage between the collected
> > data ("evidence") and some larger goal ("higher security" whatever THAT
> > means in quantitative terms) but if it's out there, I don't see it
>
> Neither do I, and that is a serious issue with models like the BSIMM 
> that measure "second order" effects like activities.  Do the activities 
> actually do any good?  Important question!

And one we can't answer without more data that comes from the developers 
who adopt any particular practice, and without some independent measure of 
what success means.  For example: I am a big fan of the attack surface 
metric originally proposed by Michael Howard and taken up by Jeanette Wing 
et al. at CMU (still need to find the time to read Manadhata's thesis, 
alas...)  It seems like common sense that if you reduce attack surface, 
you reduce the number of security problems, but how do you KNOW!?

> > The 2010 OWASP Top 10 RC1 is more data-driven than previous versions; same
> > with the 2010 Top 25 (whose release has been delayed to Feb 16, btw).
> > Unlike last year's Top 25 effort, this time I received several sources of
> > raw prevalence data, but unfortunately it wasn't in sufficiently
> > consumable form to combine.
>
> I was with you up until that last part.  Combining the prevalence data 
> is something you guys should definitely do.  BTW, how is the 2010 CWE-25 
> (which doesn't yet exist) more data driven??

I guess you could call it a more refined version of the "popularity 
contest" that you already referred to (with the associated limitations, 
and thus subject to some of the same criticisms as those pointed at 
BSIMM): we effectively conducted a survey of a diverse set of 
organizations/individuals from various parts of the software security 
industry, asking what was most important to them, and what they saw the 
most often.  This year, I intentionally designed the Top 25 under the 
assumption that we would not have hard-core quantitative data, recognizing 
that people WANTED hard-core data, and that the few people who actually 
had this data, would not want to share it.  (After all, as a software 
vendor you may know what your own problems are, but you might not want to 
share that with anyone else.)

It was a bit of a surprise when a handful of participants actually had 
real data - but, then the problem I'm referring to with respect to 
"consumable form" reared its ugly head.  One third-party consultant had 
statistics for a broad set of about 10 high-level categories representing 
hundreds of evaluations; one software vendor gave us a specific weakness 
history - representing dozens of different CWE entries across a broad 
spectrum of issues, sometimes at very low levels of detail and even 
branching into the GUI part of CWE which almost nobody pays attention to - 
but "only" for 3 products.  Another vendor rep evaluated the dozen or two 
publicly-disclosed vulnerabilities that were most severe according to 
associated CVSS scores.  Those three data sets, plus the handful of others 
based on some form of analysis of hard-core data, are not merge-able. 
The irony with CWE (and many of the making-security-measurable efforts) is 
that it brings sufficient clarity to recognize when there is no clarity... 
the "known unknowns" to quote Donald Rumsfeld.  I saw this in 1999 in the 
early days of CVE, too, and it's still going on - observers of the 
oss-security list see this weekly.

For data collection at such a specialized level, the situation is not 
unlike the breach-data problem faced by the Open Security Foundation in 
their Data Loss DB work - sometimes you have details, sometimes you don't. 
The Data Loss people might be able to say "well, based on this 100-page 
report we examined, we think it MIGHT have been SQL injection" but that's 
the kind of data we're dealing with right now.

Now, a separate exercise in which we compare/contrast the customized top-n 
lists of those who have actually progressed to the point of making them... 
that smells like opportunity to me.

> > I for one am pretty satisfied with the rate at which things are
> > progressing and am delighted to see that we're finally getting some raw
> > data, as good (or as bad) as it may be.  The data collection process,
> > source data, metrics, and conclusions associated with the 2010 Top 25 will
> > probably be controversial, but at least there's some data to argue about.
>
> Cool!

To clarify to others who have commented on this part - I'm talking 
specifically about the rate in which the software security industry seems 
to be maturing, independently of how quickly the threat landscape is 
changing.  That's a whole different, depressing problem.

- Steve