BSIMM update (informIT)

[jim at manico.net (Jim Manico)]

Why are we holding up the statistics from Google, Adobe and Microsoft ( 
http://www.bsi-mm.com/participate/ ) in BDSIMM?

These companies are examples of recent "epic security failure". Probably 
the most financially damaging infosec attack, ever. Microsoft let a 
plain-vanilla 0-day slip through ie6 for years, Google has a pretty 
basic network segmentation and policy problem, and Adobe continues to be 
the laughing stock of client side security. Why are we holding up these 
companies as BDSIMM champions?

- Jim

> On Wed, 3 Feb 2010, Gary McGraw wrote:
>
> > Popularity contests are not the kind of data we should count on.  But 
> > maybe we'll make some progress on that one day.
>
> That's my hope, too, but I'm comfortable with making baby steps along 
> the way.
>
> > > Ultimately, I would love to see the kind of linkage between the 
> > > collected
> > > data ("evidence") and some larger goal ("higher security" whatever THAT
> > > means in quantitative terms) but if it's out there, I don't see it
> >
> > Neither do I, and that is a serious issue with models like the BSIMM 
> > that measure "second order" effects like activities.  Do the 
> > activities actually do any good?  Important question!
>
> And one we can't answer without more data that comes from the 
> developers who adopt any particular practice, and without some 
> independent measure of what success means.  For example: I am a big 
> fan of the attack surface metric originally proposed by Michael Howard 
> and taken up by Jeanette Wing et al. at CMU (still need to find the 
> time to read Manadhata's thesis, alas...)  It seems like common sense 
> that if you reduce attack surface, you reduce the number of security 
> problems, but how do you KNOW!?
>
> > > The 2010 OWASP Top 10 RC1 is more data-driven than previous 
> > > versions; same
> > > with the 2010 Top 25 (whose release has been delayed to Feb 16, btw).
> > > Unlike last year's Top 25 effort, this time I received several 
> > > sources of
> > > raw prevalence data, but unfortunately it wasn't in sufficiently
> > > consumable form to combine.
> >
> > I was with you up until that last part.  Combining the prevalence 
> > data is something you guys should definitely do.  BTW, how is the 
> > 2010 CWE-25 (which doesn't yet exist) more data driven??
>
> I guess you could call it a more refined version of the "popularity 
> contest" that you already referred to (with the associated 
> limitations, and thus subject to some of the same criticisms as those 
> pointed at BSIMM): we effectively conducted a survey of a diverse set 
> of organizations/individuals from various parts of the software 
> security industry, asking what was most important to them, and what 
> they saw the most often.  This year, I intentionally designed the Top 
> 25 under the assumption that we would not have hard-core quantitative 
> data, recognizing that people WANTED hard-core data, and that the few 
> people who actually had this data, would not want to share it.  (After 
> all, as a software vendor you may know what your own problems are, but 
> you might not want to share that with anyone else.)
>
> It was a bit of a surprise when a handful of participants actually had 
> real data - but, then the problem I'm referring to with respect to 
> "consumable form" reared its ugly head.  One third-party consultant 
> had statistics for a broad set of about 10 high-level categories 
> representing hundreds of evaluations; one software vendor gave us a 
> specific weakness history - representing dozens of different CWE 
> entries across a broad spectrum of issues, sometimes at very low 
> levels of detail and even branching into the GUI part of CWE which 
> almost nobody pays attention to - but "only" for 3 products.  Another 
> vendor rep evaluated the dozen or two publicly-disclosed 
> vulnerabilities that were most severe according to associated CVSS 
> scores.  Those three data sets, plus the handful of others based on 
> some form of analysis of hard-core data, are not merge-able. The irony 
> with CWE (and many of the making-security-measurable efforts) is that 
> it brings sufficient clarity to recognize when there is no clarity... 
> the "known unknowns" to quote Donald Rumsfeld.  I saw this in 1999 in 
> the early days of CVE, too, and it's still going on - observers of the 
> oss-security list see this weekly.
>
> For data collection at such a specialized level, the situation is not 
> unlike the breach-data problem faced by the Open Security Foundation 
> in their Data Loss DB work - sometimes you have details, sometimes you 
> don't. The Data Loss people might be able to say "well, based on this 
> 100-page report we examined, we think it MIGHT have been SQL 
> injection" but that's the kind of data we're dealing with right now.
>
> Now, a separate exercise in which we compare/contrast the customized 
> top-n lists of those who have actually progressed to the point of 
> making them... that smells like opportunity to me.
>
> > > I for one am pretty satisfied with the rate at which things are
> > > progressing and am delighted to see that we're finally getting some raw
> > > data, as good (or as bad) as it may be.  The data collection process,
> > > source data, metrics, and conclusions associated with the 2010 Top 
> > > 25 will
> > > probably be controversial, but at least there's some data to argue 
> > > about.
> >
> > Cool!
>
> To clarify to others who have commented on this part - I'm talking 
> specifically about the rate in which the software security industry 
> seems to be maturing, independently of how quickly the threat 
> landscape is changing.  That's a whole different, depressing problem.
>
> - Steve
> _______________________________________________
> Secure Coding mailing list (SC-L) SC-L at securecoding.org
> List information, subscriptions, etc - 
> http://krvw.com/mailman/listinfo/sc-l
> List charter available at - http://www.securecoding.org/list/charter.php
> SC-L is hosted and moderated by KRvW Associates, LLC 
> (http://www.KRvW.com)
> as a free, non-commercial service to the software security community.
> _______________________________________________