Secure Coding mailing list archives
BSIMM update (informIT)
From: gem at cigital.com (Gary McGraw)
Date: Thu, 4 Feb 2010 14:32:37 -0500
hi jim, We chose organizations that in our opinion are doing a superior job with software security. You are welcome to disagree with our choices. Microsoft has a shockingly good approach to software security that they are kind enough to share with the world through the SDL books and websites. Google has a much different approach with more attention focused on open source risk and testing (and much less on code review with tools). Adobe has a newly reinvigorated approach under new leadership that is making some much needed progress. The three firms that you cited were all members of the original nine whose data allowed us to construct the model. There are now 30 firms in the BSIMM study, and their BSIMM data vary as much as you might expect...about which more soon. gem company www.cigital.com podcast www.cigital.com/silverbullet blog www.cigital.com/justiceleague book www.swsec.com On 2/4/10 12:50 PM, "Jim Manico" <jim at manico.net> wrote: Why are we holding up the statistics from Google, Adobe and Microsoft ( http://www.bsi-mm.com/participate/ ) in BDSIMM? These companies are examples of recent "epic security failure". Probably the most financially damaging infosec attack, ever. Microsoft let a plain-vanilla 0-day slip through ie6 for years, Google has a pretty basic network segmentation and policy problem, and Adobe continues to be the laughing stock of client side security. Why are we holding up these companies as BDSIMM champions? - Jim
On Wed, 3 Feb 2010, Gary McGraw wrote:Popularity contests are not the kind of data we should count on. But maybe we'll make some progress on that one day.That's my hope, too, but I'm comfortable with making baby steps along the way.Ultimately, I would love to see the kind of linkage between the collected data ("evidence") and some larger goal ("higher security" whatever THAT means in quantitative terms) but if it's out there, I don't see itNeither do I, and that is a serious issue with models like the BSIMM that measure "second order" effects like activities. Do the activities actually do any good? Important question!And one we can't answer without more data that comes from the developers who adopt any particular practice, and without some independent measure of what success means. For example: I am a big fan of the attack surface metric originally proposed by Michael Howard and taken up by Jeanette Wing et al. at CMU (still need to find the time to read Manadhata's thesis, alas...) It seems like common sense that if you reduce attack surface, you reduce the number of security problems, but how do you KNOW!?The 2010 OWASP Top 10 RC1 is more data-driven than previous versions; same with the 2010 Top 25 (whose release has been delayed to Feb 16, btw). Unlike last year's Top 25 effort, this time I received several sources of raw prevalence data, but unfortunately it wasn't in sufficiently consumable form to combine.I was with you up until that last part. Combining the prevalence data is something you guys should definitely do. BTW, how is the 2010 CWE-25 (which doesn't yet exist) more data driven??I guess you could call it a more refined version of the "popularity contest" that you already referred to (with the associated limitations, and thus subject to some of the same criticisms as those pointed at BSIMM): we effectively conducted a survey of a diverse set of organizations/individuals from various parts of the software security industry, asking what was most important to them, and what they saw the most often. This year, I intentionally designed the Top 25 under the assumption that we would not have hard-core quantitative data, recognizing that people WANTED hard-core data, and that the few people who actually had this data, would not want to share it. (After all, as a software vendor you may know what your own problems are, but you might not want to share that with anyone else.) It was a bit of a surprise when a handful of participants actually had real data - but, then the problem I'm referring to with respect to "consumable form" reared its ugly head. One third-party consultant had statistics for a broad set of about 10 high-level categories representing hundreds of evaluations; one software vendor gave us a specific weakness history - representing dozens of different CWE entries across a broad spectrum of issues, sometimes at very low levels of detail and even branching into the GUI part of CWE which almost nobody pays attention to - but "only" for 3 products. Another vendor rep evaluated the dozen or two publicly-disclosed vulnerabilities that were most severe according to associated CVSS scores. Those three data sets, plus the handful of others based on some form of analysis of hard-core data, are not merge-able. The irony with CWE (and many of the making-security-measurable efforts) is that it brings sufficient clarity to recognize when there is no clarity... the "known unknowns" to quote Donald Rumsfeld. I saw this in 1999 in the early days of CVE, too, and it's still going on - observers of the oss-security list see this weekly. For data collection at such a specialized level, the situation is not unlike the breach-data problem faced by the Open Security Foundation in their Data Loss DB work - sometimes you have details, sometimes you don't. The Data Loss people might be able to say "well, based on this 100-page report we examined, we think it MIGHT have been SQL injection" but that's the kind of data we're dealing with right now. Now, a separate exercise in which we compare/contrast the customized top-n lists of those who have actually progressed to the point of making them... that smells like opportunity to me.I for one am pretty satisfied with the rate at which things are progressing and am delighted to see that we're finally getting some raw data, as good (or as bad) as it may be. The data collection process, source data, metrics, and conclusions associated with the 2010 Top 25 will probably be controversial, but at least there's some data to argue about.Cool!To clarify to others who have commented on this part - I'm talking specifically about the rate in which the software security industry seems to be maturing, independently of how quickly the threat landscape is changing. That's a whole different, depressing problem. - Steve _______________________________________________ Secure Coding mailing list (SC-L) SC-L at securecoding.org List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l List charter available at - http://www.securecoding.org/list/charter.php SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com) as a free, non-commercial service to the software security community. _______________________________________________
Current thread:
- BSIMM update (informIT), (continued)
- BSIMM update (informIT) Brian Chess (Feb 04)
- BSIMM update (informIT) Gary McGraw (Feb 04)
- BSIMM update (informIT) McGovern, James F. (eBusiness) (Feb 04)
- BSIMM update (informIT) Wall, Kevin (Feb 02)
- BSIMM update (informIT) Steven M. Christey (Feb 02)
- BSIMM update (informIT) Gary McGraw (Feb 03)
- BSIMM update (informIT) Mike Boberski (Feb 03)
- BSIMM update (informIT) Steven M. Christey (Feb 03)
- BSIMM update (informIT) Jim Manico (Feb 04)
- BSIMM update (informIT) Steven M. Christey (Feb 04)
- BSIMM update (informIT) Gary McGraw (Feb 04)
- Thread is dead -- Re: BSIMM update (informIT) Kenneth Van Wyk (Feb 04)
- Message not available
- Message not available
- BSIMM update (informIT) Steven M. Christey (Feb 04)
- BSIMM update (informIT) Steven M. Christey (Feb 02)
- Metrics McGovern, James F. (eBusiness) (Feb 05)
- Metrics Steven M. Christey (Feb 05)
- Metrics Arian J. Evans (Feb 05)
- BSIMM update (informIT) Steven M. Christey (Feb 02)
- BSIMM update (informIT) Mike Boberski (Feb 02)
- BSIMM update (informIT) Gary McGraw (Feb 03)