Positive impact of an SSG

[SMigues at cigital.com (Sammy Migues)]

Hi Pravir, 

Thanks for clarifying what you're positing. I'm not sure how we could have been more clear in the BSIMM text 
accompanying the exposition of the collective activities about the need to take this information and work it into your 
own culture (i.e., do "risk management"). As a few examples:

p. 3: "BSIMM is meant as a guide for building and evolving a software security initiative. As you will see when you 
familiarize yourself with the BSIMM activities, instilling software security into an organization takes careful 
planning and always involves broad organizational change. By clearly noting objectives and goals and by tracking 
practices with metrics tailored to your own initiative, you can methodically build software security in to your 
organization?s software development practices."

p. 47: "Choosing which of the 110 BSIMM activities to adopt and in what order can be a challenge. We suggest creating a 
software security initiative strategy and plan by focusing on goals and objectives first and letting the activities 
select themselves. Creating a timeline for rollout is often very useful.
Of course learning from experience is also a good strategy."

p. 47: "Of the 110 possible activities in BSIMM, there are ten activities that all of the nine programs we studied 
carry out. Though we can?t directly conclude that these ten activities are necessary for all software security 
initiatives, we can say with confidence that these activities are commonly found in highly successful programs. This 
suggests that if you are working on an initiative of your own, you should consider these ten activities particularly 
carefully (not to mention the other 100)."

p. 48: "The chart below shows how many of the nine organizations we studied have adopted various activities. Though you 
can use this as a rough ?weighting? of activities by prevalence, a software security initiative plan is best approached 
through goals and objectives."

Your words (...BSIMM fails...) imply (to me) that you posit organizations attempting to use the collected wisdom in 
BSIMM will, inexplicably, look at it and say "Okay, we have to do all 110 of these things exactly as written, so let's 
get started" without regard to their local need. This is as opposed to, say, looking at it and thinking "Here's what 
nine companies have spent dozens of person-decades and millions of dollars learning about what works; let's see what we 
can glean from that." Uhmmmm, okay.

Yes, previous models exist. Although it may have come up in conversation, we did not ask any of the nine something like 
"What model did you start with back in the beginning?" because it simply isn't relevant to what we're trying to 
accomplish (documenting what successful organizations are doing), just as "could" and "should" aren't relevant. We 
asked "What *are* you doing now?" and documented it so others could learn from it.

--Sammy.

-----Original Message-----
From: Pravir Chandra [mailto:chandra at list.org] 
Sent: Wednesday, March 11, 2009 4:00 AM
To: Sammy Migues; sc-l-bounces at securecoding.org; sc-l at securecoding.org
Subject: Re: [SC-L] Positive impact of an SSG

Yes, I don't think its exclusive to your BSIMM interviews that you found when people put controls into place, they saw 
improvement. That's what I (and I'm sure many other consultanting firms) have been doing for years based upon previous 
models (CLASP, MS SDL, etc.). Nothing to do with BSIMM per se (actually, most of what DTCC started doing was based on 
CLASP), just that they added controls 'early into software development lifecycle' and saw benefit, which isn't 
surprising.

That being said, the important part we're missing as 'software security guys' isn't the specification of all the 
possible things that an organization *could* do, but rather what a given organization *should* do based on good 
business decisions around risk management. And that's the crux of what BSIMM fails to do. By basing the current 
maturity model on the collected practices of 9 massive firms that spend the most on that problem, anyone (aside from 
firms in a similar situation to your 9) that attempts to apply it to their organization effectively throws risk 
management decisions out the window and commits to a much more costly solution than they could have created based on 
the knowledge of their own business needs since all the practices are based solely on the behaviors of the select few 
firms you interviewed. I'm not discounting the validity of the empirical data, I'm just positting that it isn't 
scientifically valid for solving the problem at hand.

I'm interested to hear what you learn when you get to the small and medium sized businesses as well as firms using 
agile development models (something I particularly considered and accounted for with SAMM). 

Regardless of whether we agree on the percentage of orgs for which a dedicated SSG isn't cost effective, I'm sure we 
can agree that affording 'someone in charge of success' doesn't equate to a dedicated SSG. There's a myriad of ways 
that can be accomplished in any organizational structure.

Thanks!

p.

~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~ ~~~~~~~~ ~~~~~ ~~~ ~~ ~
Pravir Chandra                      chandra<at>list<dot>org
PGP:    CE60 0E10 9207 7290 06EB   5107 4032 63FC 338E 16E4
~ ~~ ~~~ ~~~~~ ~~~~~~~~ ~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~

-----Original Message-----
From: Sammy Migues <SMigues at cigital.com>

Date: Tue, 10 Mar 2009 23:15:39 
To: sc-l at securecoding.org<sc-l at securecoding.org>
Subject: Re: [SC-L] Positive impact of an SSG


Hi Pravir,

Yes, I agree completely: the data gathered in the BSIMM interviews seem to indicate that "the controls over all" led to 
what the interviewees saw as improvements in their capability to produce secure software.

In the nine companies interviewed, those controls (BSIMM activities, I think) sprang from well established SSGs -- that 
is, a specific person or persons with the responsibility for ensuring lots (110, collectively) of activities actually 
get done.

The BSIMM data to date from specific large organizations indicate that a little under 100:1 is the average ratio for 
dev/QA to SSG size. It'll be interesting to see how this changes when we get to interviewing smaller organizations and 
we see if and how they're actually getting it done.

Personally, I don't believe I agree with your guess that 95% of organizations building code can't afford an SSG. I 
believe every organization that wants to succeed can afford to have someone in charge of success, but that's just my 
opinion and isn't relevant to BSIMM.

Cheers,

--Sammy.


-----Original Message-----
From: Pravir Chandra [mailto:chandra at list.org] 
Sent: Tuesday, March 10, 2009 6:31 PM
To: Sammy Migues
Cc: sc-l at securecoding.org
Subject: Re: [SC-L] Positive impact of an SSG

Hey Sammy.

How does that pertain to a software security group (SSG) per se? The
details below seem to indicate that it was the controls over all that
lead to the positive impact.

My main point is that supporting an SSG isn't cost effective for 95%
of the organizations out there that are building code. That's why in
SAMM, we didn't mandate the structure of the organization and instead
concentrated on the functions fulfilled by security guys (regardless
of their placement in the org).

p.

On Tue, Mar 10, 2009 at 7:48 AM, Sammy Migues <SMigues at cigital.com> wrote:
> Hi all,
>
> I've received some private questions about the 110 activities in BSIMM (bsi-mm.com). Since we built the model 
> directly from the data gathered, each activity is actually being done in one of the nine organizations interviewed. 
> The question is whether there's any evidence the activities are actually effective as opposed to simply being done.
>
> Since we can't publish any private data, I'd like to point folks at this recent article in Information Security 
> Magazine. Jim Routh, CISO of DTCC (one of the nine organizations interviewed), is quoted as follows relative to the 
> impact of software security group activities:
>
> http://searchsecurity.techtarget.com/magazineFeature/0,296894,sid14_gci1346974,00.html
>
> "One of Routh's big wins is inserting security controls early into software development lifecycle at the DTCC. 
> Vulnerabilities are weeded out well before they appear in functional code that ends up in production and that has 
> resulted in close to $2 million in productivity gains on a base of $150 million spend for development, Routh says.
>
> "Those gains are exclusively the result of having mature and effective controls within our system and software 
> development lifecycle," Routh says. This is a three-year-old initiative that educates and certifies developers in all 
> DTCC environments in security. Developers are also provided with the necessary code-scanning tools and consulting and 
> services help to keep production code close to pristine."
>
> --Sammy.
>
> Sammy Migues
> Principal, Technology
> 703.404.5830 - http://www.cigital.com
> Software confidence. Achieved.
> smigues at cigital.com
>
>
>
> _______________________________________________
> Secure Coding mailing list (SC-L) SC-L at securecoding.org
> List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
> List charter available at - http://www.securecoding.org/list/charter.php
> SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
> as a free, non-commercial service to the software security community.
> _______________________________________________



-- 
~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~ ~~~~~~~~ ~~~~~ ~~~ ~~ ~
Pravir Chandra                      chandra<at>list<dot>org
PGP:    CE60 0E10 9207 7290 06EB   5107 4032 63FC 338E 16E4
~ ~~ ~~~ ~~~~~ ~~~~~~~~ ~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~

_______________________________________________
Secure Coding mailing list (SC-L) SC-L at securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
_______________________________________________