Re: BSIMM-V Article in Application Development Times

[John Steven <jsteven () cigital com>]

Christian, (Stephen)

I’ll confess I’ve only skimmed the discussion but it looks productive. The questions posed are good ones. I’ll try to 
provide a few clarifications from “inside” the BSIMM study that may be helpful in pushing the discussion along:

1) Survey structure/technique attributes BSIMM activities by first seeking
    certain confirmations. If interview-based confirmation doesn't provide
    confidence, the subject is asked for documentation. 

2) The BSIMM document indicates who is interviewed, but it’s not an 
    exhaustive list. Where confirmation is necessary Dev/Config 
    Management, architects, and others make the list. 

3) Surveying claims to (and in practice) stops short of concrete attestation
    across the board. 

4) BSIMM survey targets have included organizations, business units, and 
    in rare cases, smaller scopes.

5) At the organization-level, survey confirmation includes facilities to 
    differentiate “one group does it”,  “this is done by most”, and “the
    organization governs mandatory activity conduct”.

6) An organization does/should not get credit for “one group does it”. 

7) More qualified BSIMM interviewers exist than Sammy et al. More are 
    minted as the study grows in size. There isn’t a written certification and
   a pin, but there is an involved apprenticeship. And, Sammy runs cross-
   checking of the grading process to make sure that interviewers remain
   convergent in grading criteria. 

Addressing another question raised by the email chain below: just because the organization does the activity—as a 
rule—doesn’t mean that every team does it. Non-complaince may be a reason. Another (better) reason may be that the 
organization takes a “risk-based approach” to the activity. In other words, an organization may choose to do more 
mature architecture analysis activities on only a subset of applications—those that are higher risk.  This is what 
BSIMM activities Strategy and Metrics (SM) Level 3 are about.

Hopefully that helps a bit. 
-jOHN
 ----
John Steven                 
iCTO, Cigital
+1,703-727-4034   |  @M1splacedsoul
https://google.com/+JohnStevenCigital

On Jan 7, 2014, at 8:07 PM, Christian Heinrich <christian.heinrich () cmlh id au> wrote:

> Stephen,
>
> On Sat, Jan 4, 2014 at 8:12 PM, Stephen de Vries
> <stephen () continuumsecurity net> wrote:
> > Leaving the definition of agile aside for the moment, doesn’t the fact that the BSIMM measures
> > organisation wide activities but not individual dev teams mean that we could be drawing inaccurate
> > conclusions from the data?  E.g.  if an organisation says it is doing Arch reviews, code reviews and
> > sec testing, it doesn’t necessarily mean that every team is doing all of those activities, so it may give
> > the BSIMM reader a false impression of the use of those activities in the real world.
> >
> > In addition to knowing which activities are practiced organisation wide, it would also be valuable to
> > know which activities work well on a per-team or per-project basis.
>
> My reading of the "Roles" section of BSIMM-V.pdf is that the people
> interviewed for the BSIMM sample are:
> 1. Executive Leadership (or CISO, VP of Risk, CSO, etc)
> 2. Everyone else within the Software Security Group (SSG)
>
> What you are asking to be included is what is referred to as the
> "Satellite" within BSIMM-V.pdf and I believe this may also require the
> inclusion of http://cmmiinstitute.com/cmmi-solutions/cmmi-for-development/
> too (why not :) ).
>
> The issue with this is that it would invalidate the statistics from
> the prior five BSIMM releases due to the inclusion of new questions
> and in additional these new statistics were not gathered over time
> either hence the improvements measured over time within BSIMM would be
> invalid too due tot he new dataset.
>
> Furthermore, Gary, Sammy and Brian have limited time to interview all
> 67 BSIMM participating firms.
>
> However, I would be interested to know the "BSIMM Advisory Board" i.e.
> http://bsimm.com/community/ view on this is and if it would be
> possible to undertake this additional sampling within their own BSIMM
> participating firm to determine if there is additional value would be
> gained for BSIMM?  However, I suspect that an objective measurement
> would be too hard to quantify due to internal politics of each BSIMM
> participating firm but I could be wrong.


_______________________________________________
Secure Coding mailing list (SC-L) SC-L () securecoding org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates
_______________________________________________