BSIMM update (informIT)

[gem at cigital.com (Gary McGraw)]

Hi Arian,

Some more particulars regarding your posting.  Sorry for the delay...

On 2/2/10 4:32 PM, "Arian J. Evans" <arian.evans at anachronic.com> wrote:
> Strategic folks (VP, CxO) ...Initially ...ask for descriptive information, but once they get
> going they need strategic prescriptions.

Please see my response to Kevin.  I hope it's clear what the BSIMM is for.  It's for measuring your initiative and 
comparing it to others.  Given some solid BSIMM data, I believe you can do a superior job with strategy...and results 
measurement.  It is a tool for strategic people to use to build an initiative that works.

> Tactical folks tend to ask:
> + What should we fix first? (prescriptive)
> + What steps can I take to reduce XSS attack surface by 80%?

The BSIMM is not for tactical folks.  But should you base your decision regarding "what to fix first" on goat 
sacrifice?  What should drive that decision?  Moon phase?

> Implementation level folks ask:
> + What do I do about this specific attack/weakness?
> + How do I make my compensating control (WAF, IPS) block this specific attack?

BSIMM != code review tool, top-n list, book, coding experience, ...

> BSIMM is probably useful for government agencies, or some large
> organizations. But the vast majority of clients I work with don't have
> the time or need or ability to take advantage of BSIMM. Nor should
> they. They don't need a software security group.

Where to start.  All I can say about BSIMM so far is that is appears to be useful for 30 large commercial organizations 
carrying out real software security initiatives.  We have studied 0 (count 'em...none) government organizations to 
date.  In my experience, the government is always lagging when it comes to software security.  I'm hoping to gather 
some government data forth with, starting with the US Air Force.  We shall see.

But what about SMB (small to medium sized business)?  Arian, who are your clients?  How many developers do they have?  
Who do you report to as a consultant?  How do you help them make business decisions?

Regarding the existence of an SSG, see this article <http://www.informit.com/articles/article.aspx?p=1434903>.  Are 
your customers too small to have an SSG?  Are YOU the SSG?  Are your customers not mature enough for an SSG?  Data 
would be great.

> I agree and strongly empathize with Gary on many
> premises of his article - including that not many folks have metrics,
> and tend to have more faith and magic.

Sadly I think we're stuck with "second order" metrics like the BSIMM.  Heck, we even studied the metrics that real 
initiatives use in the BSIMM (bugs per square inch anyone?), but you know what?  Everyone has different metrics.  
Really.

> But, as should be no surprise, I cateogrically disagree with the
> entire concluding paragraph of the article. Sadly it's just more faith
> and magic from Gary's end. We all can do better than that.

You guys and your personal attacks.  Yeesh.  I am pretty sure you meant the "next to last" paragraph, because Feynman 
wrote the entire last one.  Here is the next to last one:

As I have said before, the time has come to put away the bug parade boogeyman 
<http://www.informit.com/articles/article.aspx?p=1248057>, the top 25 tea leaves 
<http://www.informit.com/articles/article.aspx?p=1322398>, black box web app goat sacrifice, and the occult reading of 
pen testing entrails. It's science time.  And the more descriptive and data driven we are, the better.

Can you be more specific about your disagreements please?  Did you read articles at the end of the pointers?  Where am 
I wrong?  Better yet, why?

We'll just ignore the Nader > Feynman stuff.

gem

company www.cigital.com
podcast www.cigital.com/silverbullet
blog www.cigital.com/justiceleague
book www.swsec.com


On Tue, Feb 2, 2010 at 9:30 AM, Wall, Kevin <Kevin.Wall at qwest.com> wrote:
> On Thu, 28 Jan 2010 10:34:30 -0500, Gary McGraw wrote:
>
> > Among other things, David [Rice] and I discussed the difference between
> > descriptive models like BSIMM and prescriptive models which purport to
> > tell you what you should do.  I just wrote an article about that for
> > informIT.  The title is
> >
> > "Cargo Cult Computer Security: Why we need more description and less
> > prescription."
> > http://www.informit.com/articles/article.aspx?p=1562220
>
> First, let me say that I have been the team lead of a small Software
> Security Group (specifically, an Application Security team) at a
> large telecom company for the past 11 years, so I am writing this from
> an SSG practitioner's perspective.
>
> Second, let me say that I appreciate descriptive holistic approaches to
> security such as BSIMM and OWASP's OpenSAMM. I think they are much
> needed, though seldom heeded.
>
> Which brings me to my third point. In my 11 years of experience working
> on this SSG, it is very rare that application development teams are
> looking for a _descriptive_ approach. Almost always, they are
> looking for a _prescriptive_ one. They want specific solutions
> to specific problems, not some general formula to an approach that will
> make them more secure. To those application development teams, something
> like OWASP's ESAPI is much more valuable than something like BSIMM or
> OpenSAMM. In fact, I you confirm that you BSIMM research would indicate that
> many companies' SSGs have developed their own proprietary security APIs
> for use by their application development teams. Therefore, to that end,
> I would not say we need less _prescriptive_ and more _descriptive_
> approaches. Both are useful and ideally should go together like hand and
> glove. (To that end, I also ask that you overlook some of my somewhat
> overzealous ESAPI developer colleagues who in the past made claims that
> ESAPI was the greatest thing since sliced beer. While I am an ardent
> ESAPI supporter and contributor, I proclaim it will *NOT* solve our pandemic
> security issues alone, nor for the record will it solve world hunger. ;-)
>
> I suspect that this apparent dichotomy in our perception of the
> usefulness of the prescriptive vs. descriptive approaches is explained
> in part by the different audiences with whom we associate. Hang out with
> VPs, CSOs, and executive directors and they likely are looking for advice on
> an SSDLC or broad direction to cover their specifically identified
> security gaps. However, in the trenches--where my team works--they want
> specifics. They ask us "How can you help us to eliminate our specific
> XSS or CSRF issues?", "Can you provide us with a secure SSO solution
> that is compliant with both corporate information security policies and
> regulatory compliance?", etc. If our SSG were to hand them something like
> BSIMM, they would come away telling their management that we didn't help
> them at all.
>
> This brings me to my fourth, and likely most controversial point. Despite
> the interesting historical story about Feynman, I question whether BSIMM
> is really "scientific" as the BSIMM community claims. I would contend
> that we are only fooling ourselves if we claim otherwise. And while
> BSIMM is a refreshing approach opposed to the traditional FUD modus
> operandi taken by most security vendors hyping their security products,
> I would argue that BSIMM is no more scientific than the those
> who gather common quality metrics of counting defects/KLOC. Certainly
> there is some correlation there, but cause and effect relationships
> are far from obvious and seem to have little predictive accuracy.
>
> Sure, BSIMM _looks_ scientific on the outside, but simply collecting
> specific quantifiable data alone does not make something a scientific
> endeavor.  Yes, it is a start, but we've been collecting quantifiable
> data for decades on things like software defects and I would contend
> BSIMM is no more scientific than those efforts. Is BSIMM moving in
> the right direction? I think so. But BSIMM is no more scientific
> than most of the other areas of computer "science".
>
> To study something scientifically goes _beyond_ simply gathering
> observable and measurable evidence. Not only does data needs to be
> collected, but it also needs to be tested against a hypotheses that offers
> a tentative *explanation* of the observed phenomena;
> i.e., the hypotheses should offer some predictive value. Furthermore,
> the steps of the experiment must be _repeatable_, not just by
> those currently involved in the attempted scientific endeavor, but by
> *anyone* who would care to repeat the experiment. If the
> steps are not repeatable, then any predictive value of the study is lost.
>
> While I am certainly not privy to the exact method used to arrive at the
> BSIMM data (I have read through the "BSIMM Begin" survey, but have not
> been involved in a full BSIMM assessment), I would contend that the
> process is not repeatable to the necessary degree required by science.
> In fact, I would claim in most organizations, you could take any group
> of BSIMM interviewers and have them question different people in the
> organization using the exact same questions and arrive at different results.
> In fact, I am willing to bet that the different members of my Application
> Security team who have all worked together for about 8 years would
> answer a significant number of the BSIMM Begin survey questions quite
> differently. (My explanation for this phenomena is the general observation
> that if you ask a group of N engineers for their opinion on something,
> they will almost certainly arrive at N+1 different opinions. ;-)
>
> I commend the BSIMM sponsors and leadership of releasing BSIMM under
> a Creative Commons license, but at the same time, I challenge them
> to put forth additional information explaining their data collection
> process and in particular, describing how it
> avoids unintentional bias. (E.g., Are assessment participants choose at
> random? By whom?  How do you know you have a representative sample of
> a company? Etc.)
>
> I also challenge BSIMM to show their data collection is repeatable by
> others following their process. The published BSIMM Begin survey is a
> good start, but information regarding the full assessment seems to be
> lacking.
>
> I challenge BSIMM to put forth their hypotheses, plainly stated.
> In your InformIT article, you wrote:
>
>    "Another distinct advantage that descriptive models have over
>    prescriptive models is the ability to compare current observations
>    with past observations."
>
> In my opinion, comparison of observations from two companies is not
> worth the paper that is printed on UNLESS we can extrapolate from
> this data and make accurate predictions based on past findings. Therefore,
> I also would challenge BSIMM to publicly make some specific predictions
> using their model and collected data so that their hypotheses can be
> tested independently by others.
>
> Finally, while I would like, as you did, to blame our Computer Security /
> software security's "Cargo Cult" mentality on "its relative youth as
> a field", I believe there is something deeper going on here. For one,
> computer science / IT / whatever you want to call this much broader
> field has the same issue. And while computer "science" is young as
> measured against most other scientific disciplines, it is by no means an
> immature field. (As a discipline, it is much older than I, and trust me,
> I am no spring chicken. :)
>
> After observing this field for 30+ years (ouch!), I have concluded that we
> have not matured into a science because as a discipline we *do NOT
> really want to!*  We can't even decide if we want this study of computers
> / information processing / etc. to be a "science", an "engineering
> discipline", or a "craft". (And some even would like it to be an "art".)
> Most of us--myself included--are too lazy to do the disciplined work
> that true science requires, and that includes having enough guts to
> challenge the academic culture and *demand* funding to do well-designed
> scientific experiment in our discipline at our leading universities. IMO,
> if we fail to do this, CS is doomed to always remain a science wannabe.
>
> Some would say that because our broader field of study--whether you
> call it Computer Science, Computer Engineering, Information Science,
> whatever--is part science, part engineering, part craft, and part
> something unique that humanity has never before encountered, attempts
> to treat it as a science will not succeed. However, surely this does
> not mean that we should not attempt to add some scientific rigor to
> it as a discipline. To that end efforts such as BSIMM should be welcomed
> by all.  But is also important for those who prefer _descriptive_ approaches
> like BSIMM, to acknowledge the importance of _prescriptive_ approaches
> such as ESAPI, WAFs, anti-malware software, etc. We truly need *both*
> approaches to be successful.
>
> Regards,
> -kevin
> ---
> Kevin W. Wall           Qwest Information Technology, Inc.
> Kevin.Wall at qwest.com    Phone: 614.215.4788
> "It is practically impossible to teach good programming to students
>  that have had a prior exposure to BASIC: as potential programmers
>  they are mentally mutilated beyond hope of regeneration"
>    - Edsger Dijkstra, How do we tell truths that matter?
>      http://www.cs.utexas.edu/~EWD/transcriptions/EWD04xx/EWD498.html
>
> This communication is the property of Qwest and may contain confidential or
> privileged information. Unauthorized use of this communication is strictly
> prohibited and may be unlawful.  If you have received this communication
> in error, please immediately notify the sender by reply e-mail and destroy
> all copies of the communication and any attachments.
>
> _______________________________________________
> Secure Coding mailing list (SC-L) SC-L at securecoding.org
> List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
> List charter available at - http://www.securecoding.org/list/charter.php
> SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
> as a free, non-commercial service to the software security community.
> _______________________________________________

_______________________________________________
Secure Coding mailing list (SC-L) SC-L at securecoding.org
List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
List charter available at - http://www.securecoding.org/list/charter.php
SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
as a free, non-commercial service to the software security community.
_______________________________________________