But what proof do we have that any of it makes a difference?

[goertzel_karen at bah.com (Goertzel, Karen)]

There are two closely-related questions that keep arising for which I
still can find no satisfying answer (for me "satisfying" means
"supported by concrete evidence"):

[1] Will using a secure SDLC methodology (or a set of secure development
"best practices") actually produce software that will, when deployed "in
the wild" resist or tolerate attacks and attempted executions of
inserted/embedded malicious code better than software whose developers
did not use a secure SDLC methodology?

[2] Will software that adheres to a set of "security principles", when
deployed "in the wild", actually resist or tolerate attacks and
attempted execution of inserted/embedded malicious code better than
software whose developers did not adhere to security principles?

Right now, all I can find are abundant (possibly TOO abundant) vaguely
supported - and frequently unsupported - assertions by SwA
practitioners, authors of books on software security, and promoters of
secure SDLC methodologies. What I can't find is concrete
cause-and-effect evidence to back up those assertions.

Possibly, the problem starts with our industry's inability to
unequivocally demonstrate that "secure software" even exists - not in
the conceptual sense (I'm overfed with concepts of what software should
be to be considered secure) - but in real world implementations. 

Where are the case studies that demonstrate that BECAUSE such-and-such a
secure SDLC methodology was used or BECAUSE such-and-such a set of
secure principles were adhered to, the resulting software, when
attacked, was able to resist (or tolerate) the attack without being
compromised.

I've found the case studies that tell me how following a given secure
SDLC methodology made regulators happy. I've found case studies saying
there were fewer vulnerabilities in software that was produced under a
secure development regime. But I have yet to see any studies that
unambigiously demonstrate that BECAUSE certain security regulations were
satisfied or BECAUSE the software had fewer vulnerabilities,  attacks on
it were unable to succeed.

Is the problem down to the still-embryonic state of software security
metrics? I think it's fair to say that there's not yet anything
approaching wide agreement even about what characteristics of software
can and should be measured to provide irrefutable evidence that software
exhibits a fixed, or even a relative, level of security. For example,
given there's probably at least one metric that can be used to quantify
the degree to which particular software design adheres to a required set
of secure design principles, what does that metric tell us about how
much more secure the software is than it would have been had its design
not adhered to those principles?

If I can't even find good metrics for quantifying how secure a given
program is, how can I hope to find a good metric for determining ex post
facto (let alone predicting) whether that program, because it was
developed using a given secure methodology or set of "best security
practices", is in fact more secure - and to what extent - when compared
against the same program developed using a traditional,
non-security-minded methodology/set of practices. 

I'm thinking that some university needs to do a spin on "n-version
programming" wherein the same user needs statement is given to 3
different development teams - #1 using a security enhanced methodology
and adhering to security principles, #2 using a traditional methodology
and adhering to security principles, and #3 not doing anything about
security. The resulting three different program versions would then be
subjected to pen tests and other blackhat tests, and measurements would
be taken vis how each one fared in terms of attack-resistance or
attack-tolerance. Not a perfect metric by any means, but at least it
would give some idea of the relative good-better-bestness of the three
approaches to developing the software.

In the absence of such metrics, we're all left to take it on faith that
because SwA practioners and authors we respect make certain
unsubstantiated assertions about "goodness" of their secure
methodologies and the necessity of security principles, following those
methods and adhering to those principles will produce an even
qualitatively verifiable benefit, i.e., that the software will be less
vulnerable to compromise when deployed "out in the wild" than software
produced "the old fashioned way". 


--
Karen Mercedes Goertzel, CISSP
Booz Allen Hamilton
703.902.6981
goertzel_karen at bah.com