Interesting tidbit in iDefense Security Advisory 06.26.07

[leichter_jerrold at emc.com (Leichter, Jerry)]

On Thu, 28 Jun 2007, J. M. Seitz wrote:
| Hey there,
|  
| > If you couldn't insert "ignore" directives, many people 
| > wouldn't use such tools at all, and would release code with 
| > vulnerabilities that WOULD be found by such tools.
| 
| Of course, much like an IDS, you have to find the baseline and adjust
| your ruleset according to the norm, if it is constantly firing on
| someone accessing /index.html of your website, then that's working
| against you.
| 
| I am not disagreeing with the fact the static source analysis is a
| good thing, I am just saying that this is a case where it failed (or
| maybe the user/developer of it failed or misunderstood it's use). Fair
| enough that on this particular list you are going to defend source
| analysis over any other method, it is about secure coding after all,
| but I definitely still strongly disagree that other methods wouldn't
| have found this bug.
| 
| Shall we take a look at the customer lists of the big source analyzer
| companies, and then cross-map that to the number of vulnerabilities
| released? 
It would be great to have that information.  Even better would be to
classify the vulnerabilities in two buckets:  Those that the analyzer
was expected to find, and those that it's not ready for.

You would have to allow for the time it takes from when the analyzer
starts being used to when software that actually went through, not just
the analyzer, but the remediation process, actually hits the streets.
For large companies and widely-used commerical products, that can be a
long time.

However ... I think the chances of getting that kind for commercial
projects is just about nil.  Companies generally consider the details of
their software development processes proprietary, and at most will give
you broad generalities about the kinds of tools and processes they use.
(Given the cost of these analyzers, you'd think that potential customers
would want some data about actual payoffs.  However, I think they
recognize that no such data exists at this point.  A prudent customer
might well collect such data for himself to help in deciding whether the
contract for the analyzer is worth renewing - though even this kind of
careful analysis is very rare in the industry.)

An OSS project might be a better starting place for this kind of study.
I know that Coverity has analyzed a couple of significant pieces of OSS
for free (including, I believe, the Linux and NetBSD kernels).  It's
likely that other analyzer vendors have done something similar, though I
haven't heard about it.  A study showing how using an analyzer led to an
x% decrease in reported bugs would make for great sales copy.  (Of
course, there's always the risk that the analyzer doesn't actually help
make the software more secure!)

|           Why are we still finding bugs in software that have the SDL?
| Why are we still finding bugs in software that have been analyzed
| before the compiler has run? Why are these companies like Fortify
| charging an arm and a leg for such a technology when the bughunters
| are still beating the snot out of this stuff? 
I'm not sure that's a fair comparison.  The defenders have to plug *all*
the holes, including those using techniques that weren't known at the
time the software was produced.  The attackers only have to find one
hole.

|                                               You guys all have much
| more experience on that end, so I am looking forward to your
| responses!
As with almost everything in software engineering, sad to say, there is
very little objective evidence.  It's hard and expensive to gather, and
those who are in a position to pay for the effort rarely see a major
payoff from making it.  Which would you rather do:  Put up $1,000,000 to
do a study which *might* show your software/methodology/whatever helps,
or pay $1,000,000 to hire a bunch of "feet on the street" to sell more
copies?  So we have to go by gut feel and engineering judgement.  Those
certainly say, for most people, that static analyzers will help.  Then
again, I'm sure most people on this list will argue that strong static
typing is essential for secure, reliable software.  "Everyone has known
that" for 20 years or more.  Except that ... if you look around, you'll
find many arguments these days that the run-time typing characteristic
of languages like Python or Ruby is just as good and lets you produce
software much faster.  Which argument is true?  You'd think that after
50 years of software development, we'd at least know how to frame a
test ... but even that seems beyond the state of the art.

                                                        -- Jerry

| Cheers! 
| 
| JS
| 
| _______________________________________________
| Secure Coding mailing list (SC-L) SC-L at securecoding.org
| List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
| List charter available at - http://www.securecoding.org/list/charter.php
| SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
| as a free, non-commercial service to the software security community.
| _______________________________________________
| 
|