Interesting Blog Entry on Tools Coverage

[coley at linus.mitre.org (Steven M. Christey)]

All,

The original blog entry stems from a CWE pie chart that won't die until we
replace it with a more well-grounded pie chart.

We posted a followup here:


http://www.matasano.com/log/912/finger-79tcp-christeymartin-evolution-of-the-cwe-pie-chart/


In short, CWE contains several types of nodes at multiple levels of
abstraction, including general categories ("input validation problems")
and arbitrary groupings ("problems related to memory management").  The
original pie chart mixed these node types with 'real' weaknesses, and we
included it in a CWE briefing as a demonstrative example of the utility of
CWE in comparing code analysis tools.

While that pie chart is still partially usable for showing a relative lack
of overlap between tools (modulo the abstraction problem), the "only 45%
of weakness types are found by tools" figure is probably low, since CWE
currently has many nodes that are organizational in nature, so they would
be excluded from any comparative analysis.  (Although we're also probably
relatively shallow with respect to design issues compared to
implementation bugs, which might pull the numbers in another direction as
CWE continues to fill in the gaps).

As vaguely implied in the followup blog entry above, we will be working on
a new pie chart with a better selection of CWE nodes, which should
generate more credible numbers.  We've been doing the ground work, e.g.
explicitly identifying the types of nodes that could then be excluded from
such analyses, but I can't be sure of when we'll have a new-and-improved
pie chart.  Rest assured that we are highly motivated to replace the
existing chart, however, and I think we've learned our lesson about
releasing "demonstrative statistics" in new technology areas that don't
have any.

- Steve