NSA comparison of source code analysis tools

[jeremy.j.epstein at gmail.com (Jeremy Epstein)]

(Apologies if I already sent this to the group; I don't think I did.)

There's an interesting presentation at
http://www.iarpa.gov/stonesoup_Merced_DHSAWGbrief.pdf about a study
done by the US NSA (National Security Agency) of C and Java source
code analysis tools.  They developed a synthetic test suite, and then
ran six tools against the Java version and five tools against the C
version (the specific tools and versions used are identified in the
presentation).  None of the tools found all of the problems, and 40%
of the problems weren't found by any of the tools.  Even where the
problems were found, there was a surprising level of inconsistency
among the tools.

Unfortunately, there's not much detail in the presentation.  There's a
report that's been written, but so far not approved for release (or so
I'm told).  I don't know whether the issue is classification (they
don't want the bad guys to know what sort of things can sneak past
their detectors), or proprietary information, or just bureaucracy.

It would be interesting to hear comments from vendors on the list as
to the limitations on such a test (certainly using synthetic programs
isn't realistic), as well as whether they've adapted the tools to find
more of these types of problems.

--Jeremy

P.S. The report is undated, but I believe it's fairly recent.