NSA comparison of source code analysis tools

[colin.cassidy at ge.com (Cassidy, Colin (GE Infra, Energy))]

The document properties suggests June 2009, and it's a shame that there
isn't much details as we are looking to evaluate 3 of the code analysis
tools for our development here.

CJC 

> -----Original Message-----
> From: sc-l-bounces at securecoding.org 
> [mailto:sc-l-bounces at securecoding.org] On Behalf Of Jeremy Epstein
> Sent: 29 September 2009 14:49
> To: sc-l
> Subject: [SC-L] NSA comparison of source code analysis tools
>
> (Apologies if I already sent this to the group; I don't think I did.)
>
> There's an interesting presentation at
> http://www.iarpa.gov/stonesoup_Merced_DHSAWGbrief.pdf about a study
> done by the US NSA (National Security Agency) of C and Java source
> code analysis tools.  They developed a synthetic test suite, and then
> ran six tools against the Java version and five tools against the C
> version (the specific tools and versions used are identified in the
> presentation).  None of the tools found all of the problems, and 40%
> of the problems weren't found by any of the tools.  Even where the
> problems were found, there was a surprising level of inconsistency
> among the tools.
>
> Unfortunately, there's not much detail in the presentation.  There's a
> report that's been written, but so far not approved for release (or so
> I'm told).  I don't know whether the issue is classification (they
> don't want the bad guys to know what sort of things can sneak past
> their detectors), or proprietary information, or just bureaucracy.
>
> It would be interesting to hear comments from vendors on the list as
> to the limitations on such a test (certainly using synthetic programs
> isn't realistic), as well as whether they've adapted the tools to find
> more of these types of problems.
>
> --Jeremy
>
> P.S. The report is undated, but I believe it's fairly recent.
> _______________________________________________
> Secure Coding mailing list (SC-L) SC-L at securecoding.org
> List information, subscriptions, etc - 
> http://krvw.com/mailman/listinfo/sc-l
> List charter available at - 
> http://www.securecoding.org/list/charter.php
> SC-L is hosted and moderated by KRvW Associates, LLC 
> (http://www.KRvW.com)
> as a free, non-commercial service to the software security community.
> _______________________________________________
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 4427 bytes
Desc: not available
URL: <http://krvw.com/pipermail/sc-l/attachments/20090929/3d794a4b/attachment.bin>