Comparing Scanning Tools

[gem at cigital.com (Gary McGraw)]

Hi All,
 
Just a quick reminder that there is a chapter on code scanning technology and its application in "Software Security" 
(www.swsec.com).  Don't forget that these tools are best used as aids to make a smart human more efficient.  They do 
not replace the human, nor are they of much use among the clueless.  Every commercial tool has its issues, but the free 
tools ITS4, RATS, and flawfinder are not worth using at all anymore given tool evolution of late.  The chapter in 
"Software Security" discusses the history of these tools, how they actually work, and points to research in academia so 
you know where they're headed.  There are also pointers to most of the commercial tools.
 
We have found in our practice at Cigital that the most powerful applications of these tools involves developing 
specific and tailored coding guidelines for a given platform (say J2EE), building those guidelines to just so happen to 
cohere with security policy (shhh, tell no one), and then enforcing the guidelines by adding rules to a static analysis 
tool.  
 
Another tip: don't use the tools with all of the default rules all at once.  Carefully turn rules on and off and feed 
the results into dev along with training.  Use the tools as part of awareness and enforcement activities.
 
gem
Cigital www.cigital.com
Software Security www.swsec.com
Silver Bullet www.cigital.com/silverbullet 
 

        -----Original Message----- 
        From: sc-l-bounces at securecoding.org on behalf of leichter_jerrold at emc.com 
        Sent: Wed 6/7/2006 4:34 PM 
        To: James.McGovern at thehartford.com 
        Cc: sc-l at securecoding.org 
        Subject: Re: [SC-L] Comparing Scanning Tools
        
        

        | Date: Mon, 5 Jun 2006 16:50:17 -0400
        | From: "McGovern, James F (HTSC, IT)" <James.McGovern at thehartford.com>
        | To: sc-l at securecoding.org
        | Subject: [SC-L] Comparing Scanning Tools
        |
        | The industry analyst take on tools tends to be slightly different than
        | software practitioners at times. Curious if anyone has looked at Fortify
        and
        | has formed any positive / negative / neutral opinions on this tool and
        | others...
        We evaluated a couple of static code scanning tools internally.  The
        following is an extract from an analysis I did.  I've deliberately
        omitted comparisons - you want to know about Fortify, not how it
        compares to other products (which raises a whole bunch of other
        issues), and included the text below.  Standard disclaimers:  This
        is not EMC's position, it's my personal take.
        
        Caveats:  This analysis is based on a 3-hour vendor presentation.  The
        presenter may have made mistakes, and I certainly don't claim that my
        recall of what he said is error-free.  A later discussion with others
        familiar with Fortify indicated that the experience we had is typical,
        but is not necessarily the right way to evaluate the tool.  Effective
        use of Fortify requires building a set of rules appropriate to a
        particular environment, method of working, constraints, etc., etc.
        This takes significant time (6 months to a year) and effort, but
        it was claimed that once you've put in the effort, Fortify is a
        very good security scanner.  I am not in a position to evaluate that
        claim myself.
        
        BTW, one thing not called out below is that Fortify can be quite slow.
        Our experience in testing was that a Fortify scan took about twice as
        long as a C++ compile/link cycle, unless you add "data flow" analysis -
        in which case the time is much, much larger.
        
        The brief summary:  In my personal view, Fortify is a worthwhile tool,
        but it would not be my first choice.  (Given the opportunity to choose
        two tools, it would probably be my second.)  Others involved in the
        evaluation reached the opposite conclusion, and rated Fortify first.
        
                                                                -- Jerry
        
        Fortify
        
        Fortify is aimed as a tool for use in a security audit.  It is
        deliberately biased in the direction of flagging all potential
        security issues.  It provides two kinds of analysis - what they call
        "semantic" and "data flow".  Neither use of terminology is consistent
        with industry practice.  Their "semantic" analysis is better described
        as a "syntactic" analysis:  It looks at surface features of the
        program (use of certain calls, for example).  It mainly ignores
        context.  Fortify's own representative describe this analysis as a
        "super grep".  This analysis is driven by a large database of rules,
        which can be extended.  (In industry practice, a semantic analysis
        would look deeply at the meaning of the program.)
        
        "Data flow" analysis is better called "taint analysis".  It traces all
        data from external sources to find code that might incorrectly rely on
        it.
        
        When run on our code, semantic analysis reports about 3000 problems.
        We looked closely at quite a number of them, and with a single
        exception (where the code was so complex that no one could be sure),
        they were false positives.  For a security audit, that's probably OK.
        The problem is: What does one do with the false positives?  If this is
        an isolated audit, the answer is - ignore them.  But in practice code
        is always changing, so you'll have to audit it again.  How do you
        avoid dealing with the same false positives every time?
        
        A look at the problems showed that in many if not most of the cases we
        looked at, there was no practical way to change the code to stop the
        complaints.  You can permanently suppress individual complaints, but
        doing so appears to be very hazardous:  The suppression is based on
        the line on which the problem was found.  It could well be that this
        is a false positive because of the surrounding context - and that
        context might change.  For example, if a line has an array reference
        a[i] that is safe because we can prove based on a test 10 lines
        earlier that i is in range - something that Fortify itself cannot do,
        since it does no value flow analysis - then we might suppress the
        warning.  But a later revision to the code could eliminate the test,
        and we apparently would not receive a warning.
        
        The "data flow" analysis gives more useful results.  While the vendor
        was here, we only had data flow analysis for a very small subset of
        the code:  Producing this is an extremely long process (days of CPU
        time).  Again, there were many false positives, though I would
        consider almost all of them to be "legitimate", in the sense that the
        program could not possibly have known certain things about our
        environment that would be needed to eliminate the warning.  However,
        in some cases, the messages were hard to understand:  There was
        insufficient information in the result to figure out just why the
        program had reached the conclusion it had.  Based on what the vendor
        told us, effective use of the "data flow" analysis requires writing
        rules that tell the product when "tainted" data becomes "cleansed".
        This is a questionable procedure:  I recall a time I spent weeks
        tracking down a bug that overwrote memory because of incorrect SNMP
        data.  The bug was triggered way after the point any reasonable
        security analyst would have considered the data "cleansed".  (The
        issue was not with the data as such, but with its interaction with
        deep, incorrect assumptions about its semantics.)
        
        We found two bugs in the Fortify code's understanding of C++
        constructs.  These lead to false positives that it would be impossible
        to suppress except on a line-by-line basis.  The vendor didn't seem
        particularly aggressive about noting these for fixing.  (He may have
        noted it down or may just have a good memory.)  In general, I get the
        feeling that Fortify is much more comfortable with C than with C++.
        For example: Suppose a class has some public member functions, each of
        which validate their arguments; and some private member functions,
        which assume that they are passed valid data (e.g., that indexes are
        in bounds).  Suppose that the public functions do actually ensure
        valid data.  Fortify will complain that the private functions *might*
        be security risks.  The only way around this - and it might not be
        enough to shut Fortify up - is to add validation in all the private
        functions as well.  This is not a style of programming that really
        makes much sense.  The alternative of suppressing the warnings is
        dangerous because someone could add a public function that does *not*
        check its inputs correctly.
        
        Summary
        
        Fortify and XXX, despite being in what is nominally the same
        market, are very different products.  Fortify has broader (but
        shallower) coverage of known security problems.  Its tainted data
        analysis could be very helpful in programs that take data from users
        on the net and deal with it in fairly simple ways.  (For example, if
        the only possible datatypes are things like strings and integers, with
        the complexity all held elsewhere, you are probably OK marking the
        function that converts to integer, or that copies the string to an
        internal form, as "cleansers".)  It could be a useful "code auditors
        workbench".  But as currently implemented, I don't see it as suitable
        for maintaining correctness as code changes.
        _______________________________________________
        Secure Coding mailing list (SC-L)
        SC-L at securecoding.org
        List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
        List charter available at - http://www.securecoding.org/list/charter.php
        



----------------------------------------------------------------------------
This electronic message transmission contains information that may be
confidential or privileged.  The information contained herein is intended
solely for the recipient and use by any other party is not authorized.  If
you are not the intended recipient (or otherwise authorized to receive this
message by the intended recipient), any disclosure, copying, distribution or
use of the contents of the information is prohibited.  If you have received
this electronic message transmission in error, please contact the sender by
reply email and delete all copies of this message.  Cigital, Inc. accepts no
responsibility for any loss or damage resulting directly or indirectly from
the use of this email or its contents.
Thank You.
----------------------------------------------------------------------------