Comparing Scanning Tools

[leichter_jerrold at emc.com (leichter_jerrold at emc.com)]

| Date: Mon, 5 Jun 2006 16:50:17 -0400
| From: "McGovern, James F (HTSC, IT)" <James.McGovern at thehartford.com>
| To: sc-l at securecoding.org
| Subject: [SC-L] Comparing Scanning Tools
| 
| The industry analyst take on tools tends to be slightly different than
| software practitioners at times. Curious if anyone has looked at Fortify
and
| has formed any positive / negative / neutral opinions on this tool and
| others...
We evaluated a couple of static code scanning tools internally.  The
following is an extract from an analysis I did.  I've deliberately
omitted comparisons - you want to know about Fortify, not how it
compares to other products (which raises a whole bunch of other
issues), and included the text below.  Standard disclaimers:  This
is not EMC's position, it's my personal take.

Caveats:  This analysis is based on a 3-hour vendor presentation.  The
presenter may have made mistakes, and I certainly don't claim that my
recall of what he said is error-free.  A later discussion with others
familiar with Fortify indicated that the experience we had is typical,
but is not necessarily the right way to evaluate the tool.  Effective
use of Fortify requires building a set of rules appropriate to a
particular environment, method of working, constraints, etc., etc. 
This takes significant time (6 months to a year) and effort, but
it was claimed that once you've put in the effort, Fortify is a
very good security scanner.  I am not in a position to evaluate that
claim myself.

BTW, one thing not called out below is that Fortify can be quite slow.
Our experience in testing was that a Fortify scan took about twice as
long as a C++ compile/link cycle, unless you add "data flow" analysis -
in which case the time is much, much larger.

The brief summary:  In my personal view, Fortify is a worthwhile tool,
but it would not be my first choice.  (Given the opportunity to choose
two tools, it would probably be my second.)  Others involved in the
evaluation reached the opposite conclusion, and rated Fortify first.

                                                        -- Jerry

Fortify

Fortify is aimed as a tool for use in a security audit.  It is
deliberately biased in the direction of flagging all potential
security issues.  It provides two kinds of analysis - what they call
"semantic" and "data flow".  Neither use of terminology is consistent
with industry practice.  Their "semantic" analysis is better described
as a "syntactic" analysis:  It looks at surface features of the
program (use of certain calls, for example).  It mainly ignores
context.  Fortify's own representative describe this analysis as a
"super grep".  This analysis is driven by a large database of rules,
which can be extended.  (In industry practice, a semantic analysis
would look deeply at the meaning of the program.)

"Data flow" analysis is better called "taint analysis".  It traces all
data from external sources to find code that might incorrectly rely on
it.

When run on our code, semantic analysis reports about 3000 problems.
We looked closely at quite a number of them, and with a single
exception (where the code was so complex that no one could be sure),
they were false positives.  For a security audit, that's probably OK.
The problem is: What does one do with the false positives?  If this is
an isolated audit, the answer is - ignore them.  But in practice code
is always changing, so you'll have to audit it again.  How do you
avoid dealing with the same false positives every time?

A look at the problems showed that in many if not most of the cases we
looked at, there was no practical way to change the code to stop the
complaints.  You can permanently suppress individual complaints, but
doing so appears to be very hazardous:  The suppression is based on
the line on which the problem was found.  It could well be that this
is a false positive because of the surrounding context - and that
context might change.  For example, if a line has an array reference
a[i] that is safe because we can prove based on a test 10 lines
earlier that i is in range - something that Fortify itself cannot do,
since it does no value flow analysis - then we might suppress the
warning.  But a later revision to the code could eliminate the test,
and we apparently would not receive a warning.

The "data flow" analysis gives more useful results.  While the vendor
was here, we only had data flow analysis for a very small subset of
the code:  Producing this is an extremely long process (days of CPU
time).  Again, there were many false positives, though I would
consider almost all of them to be "legitimate", in the sense that the
program could not possibly have known certain things about our
environment that would be needed to eliminate the warning.  However,
in some cases, the messages were hard to understand:  There was
insufficient information in the result to figure out just why the
program had reached the conclusion it had.  Based on what the vendor
told us, effective use of the "data flow" analysis requires writing
rules that tell the product when "tainted" data becomes "cleansed".
This is a questionable procedure:  I recall a time I spent weeks
tracking down a bug that overwrote memory because of incorrect SNMP
data.  The bug was triggered way after the point any reasonable
security analyst would have considered the data "cleansed".  (The
issue was not with the data as such, but with its interaction with
deep, incorrect assumptions about its semantics.)

We found two bugs in the Fortify code's understanding of C++
constructs.  These lead to false positives that it would be impossible
to suppress except on a line-by-line basis.  The vendor didn't seem
particularly aggressive about noting these for fixing.  (He may have
noted it down or may just have a good memory.)  In general, I get the
feeling that Fortify is much more comfortable with C than with C++.
For example: Suppose a class has some public member functions, each of
which validate their arguments; and some private member functions,
which assume that they are passed valid data (e.g., that indexes are
in bounds).  Suppose that the public functions do actually ensure
valid data.  Fortify will complain that the private functions *might*
be security risks.  The only way around this - and it might not be
enough to shut Fortify up - is to add validation in all the private
functions as well.  This is not a style of programming that really
makes much sense.  The alternative of suppressing the warnings is
dangerous because someone could add a public function that does *not*
check its inputs correctly.

Summary

Fortify and XXX, despite being in what is nominally the same
market, are very different products.  Fortify has broader (but
shallower) coverage of known security problems.  Its tainted data
analysis could be very helpful in programs that take data from users
on the net and deal with it in fairly simple ways.  (For example, if
the only possible datatypes are things like strings and integers, with
the complexity all held elsewhere, you are probably OK marking the
function that converts to integer, or that copies the string to an
internal form, as "cleansers".)  It could be a useful "code auditors
workbench".  But as currently implemented, I don't see it as suitable
for maintaining correctness as code changes.