Really dumb questions?

[jsteven at cigital.com (John Steven)]

James,

Not dumb questions: an unfortunate situation. I do tool bakeoffs for clients a fair amount. I'm responsible for the 
rules Cigital initially sold to Fortify. I also attempt to work closely with companies like Coverity and understand 
deeply the underpinnings of that tool's engine. I've a fair amount of experience with Klocwork, unfortunately less with 
Ounce.

I understand the situation like this: technical guys at each of these companies are all great guys, smart, and 
understand their tool's capabilities and focus. They accurately describe what their tool does and don't misrepresent it.

On the other hand, I've experienced competition bashing in the sales process as I've helped companies with tool 
selections and bake offs. I see NO value in this. As I said in a previous post to this list: the tools differ both 
macroscopically in terms of approach and microscopically in terms rule implementation. Please see my previous post 
about bake-offs and such if you'd like more information on how to disambiguate tool capabilities objectively.

No blanket statement about quality or security fits any vendors' tool; ANY vendor. Ignore this level of commentary by 
the vendors.*(1)

No boolean answer exists to your question, let me give you some of my experiences:


 *   Fortify's SCA possesses far-and-away the largest rule set, covering both topics people consider purely security 
and those that may-or-may-not create opportunity for exploit (often when combined with other factors) which one may 
call quality rules. My impression is that SCA can be effectively used by Security Folk, QA Folk, or developers with a 
mind to improve the quality or security of their code. Recent inclusion of Findbugs bolsters SCA's capabilities to give 
code quality commentary.


 *   Coverity's Prevent often gets pigeon-holed as "a quality tool", but does an exceptional job of tracking down 
memory issues in C, C++. Skilled security consultants will tell you that failing to fix Prevent's results in your code 
will result in various memory-based command injection opportunities (BO, format string, write-anywhere's, etc.). It 
also effectively targets time-and-state issues, as well as other classes of bug. Prevent can effectively be used by 
Security Folk and Developers (or your rare hardcore QA person) to improve code quality and squelch opportunity for 
exploit.


 *   Klocwork's tool targets rule spaces similar to Fortify, but possesses less. Often pegged as a quality tool (as 
well), do find its UI (more than its engine) possess helpful features that only a QA professional would enjoy. This 
includes its defect density calculation, "reverse engineering" capabilities, and its reporting/time-series style. 
Klocwork can be effectively used by a Security guy to find security bugs, but I believe Fortify and Ounce have widened 
the rules gap in recent years.

Tackling your other questions in rapid succession:

There is no difference, technically, between the ability to scan for quality or security. However, each engine focuses 
on parsing and keeping track of only that state which provides meaningful data to their rules. You can imagine that 
Fortify carries a fair amount of information about where data came from and what functions may be dangerous and can 
therefore support new security rules easily. They don't carry around information to aggregate defect density readily 
like K7 can. Does this make one intrinsically better than the other for quality or security? Perhaps having worked on 
static analysis tools I'm cranky but I say, "No." If the market clearly mandated something specifically, all the 
vendors would augment their engine to support it. Some would be in a better position to offer it than others.

When I talk to vendors about COBOL and similar support they shudder. I think this space represents a huge opportunity 
for the first vendor to support it, but as a commercial organization, I wouldn't hold your breath on near-term support.

I could answer how these tools support new languages, but that doesn't seem like public domain knowledge. I'll let the 
vendors tackle that 'un.

----
John Steven
Technical Director; Principal, Software Security Group
Direct: (703) 404-5726 Cell: (703) 727-4034
Key fingerprint = 4772 F7F3 1019 4668 62AD  94B0 AE7F EEF4 62D5 F908

Blog: http://www.cigital.com/justiceleague
Papers: http://www.cigital.com/papers/jsteven

http://www.cigital.com
Software Confidence. Achieved.


*(1) I'm also explicitly dodging the quality vs. security debate here. Having read/posted to this list for the last 7 
years, that semi-annual topic has been flogged more than your average dead equine.


________________________________
From: "McGovern, James F (HTSC, IT)" <James.McGovern at thehartford.com>

Most recently, we have met with a variety of vendors including but not
limited to: Coverity, Ounce Labs, Fortify, Klocwork, HP and so on. In
the conversation they all used interesting phrases to describe they
classify their competitors value proposition. At some level, this has
managed to confuse me and I would love if someone could provide a way to
think about this in a more boolean way.

- So when a vendor says that they are focused on quality and not
security, and vice versa what exactly does this mean? I don't have a
great mental model of something that is a security concern that isn't a
predictor of quality. Likewise, in terms of quality, other than
producing metrics on things such as depth of inheritance, cyclomatic
complexity, etc wouldn't bad numbers here at least be a predictor of a
bad design and therefore warrant deeper inspection from a security
perspective?

- Even if the rationale is more about people focus rather than tool
capability, is there anything else that would prevent one tool from
serving both purposes?

- Is it reasonable to expect that all of the vendors in this space will
have the ability to support COBOL, Ruby and Smalltalk sometime next year
so that customers don't have to specifically request it?

- Do the underlying compilers need to do something different since
languages such as COBOL aren't object-oriented which would make analysis
a bit different?