Some Interesting Topics arising from the SANS/CWE Top 25

[coley at linus.mitre.org (Steven M. Christey)]

All, I'm the editor of the Top 25 list.  Thanks to Ken and others on SC-L
who provided some amazing feedback before its publication.  I hope we were
able to address most of your concerns and am sorry that we couldn't
address all of them.

Note that MITRE's site for the Top 25 is more technically detailed and has
lots of supporting documents than the SANS site, which is really a
jumping-off point.  See http://cwe.mitre.org/top25/.  Also, a process
document and changelog is on that site.

Here are some topics that arose during the construction of the Top 25. I
thought these might make some interesting points of debate or discussion
on this list:

1) The inclusion of output encoding (CWE-116) in conjunction with
   input validation (CWE-20) generated a lot of mixed reviews.  Part of it
   seems to come down to different ways of looking at the same problem.
   For example, is SQL injection strictly an input validation
   vulnerability, or output sanitization/validation/encoding or whatever
   you want to call it? In a database, the name "O'Reilly" may pass your
   input validation step, but you need to properly quote it before sending
   messages to the database.  And the actual database platform itself has
   no domain model to "validate" whether the incoming query is consistent
   with business logic.  My personal thinking, which seems reflected by
   many web application people, is that many injection issues are related
   to encoding at their core, and the role of input validation is more
   defense-in-depth (WITH RESPECT TO INJECTION ONLY).  Yet smart people
   insist that it's still input validation, even when presented with the
   example I gave.  So So what's the perspective difference that's causing
   the disconnect?

2) Countless mitigations were suggested by contributors on top of some
   of the ones already in the CWE entries (admittedly some of them
   weak).  Fortunately, we had time (for some definition of "time"
   that sometimes excluded a personal life) to update many of the core
   CWE entries.  Many mitigations had limitations, either in terms of
   impacts on usability, whether they could be applied at all in some
   circumstances, or if they were sufficiently effective.  The variety
   of customized mitigations is staggering, which to me suggests that
   more framework/methodology definition is needed.

3) Contributors advocated selecting list items based on how often the
   weakness appears in software (prevalence) and how severe the
   consequences are when the weakness leads to a vulnerable condition
   (severity).  Many people advocated using real-world data to make
   the determination for prevalence.  Problem: there's no real-world
   data available!  CVE vulnerability data is insufficient - they
   concentrate on the vulnerability side ("XSS") instead of the
   weakness side ("e.g. use of the wrong encoding at the wrong time").
   If people have real-world, weakness-focused data, then they aren't
   telling.

4) Some questions with respect to the assignment of severity scores
   led me to attempt to build a threat model and to try to more formally
   define other supporting fields like ease of detection, in light of the
   "skilled, determined attacker."  I don't think this model was
   sufficiently vetted, and I'm sure people will have concerns with how
   it's been defined (including "your threat model is really just talking
   about a threat agent.")  HOWEVER, I don't know of any other prioritized
   list that has tried to define some threat model to help with
   prioritization.  I would love to see this kind of investigation
   continue in other efforts.  (An acronym called CWSS comes to mind...)

5) The threat model, as roughly implied by how most people were
   "voting" for which items to include on the Top 25, treated availability
   as slightly less important than integrity and confidentiality.  Thus
   CWE-400 (Resource Consumption) had the dubious distinction of being
   number 26.  (CWE-400 and other also-rans are in the "On the Cusp"
   section of MITRE's Top 25 site). Clearly, availability may be more
   important in some environments e.g. critical infrastructure or
   e-commerce.  The unsurprising implications are that no single threat
   model will work for different types of organizations when composing a
   general list like the Top 25.  Thus it has a sort of fudge factor that
   helps make it generally applicable to organizations with varying threat
   environments, within some degree of tolerance.  It seems like a
   fundamental problem with a list of that sort.

6) Many expert reviewers will notice the varying abstraction levels
   and overlapping concepts in the list.  There are various explanations
   for this as summarized in the change logs and FAQ.  My main point in
   bringing this up was, a lot of people want things to be at a fixed
   level of abstraction and mutually exclusive, but I don't think it's
   feasible given our current understanding of security issues.  The CWE
   Research View (CWE-1000) tries to achieve mutual exclusivity, but are
   still working on it.  One of the things I think the CWE project has
   brought to the table overall is the semi-formal modeling of how
   weaknesses are inter-related, in the form of chains and composites [1].
   In short, it's difficult to have a mutually exclusive list because
   every kind of bug can trigger every other kind of bug! (example: buffer
   overflow could overwrite an adjacent numeric variable that leads to a
   calculation error, or vice versa).

7) Today, Chris Wysopal announced that Veracode had 61% coverage for
   the Top 25, and maybe they could boost that up to 75%.  The gap is
   probably related to design-level issues.  Chris noted that human
   analysis will always be required for the things that tools can't
   find, so software development teams need to find other approaches
   to fill the gap.  I think this has interesting implications for
   future research and prioritization.

8) Terminology and definitions varied widely, which is probably a
   surprise to noone.  This posed some problems in writing the
   document, and it's a general challenge for us in CWE and, I
   believe, a general challenge for the software community.

9) The OWASP Top Ten became a tool for PCI to easily establish minimum
   requirements for web applications, without PCI having to get into
   gory technical details.  This was contrary to OWASP's original
   intentions for the Top Ten as an educational tool.  SANS and MITRE
   both definitely have "technical clarification of expectations for
   assurance" as a goal, but the implications are a little unclear if
   this turns into "compliance."  Nonetheless, I see it as an
   important tool for non-software-security experts to ask for
   security in terms that are amenable to some form of measurement.

I grant that many of these issues may not be surprises to SC-L, but I
thought they were interesting.

- Steve

[1] http://cwe.mitre.org/data/reports/chains_and_composites.html