SANS Institute - CWE/SANS TOP 25 Most Dangerous ProgrammingErrors

[coley at linus.mitre.org (Steven M. Christey)]

On Tue, 13 Jan 2009, Gary McGraw wrote:

> I thought you might get a kick out of it.

I did! :-)  Always good to have debates.

> Executives don't care about technical bugs

No, but they do what PCI says they have to (i.e. listen to the OWASP Top
Ten).  They do care about the bottom line.  They hate buying software and
finding out how crappy it is afterward.

The Siemens example comes to mind.  It was mind-boggling to me to hear
that they were forced to pay 150% of the original cost just to get the
software they bought in a secure form.

http://www.networkworld.com/news/2009/011209-software-security-effort.html?page=2


> Too much focus on bugs.

The Top 25 has 4 or 5 items that are clearly design-related, maybe more if
you think that improper input validation is related to design, and still
more if you count all the "external control" items which may be
implementation or design depending on who's talking and what the
programmer's responsibilities are.  The Top 25 even has a couple classic
Saltzer-and-Schroeder examples in there (rephrased to avoid the incredible
confusion and misinterpretation that has gone on with the original S&S).

> Vulnerability lists help auditors more than developers.

Agree - except the Top 25 has anywhere from 3 to 10 specific
mitigations/preventions for each CWE.

And whether we like it or not, auditors help to drive change.  Maybe not
the optimal change, but they drive change.

Also - when the developers' software managers are told by their marketers
that they could lose many, then the managers will figure out how to get
the developers to improve.  Long-term thinking here - I know, that's not
allowed for this industry.

> One person's top bug is another person's yawner.

Absolutely, a point I brought up in the other post I made to SC-L on some
interesting challenges.

> Using bug parade lists for training leads to awareness but does not
> educate.

Yep - which is why we want universities to get cracking, and if the Top 25
helps to prod them on, then so be it.

> Bug lists change with the prevailing technology winds.

Yep - which is why the official name of the Top Ten begins with "2009".
We'll do this again next year.

> Top ten lists mix levels.

Also brought up in my last email as a problem for the industry in general.

Regarding Seven Pernicious Kingdoms - how does the Top 25 map to them?  I
could classify most of the Top 25 in multiple categories.  Should "poor
output encoding" be put under the Input validation kingdom?  Sounds kind
of like using the "Start" button in Windows to shut down ;-)

Should "cleartext transmission" be put under Environment?  or Security
Features?

Again, we *all* have this problem.

> Automated tools can find bugs . let them.

Yes, and a lesson of the Top 25 (that we all already know) is that when
people start to apply it, they'll see how a tool won't be a silver bullet.
Also covered in my last email...

> When it comes to testing, security requirements are more important than
> vulnerability lists.

Which is cover a teeny bit in the other email I sent.

Also, New York State has put up draft text that mentions the Top 25 as
part of a condition for acquisition.  Is that enough?  Hardly.  But things
like the Black/Williams "software facts label" aren't mature either, and
Dept. of Homeland Security probably has a couple years to go before their
work on assurance cases begins to take shape.

> Ten is not enough.

Neither is 25.  Number 26 (another design issue) is also mentioned in the
other email I posted.

I'm of the mindset that the Top 25 is, short-term, an awareness tool for
developers and for customers.  In the longer term, maybe it will be a
little blip on the road to actionable software assurance.  Given that
approximately 1000 people have created delicious bookmarks for it, and
I've alread seen comments from a couple developers "hey, we should go
check this out" - then we are already seeing some success.

Gary, it might seem ironic since I am leading the most comprehensive bug
parade out there in CWE, but I agree with you that just following the bug
parade is not enough.  The Top 25 is a means to an end, and not the end
itself.  Only time will tell, though.

- Steve