BSIMM update (informIT)

[coley at linus.mitre.org (Steven M. Christey)]

On Thu, 4 Feb 2010, Brad Arkin wrote:

> As a result, the count per ISV of bugs submitted to the Tipping Point 
> program is more of an indicator of what technology is popular in the 
> research community rather than how secure a product is.

Using anecdotal evidence from about 40,000 [sic] published CVEs over 10 
years, I'd tend to agree - my impression is that the applied research 
community is fickle, inconsistent, unpredictable, prone to fads, and far 
from being a unified demographic.  (which in one way is a good thing 
'cause it keeps things interesting.)

Did people know that a single person is responsible for a massive spike in 
symlink discoveries in 2008?  Just 'cause he felt like looking for that 
kind of problem, and he used his trusty grep program against a zillion 
shell scripts in various Debian packages.  So, what we thought was a vuln 
type that was mostly gone, isn't, because some guy decided to look for 
'em.

Don't get me started on the 15-year-old kid who spent a maximum of 10 
minutes on every downloadable/demo program he could find back in 2005 and 
gave us vuln DB people nightmares during the winter of '05-'06, because 
even though he wasn't skilled, many of his reports were correct.  His blog 
post on his super-l33t method was illuminating, but it was a "r0tten" time 
altogether.  Thankfully, he burned out and decided to go underground and 
privately share his new findings instead of publishing them.

Once upon a time, people screamed about how Firefox was so much secure 
because it had almost no security vulns, then the product hit some kind of 
magic market-share number and suddenly they're releasing a couple dozen 
advisories a year.  Coincidence?  Must be.

No need to mention the Oracle "unbreakable" promise and the near-immediate 
counter-argument from a couple researchers.

I've heard more than once from some professional researchers that they 
wouldn't be caught dead publishing an advisory about some generic XSS. 
They only bother to publish stuff that's interesting.  I know there's some 
science-y term for that kind of "publish only new stuff" phenomenon but I 
forget what it is.

Format string vulns got identified and nearly wiped out in the course of a 
couple years.  They were easy to find and fix, and they were fun to 
exploit.  But that was 8 or 9 years ago so that's going back far enough.

You can't trust stats based on public vuln disclosures, period.

http://marc.info/?l=bugtraq&m=113650260502218&w=2

(I know, I know, it's more than 4 years old so that's ancient history in 
Internet time and thus not worth paying attention to because everything 
today is just so new and different! ;-))

My personal opinion, backed by no hard stats whatsoever, is to look at the 
types of vulns that are disclosed as a slightly more reliable indication 
of a vendor's maturity.  If you don't have a standard term for it and you 
haven't seen it a hundred times before, then either it's a really, really 
new technology that hasn't been explored by a lot of people, or the 
developer has shaken the security tree pretty hard and removed the 
low-hanging fruit.

- Steve