Re: Paper announcement: Is finding security holes a good idea?

[Oliver Friedrichs <oliver_friedrichs () symantec com>]

Eric, some thoughts (this is not an argument for or against 
full-disclosure, please take it as constructive criticism):

After reading your paper I agree that the data you used may support your 
arguments, however, you missed some important points.  You don't take into 
account the "type" of vulnerabilities being found in each of the 
applications that you've analyzed (you address Severity, but thats a 
seperate variable).  I would argue that if you did that, you would come to 
some different conclusions.  I would also argue that the caliber of bugs 
being found has increased quite substantially.  Easy to find, easy to 
exploit vulnerabilities have for the most part been exhausted in 
applications that have been "sufficiently" scrutinized.  The bugs being 
found today (in applications that have had a sufficient amount of 
scrutiny) are significantly more complex than a number of years ago (in 
the same application).  In the early 1990's, you could find a common 
buffer overflow (using a blatant strcpy/strcat) in many common core 
Internet applications.  Today, the vulnerabilities being discovered in 
these same applications are more complex off-by-one, signed/unsigned 
integer, compilar casting, or byte/character processing problems.  Some 
good examples of this behaviour are sendmail and BIND.  "Type" also infers 
the skill level required for a researcher to find a given vulnerability, 
due to it's difficulty, with some types being easy to find, while others 
extremely difficult.

Entirely new classes of vulnerabilities are rarely discovered (but they 
still are).  There will only be a finite quantity of these, and once those 
have been identified, the bar can't go any higher.  Of course there can 
only be a finite number of vulnerabilities in any given application as 
well.

Another two metrics that you don't measure is the amount of scrutiny that 
a particular application has had, or the size of an application.  A large 
application that has had 30 vulnerabilities found in it by one researcher 
over 10 years cannot be compared to a small application that has had 30 
vulnerabilities found in it by 200 researchers in 1 year.  There were 30 
vulnerabilities found in both, but the latter application will have 
improved in quality quite significantly, while the former not (assuming 
that both applications have the same average number of vulnerabilities per 
line of code).  Unfortunately scrutiny is likely something that you cannot 
measure in many applications.

So, on average I would argue that software quality (in terms of 
vulnerabilities being discovered) has improved for a given application 
that has, and continues to be, sufficiently scrutinized (not including 
substantial updates that introduce new bugs).  You simply don't have all 
of the data points to prove it, and therefore may be missing important 
conclusions.  I may be able to take specific applications, where we have 
sufficient visibility into scrutiny, size, and type/difficulty of 
vulnerabilities, and prove your theory wrong (sendmail is a possibility).

My conclusion isn't based on the numbers, but simply on my experience with 
researching vulnerabilities since the early 1990's.

Oliver Friedrichs
Sr. Manager - DeepSight
Symantec, Inc. (650) 381-8045

> Bugtraq readers might be interested in this paper:
>
>                    Is finding security holes a good idea?
>
>                              Eric Rescorla
>                    RTFM, Inc.   <http://www.rtfm.com/>
>
> A large amount of effort is expended every year on finding and patching
> security holes. The underlying rationale for this activity is that it
> increases welfare by decreasing the number of bugs available for
> discovery and exploitation by bad guys, thus reducing the total cost of
> intrusions. Given the amount of effort expended, we would expect to see
> noticeable results in terms of improved software quality. However, our
> investigation does not support a substantial quality improvement--the
> data does not allow us to exclude the possibility that the rate of bug
> finding in any given piece of software is constant over long periods of
> time. If there is little or no quality improvement, then we have no
> reason to believe that that the disclosure of bugs reduces the overall
> cost of intrusions.
>
> The paper can be downloaded from:
> http://www.rtfm.com/bugrate.pdf
> http://www.rtfm.com/bugrate.ps