Re: Graphing: Don't believe everything you see.

[jf <jf () danglingpointers net>]

Really, almost all of these metrics are flawed- of the critical
vulnerabilities listed many of them are things like critical bug in
OpenSSL, problems in ftp proxy with IPv66 sockets, et cetera; which I guess
depending on who you are, may or may not be critical, but to most of us
who aren't using any type of proxy or IPv66 sockets, it's not so important.

This is important to take into account when reviewing those number of
critical bugs comparisons. If we compare MS Office to OpenOffice in this
light, it would show that OO is greatly superior in security to MS Office
because of the number of critical flaws found, but I'd be willing to bet
that many of us may not necessarily agree with that conjecture. The number
of reported bugs are just that, and shouldn't be used as a metric to
determine if a product is secure or not. (however, when a bug is reported
and then some time in the distant future another similar bug affecting the
same region of code does indicate a failure on the vendors part to really
care at all, which IMHO is a much better metric)

Then we have things like 'time to patch' metrics, which are also flawed,
for instance does MS release patches for third-party products, or rather
if there is (yet another) bug in a CA product and MS doesn't patch it, do
we count that against them? Why do we do that for Redhat? Maybe that isn't
the best point as Redhat did indeed ship with a product, but where does
responsibility lie? What if the bug is on the 'extras' CD in an unstable
directory, do we count that? How about if it took organization Y several
weeks to produce a patch for their product and then in less than Z hours
the OS vendor provides the patch to their customers, do we count the time
as 'several weeks' or Z hours? That all said, because of different models,
comparing time to patch for Windows to Linux/BSD/any of the OSs that
comprise of mostly third party applications provides a false view of the
situation.

As for the graphs, they provide an idea of the potential amount of bugs,
but provide no real firm data. Speaking in a sense of probability of
course. To declare however that one product is more secure than another
simply based off of a graph like that is absurd and silly, and I think
everyone realizes this.



-- 

        Success is not final, failure is not fatal:
        it is the courage to continue that counts.

        -- Sir Winston Churchill

On Wed, 7 Feb 2007, Robert E. Lee wrote:

> Date: Wed, 07 Feb 2007 17:05:46 +0100
> From: Robert E. Lee <robert () dyadsecurity com>
> To: dailydave () lists immunitysec com
> Subject: Re: [Dailydave] Graphing: Don't believe everything you see.
>
> George Ou wrote:
>  > Take a look at Microsoft SQL 2005 and you'll see that's been ROCK
> SOLID with
>  > ZERO vulnerabilities.
>  > http://secunia.com/product/6782/?task=advisories
>  > Compare that to the mess of Oracle over the same time period.
>  >
>  > So let's not base our analysis on some stupid trumped up diagram and
> let's
>  > not make stupid generalizations about platforms.  Let's try and be
> objective
>  > and factual.
>
> In the spirit of "[silly] generalizations"....  the number of
> vulnerabilities publicly disclosed for a product doesn't seem to be a
> valid metric for measuring security between products. There are different
> disclosure policies for every organization/product.  Some applications
> are just going to get more attention than others.
>
> Closed source vs Open Source changes the methods available to an outside
> researcher for testing.  For results to be compared, the same tests have
> to be run
> equally for both projects.
>
> Comparing the end result (vulnerability count) without taking into account
> how we got to the end result (testing methodology) reminds me a bit of:
>
> "If... she... weighs... the same as a duck,... she's made of wood. And
> therefore? A witch!!!"
>
> Cheers :),
>
> Robert
_______________________________________________
Dailydave mailing list
Dailydave () lists immunitysec com
http://lists.immunitysec.com/mailman/listinfo/dailydave