Re: The Lost Decade of Security Metrics

[toby via Dailydave <dailydave () lists aitelfoundation org>]

I don't think you are wrong but your comparison of CVSS and the multiple
(also separately bad) metrics for a WAF isn't effective or accurate.

The values going into CVSS have something in common; they are attempts to
characterize the importance of the vulnerability in question. You are
making (have made before) the claim that the importance of a vulnerability
is too variable and specific to an environment or an attack scenario to be
effectively captured with the values included in CVSS.

In contrast, WAF throughput, false positives, and false negatives are
tradeoffs. They are equivalent to "cheap, fast, good; choose two". The
whole point of this set of values is that they can't be combined and as far
as I know, no one is trying to combine them at present. Neohapsis was the
last (qualified, thoughtful) group to try to actually produce scores for
IDS and Firewalls and I believe that died over drinks in Portland many many
years ago. No one deploying a WAF combines those. They use them to talk
about prioritization and risk tolerance. Is it better to impact
performance, risk false negatives, or suffer false positives? That's a
choice made by each company.

The equivalent of the WAF values for a vulnerability might be "importance
of the vuln", "confidence in the fix mitigating the vuln", "risk of the fix
breaking your environment". Or something like that.

The question is; do companies need CVSS to be better? Or is it sufficient
when used as a tool to think about vulnerabilities?

I'd suggest that you are more focused on the nuances of
vulnerabilities than most companies have time or desire to be. That makes
sense given what you've spent your career so far doing. But if you want to
change how things are done you can't insist that everyone step up to your
level of knowledge & focus. So, how do you create a repeatable process that
can be adopted and adapted to each company that uses it without requiring
someone who has spent 20 years deeply thinking about exploitation to do the
analysis of each new vuln?

t



On Tue, Jan 5, 2021 at 6:50 AM Dave Aitel via Dailydave <
dailydave () lists aitelfoundation org> wrote:

> A thousand years ago I subscribed to the Security Metrics mailing list.
> Metrics are important - or rather, I think good decision making is
> important, and without metrics your decision making is essentially luck.
> But we haven't seen any progress on this in a decade, and I wanted to talk
> about the meta-reason why: Oversimplification in the hopes of scaling.
>
> There's a theme in security metrics, a deep Wrong, that the community
> cannot correct, of trying to devolve features in their datasets to a single
> number. CVSS is the most obvious example, but Sasha's VEP paper here (
> https://www.lawfareblog.com/developing-objective-repeatable-scoring-system-vulnerability-equities-process)
> demonstrates most clearly the categorical example of the oversimplification
> issue, one that all of FIRST has  seemingly fallen into.
>
> If I took all the paintings in the world, and ran them through a neural
> network to score them 1.0 through 10.0, the resulting number would be, like
> CVSS, useless. Right now on the Metrics mailing list someone is soliciting
> for a survey where they ask people how they are using CVSS and how
> useful it might be for them. But the more useful you think CVSS is for you,
> the less useful it actually is being, since it can only lead you to wasting
> the little security budget you have. *CVSS is the phrenology of security
> metrics.* Being simple and easy to use does not make it helpful for
> rational decision making.
>
> If we want to make progress, we have to admit that we cannot join the
> false-positive and false-negative and throughput numbers of our WAF in any
> way. They must remain three different numbers. We can perhaps work on
> visualizing or representing this information differently, but they're in
> different dimensions and cannot be combined. The same is true for
> vulnerabilities. The reason security managers are reaching for a yes/no "Is
> there an exploit available" metric for patch prioritization is that CVSS
> does not work, and won't ever work, and despite the sunk cost the community
> has put into it, should be thrown out wholesale.
>
> -dave
> _______________________________________________
> Dailydave mailing list -- dailydave () lists aitelfoundation org
> To unsubscribe send an email to dailydave-leave () lists aitelfoundation org
_______________________________________________
Dailydave mailing list -- dailydave () lists aitelfoundation org
To unsubscribe send an email to dailydave-leave () lists aitelfoundation org