Re: CVSS is the worst compression algorithm ever

["Monroe, Bruce" <bruce.monroe () intel com>]

Uh no. CVSS scores a vulnerability and if it’s a vendor we’re scoring that without knowing how you have the vulnerable 
software/firmware/hardware/ect deployed in your environment. It’s why the CVSS Base Score is worst case. The resulting 
CVSS V3 vulnerability score is one element you can then calculate into your overall risk factoring. It’s the orgs job 
consuming the CVSS V3x vulnerability score to determine their risk and set their patching priorities.

Other Factors to consider for Risk (not a comprehensive list but it’s a good start)


  *   Ease of exploit
  *   Delivery Mechanism (is the vuln available remotely over the network? Do have to have code running locally on the 
box (if so I’ve already got code running on the platform) The more access I need to the platform the less likely the 
issue is of being exploited in the wild.
  *   Availability and sophistication level of exploit: Paper, PoC, weaponized exploit, etc…
  *   Detailed Asset knowledge for the infrastructure and how vulnerable component is deployed. Hard to determine risk 
if you don’t know what you have and how you have systems deployed.
  *   Controls that would mitigate the attack vectors (firewalls, vulnerable component not exposed, etc…)
  *   Is your deployment actually exposing the vulnerability – (Example – you may have a library in your application 
but are not exposing or using the vulnerable function)
  *   % of environment that is exposed, # of systems impacted, $ value of those systems in terms of keeping the 
business running.
  *   Where are the impacted systems located, if a system has more than one interface choose the worst
  *   Are the impacted systems multi-homed (laptops, tablets, et al.) Do they live on your network and go home to an 
ISP that’s the wild west for malware for example.
  *   What are the bottom line consequence to your organization if this issue is successfully exploited (Think $$$$, 
potential impact to Brand, stock prices…).
  *   What is the highest classification of information that could potentially be exposed? PII, Core IP, keys to the 
kingdom,…
  *   Business impact if issue is exploited <5 % of business impacted, 5-10% of business impacted, 10-25% of business 
impacted,…,
  *   Likelihood of exploit (put on your SWAMI hat here…😉)
  *   If you ever see a network RCE that is exposed by default that’s a red flag. We don’t see too many of these today 
but there is still an occasional one.

There needs to be significant thought put in to determine your orgs overall risk and what bubbles up to the top in 
terms of patching priority for typically limited resources to get those mitigations deployed.

As mentioned CVSS vulnerability scores is a single data point in making that assessment. I personally think we need a 
new tool, mechanism, that is an industry standard but that is beyond the scope of CVSS. While some improvements are 
planned for CVSS V4 this is a separate problem that needs to be solved outside of CVSS imo.

CVSS isn’t perfect but it’s pretty good at what it’s targeted to do. Use the right tool, if you need a hammer don’t try 
to use a screwdriver. If you’re using CVSS as the end all be all for Risk you’re using it wrong, it’s a single element 
to input into that overall calculation.

Regards,
Bruce

Opinions expressed are my own and may not reflect those of my employer.
From: Dailydave <dailydave-bounces () lists immunityinc com> On Behalf Of Adrian Sanabria
Sent: Thursday, January 10, 2019 8:02 AM
To: Wim Remes <wremes () gmail com>
Cc: dailydave () lists immunityinc com
Subject: Re: [Dailydave] CVSS is the worst compression algorithm ever

Okay, we keep touching on this point, that CVSS isn't intended to score risk, just vulnerability severity. I'm having a 
hard time seeing what value there is in having a vulnerability score that doesn't reflect risk. What use does it have?

Or is that exactly what we're saying? That since it doesn't reflect risk, it's essentially useless. If that's the 
conclusion, I'm on the same page.

--Adrian

On Thu, Jan 10, 2019, 9:56 AM Wim Remes <wremes () gmail com<mailto:wremes () gmail com> wrote:
Hi,

Bruce really hits the nail on the head here. CVSS != Risk. To broaden that discussion and not waste too many words, 
I’ll reference FAIR (Factor Analysis of Information Risk, https://www.fairinstitute.org/what-is-fair) to indicate where 
“Vulnerability” contributes to an eventual quantitative risk valuation.

I also always considered CVSS scoring to be qualitative instead of quantitative and the numbers to be ordinal. That 
makes them fine for ranking vulnerability, but horrible to perform math on (Jet Fuel x Peanut Butter = Shiny — hi Alex 
Hutton!).

That said, it all boils down to a point I’ve been rapping on about for a long long time now. Organizations should not 
expect third party penetration testers to make an accurate assessment of risk. The data provided by a third party 
penetration tester should feed into your risk management framework, that is also fed with internally acquired business 
data, to produce (or adjust) a risk valuation. It would be helpful if we, as consultants, wouldn’t pretend that we (a) 
can come up with any form of credible risk score during such assessments and (b) are delivering scoring that can help 
with prioritization in a business context without additional effort on the client side. On the other hand, clients that 
have a risk management framework that can actually take vulnerability scores and use them to generate risk scores 
should be clear in what they expect from us. If you are asked, whether in an RFP or an SoW, to produce a risk score for 
your findings at the very least you should be returning a question for asset valuation and threat community 
descriptions.

Cheers,
Wim




On 8 Jan 2019, at 18:33, Monroe, Bruce <bruce.monroe () intel com<mailto:bruce.monroe () intel com>> wrote:

Hi Dave,

I participate on the CVSS SIG being ran out of FIRST that is working on improvements to CVSS. So do a number of people 
out of CERT CC, NIST, MITRE along with a good representation of industry. A number of us provided feedback on this 
paper. CVSS is for scoring the severity of a vulnerability. CVSS does not = Risk.

My understanding is there is a number of government entities that believe CVSS does = Risk and are using it in a vacuum 
for that purpose. While the CVSS score is a single component - you also must look at how the vulnerable component is 
deployed, controls in place, value of asset, patching windows, likelihood of exploit,ect…there is a lot that goes into 
determining risk.

The fact that various USG entities is using CVSS wrong is an education issue imo. Yes CVSS has it’s issues with some of 
it’s elements being subjective eye of the beholder type items but that isn’t the reason for this paper…they’ve got USG 
people using it in a vacuum when it’s only a single element of determining your orgs risk due to a vulnerability. That 
isn’t a CVSS problem that’s a vulnerability management 101 problem.

Regards,
Bruce
Intel PSIRT

Opinions expressed are my own and may not reflect those of my employer.
From: Dailydave <dailydave-bounces () lists immunityinc com<mailto:dailydave-bounces () lists immunityinc com>> On 
Behalf Of Dave Aitel
Sent: Tuesday, January 08, 2019 8:14 AM
To: dailydave () lists immunityinc com<mailto:dailydave () lists immunityinc com>
Subject: [Dailydave] CVSS is the worst compression algorithm ever


I wanted to take a few minutes and do a quick highlight of a paper from CMU-CERT which I think most people have missed 
out on: https://resources.sei.cmu.edu/asset_files/WhitePaper/2018_019_001_538372.pdf
Towards Improving CVSS - 
resources.sei.cmu.edu<https://resources.sei.cmu.edu/asset_files/WhitePaper/2018_019_001_538372.pdf>
resources.sei.cmu.edu<http://resources.sei.cmu.edu>
SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY REV-03.18.2016.0 Distribution Statement A: Approved for 
Public Release; Distribution Is Unlimited TOWARDS IMPROVING CVSS

It's almost as funny a read as their previous best work on how "clientless HTTPS VPNs are insanely 
dumb<https://www.kb.cert.org/vuls/id/261869/> what were you thinking omg?"

They use a ton of big words in the paper to call CVSS out and give it a shellacking. Like most of you, we have 
extensive use of CVSS in our consulting practice and I've seen this stuff first hand. CVSS is of course just a buggy 
compression algorithm for taking complex qualitative data and then putting it on a number line. The paper has three 
angles here:

  1.  Qualitative mappings into quantitative numbers are a silly thing to do, like people trying to do "social science" 
by using SurveyMonkey.
  2.  We're pretty sure that the compression algorithm is not, in fact, putting higher risk items as bigger numbers, 
which is the whole point of the thing.
  3.  Nobody is applying this in any sort of consistent way (which is probably impossible) which is ALSO the whole 
point of the thing.

It's fine to have a lossy compression algorithm that emphasizes certain aspects of the input signal over others, of 
course, but an additional CERT/CC critique is we have no reason to think CVSS does this in any useful way.



There's definitely people in the CVSS process (who I will avoid calling out by name) who think ANY quantization is 
good. But read the paper and decide for yourself - because these are probably serious issues that are turning your 
entire risk org into a Garbage-In-Garbage-Out org...



-dave


_______________________________________________
Dailydave mailing list
Dailydave () lists immunityinc com<mailto:Dailydave () lists immunityinc com>
https://lists.immunityinc.com/mailman/listinfo/dailydave

_______________________________________________
Dailydave mailing list
Dailydave () lists immunityinc com<mailto:Dailydave () lists immunityinc com>
https://lists.immunityinc.com/mailman/listinfo/dailydave

_______________________________________________
Dailydave mailing list
Dailydave () lists immunityinc com
https://lists.immunityinc.com/mailman/listinfo/dailydave