Dailydave mailing list archives
Re: CVSS is the worst compression algorithm ever
From: Adrian Sanabria <adrian.sanabria () gmail com>
Date: Thu, 10 Jan 2019 16:38:30 -0500
I probably shouldn't have brought it up - I'm not involved much on the pentesting side. I know we've discussed replacing it, but finding little out there to replace it with. In my own work, I find most of my pentesting results come down to a binary value (hackable, not hackable) and some sense of likelihood of it getting exploited by a malicious party. Highs/mediums/lows all seem pointless when emulating the attacker perspective. Looking at DREAD, I honestly can't say I find anything fatally wrong with it. Perhaps it's because I've never known pentesting to be terribly consistent across tests or consultants in my career, so the bar is set pretty low in my mind? --Adrian On Thu, Jan 10, 2019 at 2:12 PM Adam Shostack <adam () shostack org> wrote:
On Wed, Jan 09, 2019 at 08:18:48AM -0500, Adrian Sanabria wrote:Our pentesters use DREAD, which I think most people have moved on from,but atleast the scoring is clear and consistent.I'm sorry, but I need to rant a little. A decade back, I wrote a "DREAD is DEAD, please stop" blog post for Microsoft. If you are getting consistent scoring out of DREAD, you are not using DREAD (as described in Writing Secure Code 1, which I think is the first public description). You are using some derivitive that adds tools to provide for that consistency. Those tools may be as simple as a set of examples of each of the constiuents and what constitutes a 7 or a 3. I care about this because I one of the biggest things that I see making threat modeling hard is everyone calls their specific collection of techniques 'lightweight threat modeling' or 'agile threat modeling' and people trying to learn get confused because there's 6 contradictory descriptions that have been labeled "agile tm". People writing down process so their engineers can do it consistently get confused in the same way they'd get confused if we all said "oh yeah. we're writing code, and you can assign variables with either = or <=". We name our languages, we version them. We need to start doing the same for threat modeling constructs. If you say "We're using DREADNOP 1.0" that's cool. Alternately, maybe you're using DREAD 1.0 in its raw form, in which case, how are you getting consistent scores? AdamIn addition to CVE being wrong on critical details, I've found that mostofExploitDB isn't exploits. Many are vulnerability checks and almost allareincorrectly entered. PrivEsc will be labeled RCE and RCE will be labeledDoS.It's all a mess. If I had the resources to burn it all down and startfromscratch, I'd do it. --Adrian On Tue, Jan 8, 2019, 17:47 Nathaniel Ferguson <jferguson () 126 com wrote: > They use a ton of big words in the paper to call CVSS out and giveit ashellacking. Like most of you, we have extensive use of CVSS in our consulting practice and I've seen this stuff first hand. CVSS is ofcoursejust a buggy compression algorithm for taking complex qualitativedata andthen putting it on a number line. Over the years I've worked at a few different consultancies and atleastoriginally basically no one used any sort of standardized metric, the reports were generally humorous from a technical standpoint as thenumberswere basically just made up and didn't adhere to even basicstatisticsmethodologies-- we take the X and multiple it by Y and add the Z and there's your score! Some even plotted them along cartoon lookinggraphs andplots and my suspicion was that they were really included to givedepth tothe reports and break up the monotony of text on text on text.Oddly, I'venever even worked at a place that described the methodology asoutlined intheir reports to their employees leaving some question as to how a methodology was to be implemented if only the client ever heardabout it.In that sense, CVSS et al make some amount of sense, if nothing elseitstandardizes to a common metric and isn't what the sales guy oroperationsmanager made up. Additionally, what a strange word-- shellacking,there isno 'k' in the word until its made into a present participle. > The paper has three angles here: > Qualitative mappings into quantitative numbers are a silly thingto do,like people trying to do "social science" by using SurveyMonkey. Which is what most people are or were selling. > It's fine to have a lossy compression algorithm that emphasizescertainaspects of the input signal over others, of course, but anadditional CERT/CC critique is we have no reason to think CVSS does this in anyuseful way.Well there 's a missing line here, you can see it from the way that client-side attacks perverted the concept of remote and so they madethemremote also instead of adding the new line to the plot. Because ofstufflike this. everything is remote now which limits its usefulness. This doesn't even touch on the fact that most of the CVE database isbasicallywrong from submissions including very limited data, id est "memory corruption results in a DoS". Nathaniel 在 2019-01-09 00:14:00,"Dave Aitel" <dave.aitel () cyxtera com> 写道: I wanted to take a few minutes and do a quick highlight of apaper fromCMU-CERT which I think most people have missed out on: https://resources.sei.cmu.edu/asset_files/WhitePaper/2018_019_001_538372.pdfTowards Improving CVSS - resources.sei.cmu.edu resources.sei.cmu.edu SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY REV-03.18.2016.0 Distribution Statement A: Approved for PublicRelease;Distribution Is Unlimited TOWARDS IMPROVING CVSS It's almost as funny a read as their previous best work on how " clientless HTTPS VPNs are insanely dumb what were you thinkingomg?"They use a ton of big words in the paper to call CVSS out andgive it ashellacking. Like most of you, we have extensive use of CVSS inourconsulting practice and I've seen this stuff first hand. CVSS isofcourse just a buggy compression algorithm for taking complex qualitative data and then putting it on a number line. The paperhasthree angles here: 1. Qualitative mappings into quantitative numbers are a sillything todo, like people trying to do "social science" by using SurveyMonkey. 2. We're pretty sure that the compression algorithm is not, infact,putting higher risk items as bigger numbers, which is thewholepoint of the thing. 3. Nobody is applying this in any sort of consistent way (whichisprobably impossible) which is ALSO the whole point of thething.It's fine to have a lossy compression algorithm that emphasizescertainaspects of the input signal over others, of course, but anadditionalCERT/CC critique is we have no reason to think CVSS does this inanyuseful way. There's definitely people in the CVSS process (who I will avoidcallingout by name) who think ANY quantization is good. But read thepaper anddecide for yourself - because these are probably serious issuesthatare turning your entire risk org into a Garbage-In-Garbage-Outorg...-dave _______________________________________________ Dailydave mailing list Dailydave () lists immunityinc com https://lists.immunityinc.com/mailman/listinfo/dailydave_______________________________________________ Dailydave mailing list Dailydave () lists immunityinc com https://lists.immunityinc.com/mailman/listinfo/dailydave-- Adam Shostack President, Shostack & Associates https://associates.shostack.org • +1 917 391 2168 Join my very quiet annnoucement list: https://adam.shostack.org/newthing
_______________________________________________ Dailydave mailing list Dailydave () lists immunityinc com https://lists.immunityinc.com/mailman/listinfo/dailydave
Current thread:
- Re: CVSS is the worst compression algorithm ever, (continued)
- Re: CVSS is the worst compression algorithm ever Monroe, Bruce (Jan 08)
- Re: CVSS is the worst compression algorithm ever Wim Remes (Jan 10)
- Re: CVSS is the worst compression algorithm ever Adrian Sanabria (Jan 10)
- Re: CVSS is the worst compression algorithm ever Thierry Zoller (Jan 10)
- Re: CVSS is the worst compression algorithm ever Monroe, Bruce (Jan 10)
- Re: CVSS is the worst compression algorithm ever Adrian Sanabria (Jan 11)
- Re: CVSS is the worst compression algorithm ever Wim Remes (Jan 10)
- Re: CVSS is the worst compression algorithm ever Dennis Groves (Jan 10)
- Re: CVSS is the worst compression algorithm ever Monroe, Bruce (Jan 08)
- Re: CVSS is the worst compression algorithm ever Adrian Sanabria (Jan 10)
- Re: CVSS is the worst compression algorithm ever Adam Shostack (Jan 10)
- Re: CVSS is the worst compression algorithm ever Adrian Sanabria (Jan 11)
- Re: CVSS is the worst compression algorithm ever Adam Shostack (Jan 11)
- Re: CVSS is the worst compression algorithm ever Adrian Sanabria (Jan 11)
- Re: CVSS is the worst compression algorithm ever Nathaniel Ferguson (Jan 11)
- Re: CVSS is the worst compression algorithm ever Dave Aitel (Jan 10)
- Re: CVSS is the worst compression algorithm ever Eric Schultz (Jan 10)