Re: CVSS is the worst compression algorithm ever

[Adrian Sanabria <adrian.sanabria () gmail com>]

Everywhere I've ever pentested, we've used a low/medium/high or
low/medium/high/critical scale - this is my first encounter with DREAD.
What you describe though - clients attempting to negotiate down the
severity of vulns on the report - was common regardless of the scoring
system used. I don't see DREAD being unique in that respect.

Reflecting, it's probably what pushed me towards the binary system I ended
up using. No score - just exploitable or not. Then, if it's someone that
just refuses to have a critical finding on their report, they'll argue over
the likelihood of it being exploited. Or over whether the finding was in
scope. Then I consider firing that client.

I try to explain - when you're hiring a pentester, you're paying for their
opinion as they perform a point-in-time assessment. The client is free to
debate the findings and they don't have to agree with the consultant, but
the client can't change/rewrite the report without destroying the integrity
of the report.

--Adrian

On Thu, Jan 10, 2019 at 5:21 PM Adam Shostack <adam () shostack org> wrote:

> Okay, I'll respond generally about DREAD.  The issue comes up when
> people say "We'll treat a DREAD rating of >= 8 as critical."  Then
> someone looks at your discoverability of 7, and says "hmm, if this
> were a 6, then DREAD would be 7.9...can we change it?"  Lacking any
> guidance on the difference, it's hard to say no.
>
> Really, it's often "You're being unreasonable by making
> discoverability a 7 here.  It's not that high!"
>
> Adam
>
>
>
> On Thu, Jan 10, 2019 at 04:38:30PM -0500, Adrian Sanabria wrote:
> > I probably shouldn't have brought it up - I'm not involved much on the
> > pentesting side. I know we've discussed replacing it, but finding little
> out
> > there to replace it with.
> >
> > In my own work, I find most of my pentesting results come down to a
> binary
> > value (hackable, not hackable) and some sense of likelihood of it getting
> > exploited by a malicious party. Highs/mediums/lows all seem pointless
> when
> > emulating the attacker perspective.
> >
> > Looking at DREAD, I honestly can't say I find anything fatally wrong
> with it.
> > Perhaps it's because I've never known pentesting to be terribly
> consistent
> > across tests or consultants in my career, so the bar is set pretty low
> in my
> > mind?
> >
> > --Adrian
> >
> > On Thu, Jan 10, 2019 at 2:12 PM Adam Shostack <adam () shostack org> wrote:
> >
> >     On Wed, Jan 09, 2019 at 08:18:48AM -0500, Adrian Sanabria wrote:
> >     > Our pentesters use DREAD, which I think most people have moved on
> from,
> >     but at
> >     > least the scoring is clear and consistent.
> >
> >     I'm sorry, but I need to rant a little.
> >
> >     A decade back, I wrote a "DREAD is DEAD, please stop" blog post for
> >     Microsoft.  If you are getting consistent scoring out of DREAD, you
> >     are not using DREAD (as described in Writing Secure Code 1, which I
> >     think is the first public description).
> >
> >     You are using some derivitive that adds tools to provide for
> >     that consistency.  Those tools may be as simple as a set of examples
> >     of each of the constiuents and what constitutes a 7 or a 3.
> >
> >     I care about this because I one of the biggest things that I see
> >     making threat modeling hard is everyone calls their specific
> >     collection of techniques 'lightweight threat modeling' or 'agile
> >     threat modeling' and people trying to learn get confused because
> >     there's 6 contradictory descriptions that have been labeled "agile
> >     tm".  People writing down process so their engineers can do it
> >     consistently get confused in the same way they'd get confused if we
> >     all said "oh yeah. we're writing code, and you can assign variables
> >     with either = or <=".  We name our languages, we version them. We
> need
> >     to start doing the same for threat modeling constructs.
> >
> >     If you say "We're using DREADNOP 1.0" that's cool.  Alternately,
> maybe
> >     you're using DREAD 1.0 in its raw form, in which case, how are you
> >     getting consistent scores?
> >
> >     Adam
> >
> >
> >     > In addition to CVE being wrong on critical details, I've found
> that most
> >     of
> >     > ExploitDB isn't exploits. Many are vulnerability checks and almost
> all
> >     are
> >     > incorrectly entered. PrivEsc will be labeled RCE and RCE will be
> labeled
> >     DoS.
> >     > It's all a mess. If I had the resources to burn it all down and
> start
> >     from
> >     > scratch, I'd do it.
> >     >
> >     > --Adrian
> >     >
> >     > On Tue, Jan 8, 2019, 17:47 Nathaniel Ferguson <jferguson () 126 com
> wrote:
> >     >
> >     >     > They use a ton of big words in the paper to call CVSS out
> and give
> >     it a
> >     >     shellacking. Like most of you, we have extensive use of CVSS
> in our
> >     >     consulting practice and I've seen this stuff first hand. CVSS
> is of
> >     course
> >     >     just a buggy compression algorithm for taking complex
> qualitative
> >     data and
> >     >     then putting it on a number line.
> >     >
> >     >
> >     >     Over the years I've worked at a few different consultancies
> and at
> >     least
> >     >     originally basically no one used any sort of standardized
> metric, the
> >     >     reports were generally humorous from a technical standpoint as
> the
> >     numbers
> >     >     were basically just made up and didn't adhere to even basic
> >     statistics
> >     >     methodologies-- we take the X and multiple it by Y and add the
> Z and
> >     >     there's your score! Some even plotted them along cartoon
> looking
> >     graphs and
> >     >     plots and my suspicion was that they were really included to
> give
> >     depth to
> >     >     the reports and break up the monotony of text on text on text.
> Oddly,
> >     I've
> >     >     never even worked at a place that described the methodology as
> >     outlined in
> >     >     their reports to their employees leaving some question as to
> how a
> >     >     methodology was to be implemented if only the client ever
> heard about
> >     it.
> >     >
> >     >     In that sense, CVSS et al make some amount of sense, if
> nothing else
> >     it
> >     >     standardizes to a common metric and isn't what the sales guy or
> >     operations
> >     >     manager made up. Additionally, what a strange word--
> shellacking,
> >     there is
> >     >     no 'k' in the word until its made into a present participle.
> >     >
> >     >     > The paper has three angles here:
> >     >     > Qualitative mappings into quantitative numbers are a silly
> thing to
> >     do,
> >     >     like people trying to do "social science" by using
> SurveyMonkey.
> >     >
> >     >     Which is what most people are or were selling.
> >     >
> >     >     > It's fine to have a lossy compression algorithm that
> emphasizes
> >     certain
> >     >     aspects of the input signal over others, of course, but an
> additional
> >     CERT/
> >     >     CC critique is we have no reason to think CVSS does this in any
> >     useful way.
> >     >
> >     >     Well there 's a missing line here, you can see it from the way
> that
> >     >     client-side attacks perverted the concept of remote and so
> they made
> >     them
> >     >     remote also instead of adding the new line to the plot.
> Because of
> >     stuff
> >     >     like this. everything is remote now which limits its
> usefulness. This
> >     >     doesn't even touch on the fact that most of the CVE database is
> >     basically
> >     >     wrong from submissions including very limited data, id est
> "memory
> >     >     corruption results in a DoS".
> >     >
> >     >     Nathaniel
> >     >
> >     >     在 2019-01-09 00:14:00，"Dave Aitel" <dave.aitel () cyxtera com>
> 写道：
> >     >
> >     >
> >     >         I wanted to take a few minutes and do a quick highlight of
> a
> >     paper from
> >     >         CMU-CERT which I think most people have missed out
> on: https://
> >     >         resources.sei.cmu.edu/asset_files/WhitePaper/
> >     2018_019_001_538372.pdf
> >     >
> >     >         Towards Improving CVSS - resources.sei.cmu.edu
> >     >         resources.sei.cmu.edu
> >     >         SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY
> >     >         REV-03.18.2016.0 Distribution Statement A: Approved for
> Public
> >     Release;
> >     >         Distribution Is Unlimited TOWARDS IMPROVING CVSS
> >     >         It's almost as funny a read as their previous best work on
> how "
> >     >         clientless HTTPS VPNs are insanely dumb what were you
> thinking
> >     omg?"
> >     >
> >     >         They use a ton of big words in the paper to call CVSS out
> and
> >     give it a
> >     >         shellacking. Like most of you, we have extensive use of
> CVSS in
> >     our
> >     >         consulting practice and I've seen this stuff first hand.
> CVSS is
> >     of
> >     >         course just a buggy compression algorithm for taking
> complex
> >     >         qualitative data and then putting it on a number line. The
> paper
> >     has
> >     >         three angles here:
> >     >          1. Qualitative mappings into quantitative numbers are a
> silly
> >     thing to
> >     >             do, like people trying to do "social science" by using
> >     >             SurveyMonkey.
> >     >          2. We're pretty sure that the compression algorithm is
> not, in
> >     fact,
> >     >             putting higher risk items as bigger numbers, which is
> the
> >     whole
> >     >             point of the thing.
> >     >          3. Nobody is applying this in any sort of consistent way
> (which
> >     is
> >     >             probably impossible) which is ALSO the whole point of
> the
> >     thing.
> >     >
> >     >         It's fine to have a lossy compression algorithm that
> emphasizes
> >     certain
> >     >         aspects of the input signal over others, of course, but an
> >     additional
> >     >         CERT/CC critique is we have no reason to think CVSS does
> this in
> >     any
> >     >         useful way.
> >     >
> >     >
> >     >         There's definitely people in the CVSS process (who I will
> avoid
> >     calling
> >     >         out by name) who think ANY quantization is good. But read
> the
> >     paper and
> >     >         decide for yourself - because these are probably serious
> issues
> >     that
> >     >         are turning your entire risk org into a
> Garbage-In-Garbage-Out
> >     org...
> >     >
> >     >
> >     >         -dave
> >     >
> >     >
> >     >
> >     >
> >     >
> >     >
> >     >
> >     >     _______________________________________________
> >     >     Dailydave mailing list
> >     >     Dailydave () lists immunityinc com
> >     >     https://lists.immunityinc.com/mailman/listinfo/dailydave
> >     >
> >
> >     > _______________________________________________
> >     > Dailydave mailing list
> >     > Dailydave () lists immunityinc com
> >     > https://lists.immunityinc.com/mailman/listinfo/dailydave
> >
> >
> >     --
> >     Adam Shostack
> >     President, Shostack & Associates
> >     https://associates.shostack.org • +1 917 391 2168
> >
> >     Join my very quiet annnoucement list:
> https://adam.shostack.org/newthing
> >
>
> --
> Adam Shostack
> President, Shostack & Associates
> https://associates.shostack.org • +1 917 391 2168
>
> Join my very quiet annnoucement list: https://adam.shostack.org/newthing
_______________________________________________
Dailydave mailing list
Dailydave () lists immunityinc com
https://lists.immunityinc.com/mailman/listinfo/dailydave