Re: CVSS is the worst compression algorithm ever

["Nathaniel Ferguson" <jferguson () 126 com>]

> I try to explain - when you're hiring a pentester, you're paying for their opinion as they perform a point-in-time 
> assessment.

Well that's not entirely true, a significant percentage of work comes from vendors seeking to acquire or utilize 
another product or an institution going through some sort of audit wherein both cases the client is someone that 
doesn't really even want to be going through it and it's something being forced on them. Those are the instances I've 
encountered where the sort of negotiating down or into entire absence findings are or exist, which makes some amount of 
sense.

> clients attempting to negotiate down the severity of vulns on the report - was common regardless of the scoring 
> system used.

My experience has been its only an issue in the aforementioned instances. Generally clients that inquire externally are 
eager and willing and happy.

> we've used a low/medium/high or low/medium/high/critical scale

I've generally avoided the argument by just rating everything as medium unless there was some specific reason to not do 
so. A buffer overflow in a high-traffic code path would be high, a theoretical out of bounds read in a low risk portion 
would be low, anything in the middle would be in the middle.

> No score - just exploitable or not.

I'm not always certain what the fascination with exploitability in a review is, spot checking to some extent makes sense

So much emphasis is put on exploitability that the preference trends towards finding a vulnerability versus eliminating 
as many as possible that it frequently feels like marketing masquerading as security, but that is another matter. As an 
industry, security has come a long ways since the inception-- long gone are the strcpy(buf, argv[1]) days, but its sort 
of BS the way things are done and often a sincere disservice to the customer-- eventually some sort of racketeering 
charge is going to hit one of the consultancies and the entire industry will freak out as a result. So, the finding is 
that you can hack the lock with physical access, sort of like with access to the lock you could make a copy of the key? 
The argument is that this mechanism to break the entire line of DRM products in a generic manner isn't an issue even 
though solutions exist, but rather they should keep having you test every single one of these products broken in the 
same way?

This is more problematic in product reviews than network penetration testing and frequently slight modifications to the 
sales and project management functionality can have drastic impacts. In the initial stages, managing a clients 
expectations, telling them the things the team will realistically need, finding out what threshold for exploitability 
they desire and making sure its realistic given the time scale and resources and so forth helps with a lot of that. 
Having the team split into functionality that cross-checks one another's findings helps eliminate a lot of the errors 
from not having run-time functionality or similar, et cetera-- but no one really does any of that for some reason. It 
always amazed me to read on the reports what the methodology for the companies were given that no one would instruct or 
inform the employees of what it was, just this sort of paragraph sold to the clients that the persons intended to 
follow the methodology were the last to know about-- I don't think I've ever even seen a company engage in any sort of 
formalized methodology, its frequently sort of uncanning a room full of monkeys that do whatever they want or think 
where even small attempts at ensuring that people aren't covering the same portions of the product redundantly is rare.

As a consultant, most of the clients have generally been in disarray and instructing them on the best way to help you 
help them is sort of part of the process. I don't think I've ever had a product that had a suitable demonstration or 
test network setup for it, which often means guessing at how the product is deployed-- if a functioning version of the 
product is even available (often the sort of continuous development models ends up meaning the version you're looking 
at doesn't exist anywhere). I used to ask for debug builds as a standard rule of thumb, but I don't think I've ever 
received one and frequently you end up with things like the debugger locking up the entire machine and being entirely 
undebuggable making developing a PoC borderline impossible. I think I've only ever had one customer that really was 
prepared for a security review and formalized the process in a proper manner and they were a large software shop that 
could use their own security teams, but despite everyone having worked for them at some point, no one takes away the 
processes and formalizes they use to export them to the smaller shops trying to figure out how to implement things on 
their own, which ends up in this sort of loop of forced dependence where the clients interests are not being overly 
well served. Most of it seems to be a general disconnect between business oriented personnel and technical personnel, 
not often malfeasance on any part.

Again, that is a totally different subject however.

> they'll argue over the likelihood of it being exploited.

If someone wants to debate whether an issue is a bug or not that is one thing, if fixing the issue is substantial and 
requires large modifications akin to a design flaw then this becomes more applicable, otherwise its mostly mental 
masturbation. It seems to mostly be an artifact of where the industry started and a necessity to show necessity more so 
than add value. x = malloc(user*user2); memcpy(x, user3, user*user2-1) means just add a single if statement, not argue 
for an hour or spend a week trying to establish exploitability.

At 2019-01-11 13:23:24, "Adrian Sanabria" <adrian.sanabria () gmail com> wrote:

Everywhere I've ever pentested, we've used a low/medium/high or low/medium/high/critical scale - this is my first 
encounter with DREAD. What you describe though - clients attempting to negotiate down the severity of vulns on the 
report - was common regardless of the scoring system used. I don't see DREAD being unique in that respect.



Reflecting, it's probably what pushed me towards the binary system I ended up using. No score - just exploitable or 
not. Then, if it's someone that just refuses to have a critical finding on their report, they'll argue over the 
likelihood of it being exploited. Or over whether the finding was in scope. Then I consider firing that client.


I try to explain - when you're hiring a pentester, you're paying for their opinion as they perform a point-in-time 
assessment. The client is free to debate the findings and they don't have to agree with the consultant, but the client 
can't change/rewrite the report without destroying the integrity of the report.


--Adrian


On Thu, Jan 10, 2019 at 5:21 PM Adam Shostack <adam () shostack org> wrote:

Okay, I'll respond generally about DREAD.  The issue comes up when
people say "We'll treat a DREAD rating of >= 8 as critical."  Then
someone looks at your discoverability of 7, and says "hmm, if this
were a 6, then DREAD would be 7.9...can we change it?"  Lacking any
guidance on the difference, it's hard to say no.

Really, it's often "You're being unreasonable by making
discoverability a 7 here.  It's not that high!"

Adam



On Thu, Jan 10, 2019 at 04:38:30PM -0500, Adrian Sanabria wrote:
> I probably shouldn't have brought it up - I'm not involved much on the
> pentesting side. I know we've discussed replacing it, but finding little out
> there to replace it with.
>
> In my own work, I find most of my pentesting results come down to a binary
> value (hackable, not hackable) and some sense of likelihood of it getting
> exploited by a malicious party. Highs/mediums/lows all seem pointless when
> emulating the attacker perspective.
>
> Looking at DREAD, I honestly can't say I find anything fatally wrong with it.
> Perhaps it's because I've never known pentesting to be terribly consistent
> across tests or consultants in my career, so the bar is set pretty low in my
> mind?
>
> --Adrian
>
> On Thu, Jan 10, 2019 at 2:12 PM Adam Shostack <adam () shostack org> wrote:
>
>     On Wed, Jan 09, 2019 at 08:18:48AM -0500, Adrian Sanabria wrote:
>     > Our pentesters use DREAD, which I think most people have moved on from,
>     but at
>     > least the scoring is clear and consistent. 
>
>     I'm sorry, but I need to rant a little.
>
>     A decade back, I wrote a "DREAD is DEAD, please stop" blog post for
>     Microsoft.  If you are getting consistent scoring out of DREAD, you
>     are not using DREAD (as described in Writing Secure Code 1, which I
>     think is the first public description).
>
>     You are using some derivitive that adds tools to provide for
>     that consistency.  Those tools may be as simple as a set of examples
>     of each of the constiuents and what constitutes a 7 or a 3.
>
>     I care about this because I one of the biggest things that I see
>     making threat modeling hard is everyone calls their specific
>     collection of techniques 'lightweight threat modeling' or 'agile
>     threat modeling' and people trying to learn get confused because
>     there's 6 contradictory descriptions that have been labeled "agile
>     tm".  People writing down process so their engineers can do it
>     consistently get confused in the same way they'd get confused if we
>     all said "oh yeah. we're writing code, and you can assign variables
>     with either = or <=".  We name our languages, we version them. We need
>     to start doing the same for threat modeling constructs.
>
>     If you say "We're using DREADNOP 1.0" that's cool.  Alternately, maybe
>     you're using DREAD 1.0 in its raw form, in which case, how are you
>     getting consistent scores?
>
>     Adam
>
>
>     > In addition to CVE being wrong on critical details, I've found that most
>     of
>     > ExploitDB isn't exploits. Many are vulnerability checks and almost all
>     are
>     > incorrectly entered. PrivEsc will be labeled RCE and RCE will be labeled
>     DoS.
>     > It's all a mess. If I had the resources to burn it all down and start
>     from
>     > scratch, I'd do it. 
>     >
>     > --Adrian
>     >
>     > On Tue, Jan 8, 2019, 17:47 Nathaniel Ferguson <jferguson () 126 com wrote:
>     >
>     >     > They use a ton of big words in the paper to call CVSS out and give
>     it a
>     >     shellacking. Like most of you, we have extensive use of CVSS in our
>     >     consulting practice and I've seen this stuff first hand. CVSS is of
>     course
>     >     just a buggy compression algorithm for taking complex qualitative
>     data and
>     >     then putting it on a number line.
>     >
>     >
>     >     Over the years I've worked at a few different consultancies and at
>     least
>     >     originally basically no one used any sort of standardized metric, the
>     >     reports were generally humorous from a technical standpoint as the
>     numbers
>     >     were basically just made up and didn't adhere to even basic
>     statistics
>     >     methodologies-- we take the X and multiple it by Y and add the Z and
>     >     there's your score! Some even plotted them along cartoon looking
>     graphs and
>     >     plots and my suspicion was that they were really included to give
>     depth to
>     >     the reports and break up the monotony of text on text on text. Oddly,
>     I've
>     >     never even worked at a place that described the methodology as
>     outlined in
>     >     their reports to their employees leaving some question as to how a
>     >     methodology was to be implemented if only the client ever heard about
>     it.
>     >
>     >     In that sense, CVSS et al make some amount of sense, if nothing else
>     it
>     >     standardizes to a common metric and isn't what the sales guy or
>     operations
>     >     manager made up. Additionally, what a strange word-- shellacking,
>     there is
>     >     no 'k' in the word until its made into a present participle.
>     >
>     >     > The paper has three angles here:
>     >     > Qualitative mappings into quantitative numbers are a silly thing to
>     do,
>     >     like people trying to do "social science" by using SurveyMonkey.
>     >
>     >     Which is what most people are or were selling.
>     >
>     >     > It's fine to have a lossy compression algorithm that emphasizes
>     certain
>     >     aspects of the input signal over others, of course, but an additional
>     CERT/
>     >     CC critique is we have no reason to think CVSS does this in any
>     useful way.
>     >
>     >     Well there 's a missing line here, you can see it from the way that
>     >     client-side attacks perverted the concept of remote and so they made
>     them
>     >     remote also instead of adding the new line to the plot. Because of
>     stuff
>     >     like this. everything is remote now which limits its usefulness. This
>     >     doesn't even touch on the fact that most of the CVE database is
>     basically
>     >     wrong from submissions including very limited data, id est "memory
>     >     corruption results in a DoS".
>     >
>     >     Nathaniel
>     >
>     >     在 2019-01-09 00:14:00，"Dave Aitel" <dave.aitel () cyxtera com> 写道：
>     >
>     >       
>     >         I wanted to take a few minutes and do a quick highlight of a
>     paper from
>     >         CMU-CERT which I think most people have missed out on: https://
>     >         resources.sei.cmu.edu/asset_files/WhitePaper/
>     2018_019_001_538372.pdf
>     >
>     >         Towards Improving CVSS - resources.sei.cmu.edu
>     >         resources.sei.cmu.edu
>     >         SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY
>     >         REV-03.18.2016.0 Distribution Statement A: Approved for Public
>     Release;
>     >         Distribution Is Unlimited TOWARDS IMPROVING CVSS
>     >         It's almost as funny a read as their previous best work on how "
>     >         clientless HTTPS VPNs are insanely dumb what were you thinking
>     omg?"
>     >
>     >         They use a ton of big words in the paper to call CVSS out and
>     give it a
>     >         shellacking. Like most of you, we have extensive use of CVSS in
>     our
>     >         consulting practice and I've seen this stuff first hand. CVSS is
>     of
>     >         course just a buggy compression algorithm for taking complex
>     >         qualitative data and then putting it on a number line. The paper
>     has
>     >         three angles here: 
>     >          1. Qualitative mappings into quantitative numbers are a silly
>     thing to
>     >             do, like people trying to do "social science" by using
>     >             SurveyMonkey.
>     >          2. We're pretty sure that the compression algorithm is not, in
>     fact,
>     >             putting higher risk items as bigger numbers, which is the
>     whole
>     >             point of the thing.  
>     >          3. Nobody is applying this in any sort of consistent way (which
>     is
>     >             probably impossible) which is ALSO the whole point of the
>     thing.
>     >
>     >         It's fine to have a lossy compression algorithm that emphasizes
>     certain
>     >         aspects of the input signal over others, of course, but an
>     additional
>     >         CERT/CC critique is we have no reason to think CVSS does this in
>     any
>     >         useful way. 
>     >
>     >
>     >         There's definitely people in the CVSS process (who I will avoid
>     calling
>     >         out by name) who think ANY quantization is good. But read the
>     paper and
>     >         decide for yourself - because these are probably serious issues
>     that
>     >         are turning your entire risk org into a Garbage-In-Garbage-Out
>     org...
>     >
>     >
>     >         -dave
>     >
>     >
>     >
>     >
>     >
>     >      
>     >
>     >     _______________________________________________
>     >     Dailydave mailing list
>     >     Dailydave () lists immunityinc com
>     >     https://lists.immunityinc.com/mailman/listinfo/dailydave
>     >
>
>     > _______________________________________________
>     > Dailydave mailing list
>     > Dailydave () lists immunityinc com
>     > https://lists.immunityinc.com/mailman/listinfo/dailydave
>
>
>     --
>     Adam Shostack
>     President, Shostack & Associates
>     https://associates.shostack.org • +1 917 391 2168
>
>     Join my very quiet annnoucement list: https://adam.shostack.org/newthing

--
Adam Shostack
President, Shostack & Associates
https://associates.shostack.org • +1 917 391 2168

Join my very quiet annnoucement list: https://adam.shostack.org/newthing

_______________________________________________
Dailydave mailing list
Dailydave () lists immunityinc com
https://lists.immunityinc.com/mailman/listinfo/dailydave