Re: What is the state of vulnerability research?

[security curmudgeon <jericho () attrition org>]

On most days I would agree with you, and come up with a reply along these 
lines =) However, since I am a cynic AND involved in the same 'scene' and 
core interests Christey is, I think I understand this fairly well.

: MITRE's Centers of Technical Excellence

: Our Centers of Technical Excellence are centralized areas in which we 
: build knowledge of key technologies. They act as a resource for the 

: And they bring back knowledge to the center about how technology is 
: being used in the field and what the customers might need next.

: Why do I think Mitre should be coming out with answers, not questions?

Vulnerability research is straight forward. There isn't a lot of black 
magic and secret arts when it comes to finding vulnerabilities. For the 
most part, 99% of vulnerabilites are very well documented (even if the 
'researcher' doesn't document it), easy to understand by others in the 
field, and leave little to imagination. It has been years since we've seen 
a truly new class of vulnerability surface. If I post details of an 
overflow of *any kind* to this list, there are a hundred folks that can 
digest what I post in seconds, then go to town on me for not going into 
details, not looking at VectorX, FunctionY or Z.c =) 

The other side of vulnerability disclosure is the human element. The 
sociology and mindset behind what we do, and why we do it. This is the 
angle that has interested me for years, and the type of book I will grab 
before any 'technical' (generous term usually) security/hacking book. Not 
only are there dozens of questions that can be asked of the researcher 
about his mindset and ethical views, there are countless other people 
involved in the process. Does the researcher have partners? Is he an 
employee of a security company? What vendor is he dealing with? Which 
vendor is it? How many people is he dealing with on the vendor side?

All of that will factor in to how a disclosure plays out, along with a 
thousand other variables. Trying to understand that is not easy (if at all 
possible). If Christey or MITRE said "we understand the state of vuln 
research, here it is", you and I would both be flaming him until we pass 
out at the keyboard. The fact that he *knows* he doesn't know is actually 
an impressive quality. The fact he will actually *admit* that he doesn't 
know, is staggering. The fact he will ask the community their thoughts 
(hey, peer review?!) and factor it into his own research and answers.. 
wow! If you think Mitre should be coming out with answers, without asking 
these questions, then I think you should step back and think about the 
state of vulnerability research.

I don't say this as an insult or a derogatory comment at all. A couple 
years ago I would have posted the same thing, without realizing just how 
complex of an issue this really is. Since I started working with OSVDB.org 
and running such a database, my appreciation and understanding of the 
complexity of vulnerability research and disclosure has skyrocketed. That 
skyrocket has now put me somewhere between "i don't have a fuckin clue" 
and "i think i have an inkling of understanding here". I say this because 
behind the scenes, after you read the bugtraq or full-disclosure post 
about an issue, the real research frequently takes place. If you or anyone 
else knew just how many mistakes, oversights and shortcomings occured in 
the average disclosure, you may be surprised. If Christey or I had a 
nickel for everytime we mailed the other about a flaw or oversight in the 
other's database, we'd both be surrounded by a dozen hookers feeding us 
tropical fruits and exotic drugs on our own private island.

The current state of vulnerability research SUCKS ASS. Yes, quote me on 
that. I say that in the context of the overall research we see. I say that 
based on a cynical and jaded perspective I have after spending countless 
hours clarifying, correcting and further researching issues that have 
already been 'researched'. I say this after dealing with dozens of vendors 
who tell me that *I* am wrong because OSVDB has information culled from 
other sources, and that say *I* am a moron for thinking X is vulnerable to 
Y and Z is the impact. Doesn't matter that half the time (or more) the 
vendor is wrong and sending me a snap judgement mail defending their 
product. Doesn't matter that sometimes they are right, but in figuring 
that out I find several other vulnerabilities in their product. Doesn't 
matter that CVE, OSVDB, SecurityTracker and others spend a few days to 
finally figure out and resolve discrepancies in the most boring and 
simples of disclosures days before. Doesn't matter that one such vendor 
had an office 1.4 miles from me and the only thing that stopped me from 
pissing on their front door was 18 degree weather.

Think about it.. why would ANY 'vulnerability report' require that much 
followup and analysis? One of the many things on my 'to blog about' list 
is a subset of Christey's question. Why do people think they are actually 
doing 'vulnerability research' by pasting a ' into a field and getting an 
SQL error? Hello.. that doesn't mean you can inject SQL queries! Pasting 
the standard <script> tags into a field and getting a popup doesn't mean 
you have 'researched' an XSS vulnerability. Hell, even a lot of 
researchers we all know and respect fairly well continually post "X is 
vulnerable to an Overflow" without clarifying if it is an overflow that 
allows code execution, or an overflow that is ONLY good for crashing an 
application! If a researcher makes no claims either way, and makes it 
clear that they did not fully test it, fine! It's all about qualifying 
your research and explaining your process. I know you have as little free 
time as I do, I know you can't fully test an application. As long as you 
make that clear, then the rest of the world can take your research for 
what it is and expand on it, or move forward knowing it hasn't been fully 
tested.

Do you ever wonder about the cut and paste kiddies discovering XSS and SQL 
injection vulnerabilities? Why they find a vuln in a product, then a week 
later another person finds a different vuln in the same version of the 
same product? How could the original person miss it? Easy, because they 
didn't test it. Why didn't they? They wanted to find one vulnerability, 
ANY vulnerability in that product, publish it and move on. It's about the 
same mindset and skillset as the web defacers who would tag 
somerandom-123-456.webhost.com domain with "0mg d4v3 w4s h343" and move 
on. They didn't own that machine any more than these vulnerability 
researchers 'audited' a product.

So.. next time you are reading Bugtraq or Full-Disclosure, stop and 
examine the post for a few seconds. Did they include all the vital info to 
make the issue clear? Did they forget to include the exact vendor name 
and/or vendor URL (so many don't)? Did they say every version prior to the 
current was vulnerable (even though the vulnerable function was added ONE 
revision back)? Did they produce one error condition in one script and 
nothing else? Did they forget to tell you that it only occurs when 
register_globals is set to 'on', something that is NOT the default, and 
hasn't been that way since Aug 29, 2000? Did they fail to mention that the 
same issue was disclosed a year before, and that the vendor never fixed 
it?

The simple truth is, formal and public vulnerability research is 
primitive. While there are a few (percentage wise) researchers who know 
the score and do a good job with their work, a majority of researchers are 
new to the field, undisciplined, or lacking some fundamental knowledge 
needed to properly research and test software vulnerabilities. The 
databases that track these disclosures are equally primitive. The only 
reason they have evolved a few years beyond the researchers, is that they 
are *aware* of just how lacking it all is, and want to see it change for 
the better.


jericho