Re: What is the state of vulnerability research?

[Etaoin Shrdlu <shrdlu () deaddrop org>]

Steven M. Christey wrote:

> [sorry for the cross-post, but I hold high hopes for a good
> signal-to-noise ratio on dailydave compared to the other lists ;-)]


Please note that I'm replying only on Daily Dave. I am happy that you 
did *not* cross post, but rather (in the vernacular of Usenet) used 
multi-posting instead. This list (DD) tends to have a higher tone than 
some others, and is not restrictive or slow in response as some we can 
name. I have no real interest in having the current crop of ankle biters 
providing commentary to what I hope are thoughtful responses to your 
questions.

MindsX wrote:

> Why do I think Mitre should be coming out with answers, not questions?

security curmudgeon wrote:

>         
> On most days I would agree with you, and come up with a reply along these 
> lines =) However, since I am a cynic AND involved in the same 'scene' and 
> core interests Christey is, I think I understand this fairly well.

Considering the turns and twists that we've all seen in the industry, I 
would say that Christey's standards have held up rather well. I can 
understand his interest in soliciting the views of others. I have only 
recently retired from a world that seems cloistered and inbred from 
outside examination, and we have all experienced the narrow views of our 
compatriots, when encouraging them to look outward.

> This is a series of open questions to people who consider themselves
> to be vulnerability researchers. Hopefully this will open a number of
> fruitful public discussions.


You would have thought that you'd be inundated with answers. I've 
examined the larger picture, myself, and have engaged a discussion with 
various groups on the subject. It seems that there is still a basic 
split in the picture.

> 1) What is the state of vulnerability research?


We should first examine what is meant by that topic. Vulnerability 
research has come to imply that there is an expectation of a formal (or 
otherwise) release of the results of such research. It seems that it is 
unusual for someone to experiment in the area of vulnerabilities, and 
yet not publish. I note that Forno's survey predicates the role of 
researcher as one who publishes, and I see that your questions expect 
the same. This leaves me at a loss. I have been involved in research of 
various types and kinds of vulnerabilities since 1980, or thereabouts. 
With extremely rare exceptions, I have never notified a vendor. When I 
have done so, it has been anonymously, either through such things as 
mixmaster, or via hushmail. I have *never* published any findings, and 
that will probably continue to be the case.

I have shared certain suggestions with various people, sometimes 
anonymously, sometimes not. If I feel that it is an item not being 
exploited, I may just keep it to myself. If I see signs (via what are 
now termed honeynets) that there are exploits in the wild, then I may 
make suggestions to encourage research in the area. Sometimes those 
suggestions are made in large groups of people (my favorite place for 
this is defcon), sometimes it's in private conversation. This behavior 
may change, now that I am not concerned about security clearances and 
intellectual property agreements.

Still, I imagine that I am not the only researcher thus engaged in 
research, where results are simply noted (for further investigation, or 
for remediation of local products).

There is also the question of what vulnerability research is. Do we 
consider every moronic cross site scripting event noted to be a result 
of vulnerability research? This desperate search for notoriety is 
amusing, but I question the usefulness of it. Certainly, for those three 
people that use a particular application, it may be significant, but I 
confess I do not think it falls into the class of research. Marcus Ranum 
recently posted a reply in an interesting thread on Bugtraq (entitled 
"Vulnerabilities in new laws on computer hacking"), which I think bears 
repeating. He is speaking here of pentesters, but it holds for the class 
in general.

"I would say that most pentesters are failed security analysts who do 
not understand engineering discipline and have chosen to engage in the 
war of band-aids instead of learning how to build correct systems. And 
then there are the pentesters who really are cybertrespassers at heart, 
who have found a financial and moral justification for doing something 
for money that they'd otherwise do anyhow, for free, in the wee hours of 
the night."

He is correct. I have been as guilty as most of amusing myself by 
implying that breaking into a system and leaving no trace was acceptable 
in the old days. It's no more acceptable than to walk through your 
neighbor's house in the dead of night, just because he left the window 
open. In the same way, finding vulnerabilities in various classes of 
software, or in TCP/IP, still require a standard of ethical behavior 
that states that we must hold ourselves above even the appearance of 
doing evil.

> 2) What have researchers accomplished so far?


It depends on what you consider an accomplishment. The full disclosure 
of various vulnerabilities is significant, but far too many 
"researchers" are using the mantle of full disclosure to rush every 
trifling buffer overflow into print, in order to see their name in 
lights. Full disclosure has done some important things. Microsoft, 
Oracle, Sun Microsystems, Apple, and many others are much more likely to 
move quickly against security and usability problems than in past. This 
is a direct result of the embarrassments suffered on places like Bugtraq 
(the Full Disclosure mailing list was not yet needed in those days). 
Many vendors are trying to reverse engineer security into their 
products. Some even have rewritten software, with mixed results. 
Vanishingly few are following good security engineering practices (or 
even good engineering practices, let alone mixing good sense about 
security into it). Researchers have provided the impetus for this, but 
they've also laid the groundwork to encourage every juvenile delinquent 
on the internet to believe that announcing the buffer overflow of the 
week, will somehow bring them fame, fortune, and a social life.

> 3) What are the greatest challenges that researchers face?


DMCA and its cousins threatens to throw a chill on the industry. The 
infusion of money into the field has also blinded many to the ethics 
that we should all hold to. We must all have a certain ambivalence when 
vendors offer bounties for new exploits, yet we know also that the 
criminal element does the same. Do we secure the boundaries, concerning 
ourselves with each and every scripting error that exposes a web page to 
inappropriate release of information? Do we concern ourselves with the 
basics of security, publishing white papers on secure programming? How 
do we assist our fellows in securing their applications, when examples 
abound in products that should be held to a higher standard?

When is it appropriate to keep quiet about a vulnerability? If a problem 
is exposed for which there is no readily apparent solution, what is 
accomplished? On the one hand, perhaps a solution is found (either by 
the vendor, or by other researchers). On the other hand, if it is not, 
then what is accomplished by exposing it?

> 4) What, if anything, could researchers accomplish collectively that
> they have not been able to accomplish as individuals?


It might be useful to have an accrediting body, like the CISSP (I know, 
I know, but it still has its purpose). Perhaps a code of ethics might be 
appropriate. For an example, please see:

http://www.sage.org/ethics.mm

It might be nice to espouse such a code. It would certainly say that you 
belong to a group that are concerned with a higher good, and not just 
the excitement of the chase.

> 5) Should the ultimate goal of research be to improve computer
> security overall?


No. The ultimate goal of research must always be pure knowledge, else it 
is not research. It would be nice to sign up to the feel-good notion 
that what we do improves the security stance of the computing world, and 
of the internet. Certainly, that is an effect. It simply cannot be 
viewed as a goal, merely as a positive by-product. In the same way, 
honeypots and honeynets discover anomalous attempts that may predict 
attacks from previously unexpected quarters, but the goal of such a 
device should not be simply to discover such attacks. Rather that goal 
should be to further the investigation of each class of attacks, an 
active vulnerability research, as it were.

> 6) What is an "elite" researcher? Who are the elite researchers?


Now there's an overused word. I prefer to believe that the questions is 
thus:

What constitutes a vulnerability researcher?

The answer lies in the steadfast pursuit of knowledge for its own sake. 
Each person brings certain skills to bear; this one has great skill with 
disassemblers; that one recognizes anomalies in the packet stream of a 
networked application. The challenge of realizing that there are 
weaknesses, either in a protocol, or an application, and that the light 
of examination and testing may expose it, satisfies in a way that little 
else compares to.

> 7) Who are the researchers who do not get as much recognition as they
> deserve?


Researchers who need recognition are around us by the thousands. The 
satisfaction of knowledge gained should be enough. There are far too 
many "researchers" shouting at the top of their lungs in the 
marketplace. Any benefit to doing so has long passed.

> P.S. If you're further interested in letting your voice be heard,
> check out Richard Forno's disclosure survey at
> http://www.infowarrior.org/survey.html


Been there.

--
"It is necessary to the happiness of man, that he be mentally faithful
to himself. Infidelity does not consist in believing, or in
disbelieving, it consists in professing to believe what he does not
believe." Thomas Paine