Re: Automatic Exploitation Paper Peer Review

[Sean Heelan <sean () immunityinc com>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I think 'lets create a list of real world problems for academics to
consider' is missing the point somewhat. The problem here isn't that AEG
isn't a worthy or difficult problem, it is. The problem is that in order
to work realistically on AEG (and follow up with claims of changing
threat models etc) you need a good working knowledge of real world
exploit writing. The general impression I get from some of the research
groups working on the problem is they are unwilling to invest the time
required to gain this knowledge. (To kick a dead horse some more, one
should feel free to play around in a sandbox of vulnerabilities and
protection mechanisms from the past decade. It's obviously necessary for
tool development and trying out new ideas. The problem starts when you
forget half-way through that you're playing within a sandbox and pretend
it's the real world in your paper).

Instead of a list of problems to attack how about list of real world
tasks that the authors should be able to complete manually before
deciding to automate the process? I would have thought this was a pretty
sensible thing that most people would do but apparently not. Trying to
automate a process that you only have a vague idea of doesn't sound like
something that is every going to go too well.

Its very easy to run off down the path of an under-explored research
problem (I should know, I've done it :P), it's a lot less glamorous in
academia to spend weeks/months sitting in front of a debugger in order
to figure out the subtleties of the problem you are actually addressing.
This can hardly be considered an excessive request if one is working at
a well funded research group at a respected university though. Without
putting in this effort then the research output and paper quality have a
ceiling in terms of real-world applicability that will never be broken
through and the disparity between claims and facts will continue to
induce ong-winded and 'venomous' (chuckle) blog posts.

(Btw, at some point during the discussion a few people began to assume I
was criticising all of academia. This isn't the case. I was pretty
specific in my initial blog post (http://bit.ly/ikvR0y) where my issues
lay and they are with small proportion of the overall research output of
academia and industry. There is no 'us vs them' here. I would expect
anyone writing a paper to at least have a cursory understanding of
everything they discuss. Furthermore, I wasn't criticising the people
cited in the paper when I suggested less nepotism in citations would be
useful. I was suggesting that the papers authors perhaps read something
like Phrack, or Uninformed a little more extensively than 'Smashing the
Stack for Fun and Profit'.

My general opinion is that academia is both necessary and useful. That's
partially why I wrote the blog post to begin with - the paper is a
perfect example of the stereotype many have of academics as people with
their heads in the clouds dictating to those with their feet on the
ground. I know this isn't true for many so it's annoying when someone
comes along and proves it correct.

It's also worth mentioning that CMU's response "If Mr. Heelan feels
there are real scientific issues to discuss, he is welcome to call or
visit us at CMU to discuss them." conveniently ignores the fact that I
did send them an email outlining (yet again) both my issues with the
paper and some technical issues. I received one reply requesting a phone
call instead of email, which I declined as real-time conversations on
technical matters tend to miss a lot IMO, and then never heard anything
back. No feedback on their heap claims, nothing on their stack fix-ups,
nothing on their plans to scale to modern bugs/exploits and no response
to any of the valid complaints raised here. The only point of their
response.html [1] seems to be to foster the image that my complaints
stem from an anti-academia sentiment instead of engaging on the issues
raised here and elsewhere. Hardly the most common way for a research
group to deal with questions and comments from a pretty sizeable
proportion of their target audience.)

In hopeful expectation of a productive discussion (or failing that a
link or two to some funny cat videos),

Sean

[1] http://security.ece.cmu.edu/aeg/response.html

On 12/15/2010 08:54 AM, Miles Fidelman wrote:
> Anton Chuvakin wrote:
> > > I would love to see a resource for real-world problems that the academic
> > > community could consider... or even a resource for other up-and-coming
> > > researchers to examine at for ideas.  Such a site might not be relevant
> > > enough for PhD thesis work (which thrives on originality as I understand
> > > it?)
> > >      
> > Well, if it is created by the industry, the academics will ignore it.
> > And if created by academics, well, see discussion in this thread.
> >    
> Call me cynical but....
>
> If it has serious commercial potential, academics may be doing the 
> research, but saving the results for their side/spinout companies.
>
> The really interesting research (or least the well funded-research) gets 
> funded by DoD, with classified results, and never gets published.
>
> And folks who have serious countermeasures to large spambot networks 
> might just not want their names visible to the unsavory characters who 
> run large spambot networks.
>
> Now a list of relevant problems to research would be interesting, but I 
> expect there will be little feedback as to which problems people end up 
> taking on.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJNCkYoAAoJEMW6jFWLazyQbSMH/3azWDftzsCwVs3H3xvO8YW9
OII+v+fa20Jpkqh/KtSx9g4AjvootsxahTXv5e0pqqOIsRwQkP+eemC9xcDs/Kk/
BhGnIyvz54tANy2/TgKQZwLTPvkbICfbtyP7gQCr9rKk9DJaC7SyEcKjBDdaDEGF
jBFXFufjQZqpcF8kYOE7c5sLqYp2Lsfy/Kzroa4lKeoQFDyp5MjMTWzLqzULcRFl
zfdt8jNbZR3iAGYJdzbhPSFRsfseI69UOKsXLuZwGJUvNThDkyOpvlguqjkYwJ8J
THQ6ULvNuPLiSLbFLYKPI2KDYMWgjF1DXjswjmfSN8/Zv9RF10K5+n0hBlfp3r8=
=Wfg6
-----END PGP SIGNATURE-----
_______________________________________________
Dailydave mailing list
Dailydave () lists immunityinc com
https://lists.immunityinc.com/mailman/listinfo/dailydave