Re: Today's thought

[Matt Hargett <matt () use net>]

Dave Aitel wrote:
> Just because you've visited a code block does not mean you've
> triggered a potential bug in that code block. The most basic case of
> this is bugs that require global variables to be set to certain values
> to trigger bugs, or pointers that need to be moved to certain places.
>
> There's a lot of binary and source analysis technologies coming out -
> people need to understand that the false-positive problem has still
> not really been solved by a machine. When you don't have false
> positives in binary analysis, what that means is that you have a huge
> number of false negatives.
>
> People are probably solving this with massive iterations over the code
> space. I'm not sure this is the correct answer.
>
> IMHO.

Agreed, code coverage is not the end-all-be-all metric, but is severely 
underestimated by most people in its' usefulness. People new to QA 
usually think it is (as do pointy-haired managers), and you can spend a 
lot of time getting coverage that doesn't mean anything. (We recently 
hit a wall of 85% code coverage in our unit tests because we weren't 
testing exception handlers thoroughly; we could've spent another 2 weeks 
getting another 10%, but after a point it doesn't deliver much value to 
anyone since those details aren't part of the API contract.)

Static analysis, by itself and at its' best, is less than a 70% solution 
IMO. That being said, current approaches to runtime analysis also have 
some insurmountable that make them less than ideal. It was only when I 
took a step back and started analysis from use cases that I came up with 
something interesting. Hopefully I'll be talking about it at BlackHat 
Vegas, if they accept my talk.

There are a lot of companies getting funding right now that do source 
code analysis, varying from fancy regexp matching on gcc's preprocessor 
output to real AST generation and inspection. No interfunction value 
tracking (similar to code coverage in that people underestimate its' 
usefulness in these scenarios) yet, as far as I know, though.

BugScan's buffer iteration detection still has about a 1 in 10 false 
positive rate averaged over 40+ binaries. I know what I need to do to 
get it even better, but I don't think it will ever be "perfect". The 
question really is, does it provide enough value to the target user? So 
far, the answer is yes. Exploit developers, in their last mile, want 
things like backtracing to user input (which is in BugScan but doesn't 
works in very limited scenarios -- I did not write that code) and 
tracking buffer sizes to find OB1 bugs and the like. It turns out that 
defensive customers want those same features, so they'll make it in 
there eventually. The point is that the value just increases -- these 
things aren't necessary to provide the value in the first place. The 
code as it stands has found exploitable heap overflows in the PGP Key 
Server, Trillian, and several other things (vendors all contacted). I 
wish I had time to write advisories on these, but I just don't. Maybe my 
priorities are screwed up. I'm curious what you think, Dave.

Another thing is to determine what the ultimate goal is. Are you trying 
to find 0-day automatically with no human intervention? That isn't very 
useful, in my opinion. You want to have that available as an option, but 
the more desirable option is to optimize the experience of the reverse 
engineer, should they want to expose themself to the details. When I 
wrote up the use cases while at Sundance this year, I came up with the 
phease "Integrated Reversing Environment" (inspired by Dildog's original 
UnDeveloper Studio, I think). When there is real integration between all 
the tools like VS.NET provides (Visio for UML and code generation, XML 
data modelling and code generation, etc), and that is optimized for the 
revengineer (who is not necessarily an exploit developer), that is when 
I think static analysis will be at its' most useful -- but only if 
runtime analysis can be integrated in a meaningful way. The current 
paradigms for displaying these informations doesn't allow that to be 
possible, I don't think. That is project Pensieve, which will hopefully 
come out eventually.

I think you heard me describing the Pensieve to someone at BlackHat 
Amsterdam last year, Dave.

gah, that's a lot of text. Sorry for the ramble.
_______________________________________________
Dailydave mailing list
Dailydave () lists immunitysec com
http://www.immunitysec.com/mailman/listinfo/dailydave