Re: Rcov - interactive code coverage for fuzzers

[Kowsik <kowsik () gmail com>]

This thread was not about vendors, nor was it about how long they've
been around.

A few points, mostly to clarify my last post:

- 100% code coverage != zero vulnerabilities
- Rcov is there so it can help you spot exceptional conditions easily
- It shows you code that's reachable through the attack surface, interactively
- Fuzzers don't automagically become better because of code coverage
- Or vice versa

Fuzzing, robustness testing, negative testing, unit tests,
non-functional testing; same difference; potayto, potahto; Effective
ways to build secure, robust systems.

Regards,

K.

On 3/27/07, Ari Takanen <ari.takanen () codenomicon com> wrote:
> Hello All,
>
> I think code coverage is a very interesting topic when combined with
> Fuzzing! A few comments below:
>
> On Mon, Mar 26, 2007 at 10:29:15PM -0400, dailydave-request () lists immunitysec com wrote:
> > From: Kowsik <kowsik () musecurity com>
> >
> > - Uses gcov's notes/data files to get at blocks and function summaries
>
> There is a very interesting Masters' Thesis on this topic made by one
> of our guys for Codenomicon (and University of Oulu) written in
> 2004. Basically as anyone with common sense can tell you, the research
> indicated that in some areas fuzzing has better code coverage than any
> other test technique, but also that code coverage gives no indication
> of the quality of fuzzing. I will see if we can publish that and I
> will send you the link on it later. Please let me know if you want a
> private copy.
>
> > From: Gadi Evron <ge () linuxbox org>
> >
> > We have three levels or layers (depends on approach):
> > 1. Building better fuzzers (which you cover).
> > 2. Helping the fuzzing process, fuzzing better.
> > 3. Making the process of finding the actual vulnerability once an
> >    indication is found (a successful test case, or as they say in QA, a
> >    passing one) easier.
>
> As I mentioned, code coverage indicates nothing about the quality of
> fuzzing. Well it can indicate that your fuzzing is very bad though if
> you do not get any coverage. But more importantly, I think code
> coverage is an excellent metric for "attack surface" i.e. when you
> fuzz you will discover the lines of code that are security critical. A
> study we did in 2004 definitely indicated that traditional QA did not
> touch large part of the code, whereas fuzzing (using Codenomicon
> tools) increased the amount of code touched during testing. I would be
> interested in seeing how this can be used to improve fuzzers, but more
> importantly this word should be sent to our friends in the white-box
> testing who have the challenges in auditing millions of lines of
> code. Any metric of what code can be exploited through different
> interfaces can prioritize the focus of those analysist tools, and
> therefore improve the code quality significantly. This is 'indication'
> of the real attack surface. Note that there are some very bad
> definitions of attack surface used by some researchers.
>
> > Working for a fuzzing vendor I am only too familiar with the Turing
> > halting problem and seeking reality in the midst of eternal runs,
> > but the most interesting thing I found in the past few months (which
> > wasn't technical) is the clash of cultures between QA engineers and
> > Security professionals. It will be very interesting to see where we
> > end up.
>
> Heh, nice to notice that there are three commercial fuzzer vendors
> involved in this discussion...
>
> There are two branches of negative testing:
> 1) Fuzzing starts from infinite and tries to learn intelligence
> 2) Robustness testing starts from finite and tries to add intelligence
>
> After some development both result in better tests in the end, and
> probably into one semi-random approach with a lot of intelligence on
> interface specifications and real-life vulnerability knowledge. Best
> approach is probably somewhere in the middle. For some reason security
> people start with the approach one (maybe because it is easiest for
> finding one problem), and it fits in that scene perfectly (given that
> you only need to find one flaw, and you can invoice for all the used
> time). Us QA people prefer the second approach as it fits perfectly to
> the existing QA processes (as the purpose is to find "all" flaws, and
> as you can basically invoice for the saved time). In my opinion, the
> Turing problem applies to approach one more than to the approach two.
>
> Sorry if I hurt any feelings again (I seem to do that every time I
> write to DailyDave)! As you can see, I tend to have very strong
> opinions against any randomness in fuzzing... But this is the result
> of 10+ years of research experience (6+ years commercially) in fuzzing
> with all possible variants... ;)
>
> Those of you that are planning to attend Blackhat Europe: Codenomicon
> will be there! Come and meet with us (maybe over a beer)!
>
> Best regards,
>
> /Ari
>
> --
> -o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-
> Ari Takanen                       Codenomicon Ltd.
> ari.takanen () codenomicon com       Tutkijantie 4E
> tel: +358-40 50 67678             FIN-90570 Oulu
> http://www.codenomicon.com        Finland
> PGP: http://www.codenomicon.com/codenomicon-key.asc
> -o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-
> _______________________________________________
> Dailydave mailing list
> Dailydave () lists immunitysec com
> http://lists.immunitysec.com/mailman/listinfo/dailydave
_______________________________________________
Dailydave mailing list
Dailydave () lists immunitysec com
http://lists.immunitysec.com/mailman/listinfo/dailydave