Re: The difference between block-based fuzzing and AFL

[Ryan Stortz <rstortz () gmail com>]

I don't think it's an apples-to-oranges comparison to compare these fuzzers
against the Cyber Grand Challenge test set (
https://github.com/trailofbits/cb-multios). In fact, the CGC test set is a
perfect shooting gallery. The test set is entirely comprised of network
services that implement protocols that represent real world software.
DECREE has no knowledge of file systems or files at all. The protocols are
frequently simplified, but over the 241 challenges you can make plenty of
fair comparisons.

There are challenges that implement:

Nonces

*
https://github.com/trailofbits/cb-multios/tree/master/original-challenges/NoHiC

*
https://github.com/trailofbits/cb-multios/tree/master/original-challenges/Griswold
)

Checksums

*
https://github.com/trailofbits/cb-multios/tree/master/original-challenges/ValveChecks

*
https://github.com/trailofbits/cb-multios/blob/master/original-challenges/Packet_Receiver

RSA-like authentication

*
https://github.com/trailofbits/cb-multios/tree/master/original-challenges/FASTLANE

Several challenges are "in spirit" re-implementations of vulnerabilities
that were publicly exploited, for example:

•    Morris Worm (
https://github.com/trailofbits/cb-multios/tree/master/original-challenges/REMATCH_1--Hat_Trick--Morris_Worm
)

•    Crackaddr (
https://github.com/trailofbits/cb-multios/tree/master/original-challenges/REMATCH_2--Mail_Server--Crackaddr
)

•    Stuxnet LNK (
https://github.com/trailofbits/cb-multios/tree/master/original-challenges/REMATCH_5--File_Explorer--LNK_Bug
)

•    Heartbleed (
https://github.com/trailofbits/cb-multios/tree/master/original-challenges/REMATCH_6--Secure_Server--Heartbleed
)

•    Linux FUTEX / TowelRoot (
https://github.com/trailofbits/cb-multios/tree/master/original-challenges/REDPILL
)

It's true that CGC's ABI was simplified to make this analysis more
feasible, but it's still expressive enough to support complicated
real-world vulnerabilities. Many of these challenges were solved in
competition, although we're still waiting for someone to do the analysis
and present it publicly. Anecdotally, the Trail of Bits CRS can generate
crashing inputs for ~50% of these challenges, including ones with nonces
and checksums.

In the next few months, I will compare both static and dynamic analysis
tools using this software repository. We (Trail of Bits) hope to release
repeatable DevOps-style test results. You're absolutely right about
optimizing fuzzers for your target and I hope to have statistics soon to
back up that claim. Some of my initial results will be presented at
inbot.xyz at the end of September.

Cheers,

Ryan

@withzombies


On Tue, Sep 13, 2016 at 11:33 AM, Dave Aitel <dave.aitel () gmail com> wrote:

> So let's take a quick break from thinking about how messed up Wassenaar is
> or what random annoying thing the EFF or ACLU said about 0day today and
> talk about fuzzers. AFL has everyone's mind share, but I you have to point
> out that it is still a VERY specialized tool.
>
> The process of taking a file, sending it into some processing unit, and
> then figuring out if it crashes, sounds easy and generic. But in practice
> you have to carefully optimize how you do it to get any kind of speed and
> effectiveness out of it.
>
> This is another thing about the Cyber Grand Challenge: I think they
> optimized the problem set in a way using that limited system call VM for
> AFL-like fuzzers. I'm just going to assume none of the problem sets were a
> complex RPC-like protocol, because we would have seen zero people solve
> them and DARPA knows that.
>
> What I mean is this: It is very hard to optimize the block-based fuzzing
> technique for automation. But they solve two completely different types of
> problems.
>
> AFL-like fuzzers excel at files for one reason: Files don't do
> computation. SPIKE-like fuzzers excel at protocols because they are there
> to handle challenge responses, size-fields, checksums, encryption, and
> other things common in network protocols. There's also minor differences in
> how they handle mutation. And of course, in many cases a SPIKE-like fuzzer
> is EASIER to set up and use than something like AFL, with less
> problem-optimization needed for valuable results.
>
> But still, no comparison of a file-fuzzer to a block-based or protocol
> fuzzer (PEACH/SPIKE/CODENOMICON) is going to be apples to apples. It's more
> like apples to dragons.
>
> -dave
>
>
> _______________________________________________
> Dailydave mailing list
> Dailydave () lists immunityinc com
> https://lists.immunityinc.com/mailman/listinfo/dailydave
_______________________________________________
Dailydave mailing list
Dailydave () lists immunityinc com
https://lists.immunityinc.com/mailman/listinfo/dailydave