Dailydave mailing list archives
Re: The difference between block-based fuzzing and AFL
From: Ryan Stortz <rstortz () gmail com>
Date: Tue, 13 Sep 2016 14:55:37 -0400
I don't think it's an apples-to-oranges comparison to compare these fuzzers against the Cyber Grand Challenge test set ( https://github.com/trailofbits/cb-multios). In fact, the CGC test set is a perfect shooting gallery. The test set is entirely comprised of network services that implement protocols that represent real world software. DECREE has no knowledge of file systems or files at all. The protocols are frequently simplified, but over the 241 challenges you can make plenty of fair comparisons. There are challenges that implement: Nonces * https://github.com/trailofbits/cb-multios/tree/master/original-challenges/NoHiC * https://github.com/trailofbits/cb-multios/tree/master/original-challenges/Griswold ) Checksums * https://github.com/trailofbits/cb-multios/tree/master/original-challenges/ValveChecks * https://github.com/trailofbits/cb-multios/blob/master/original-challenges/Packet_Receiver RSA-like authentication * https://github.com/trailofbits/cb-multios/tree/master/original-challenges/FASTLANE Several challenges are "in spirit" re-implementations of vulnerabilities that were publicly exploited, for example: • Morris Worm ( https://github.com/trailofbits/cb-multios/tree/master/original-challenges/REMATCH_1--Hat_Trick--Morris_Worm ) • Crackaddr ( https://github.com/trailofbits/cb-multios/tree/master/original-challenges/REMATCH_2--Mail_Server--Crackaddr ) • Stuxnet LNK ( https://github.com/trailofbits/cb-multios/tree/master/original-challenges/REMATCH_5--File_Explorer--LNK_Bug ) • Heartbleed ( https://github.com/trailofbits/cb-multios/tree/master/original-challenges/REMATCH_6--Secure_Server--Heartbleed ) • Linux FUTEX / TowelRoot ( https://github.com/trailofbits/cb-multios/tree/master/original-challenges/REDPILL ) It's true that CGC's ABI was simplified to make this analysis more feasible, but it's still expressive enough to support complicated real-world vulnerabilities. Many of these challenges were solved in competition, although we're still waiting for someone to do the analysis and present it publicly. Anecdotally, the Trail of Bits CRS can generate crashing inputs for ~50% of these challenges, including ones with nonces and checksums. In the next few months, I will compare both static and dynamic analysis tools using this software repository. We (Trail of Bits) hope to release repeatable DevOps-style test results. You're absolutely right about optimizing fuzzers for your target and I hope to have statistics soon to back up that claim. Some of my initial results will be presented at inbot.xyz at the end of September. Cheers, Ryan @withzombies On Tue, Sep 13, 2016 at 11:33 AM, Dave Aitel <dave.aitel () gmail com> wrote:
So let's take a quick break from thinking about how messed up Wassenaar is or what random annoying thing the EFF or ACLU said about 0day today and talk about fuzzers. AFL has everyone's mind share, but I you have to point out that it is still a VERY specialized tool. The process of taking a file, sending it into some processing unit, and then figuring out if it crashes, sounds easy and generic. But in practice you have to carefully optimize how you do it to get any kind of speed and effectiveness out of it. This is another thing about the Cyber Grand Challenge: I think they optimized the problem set in a way using that limited system call VM for AFL-like fuzzers. I'm just going to assume none of the problem sets were a complex RPC-like protocol, because we would have seen zero people solve them and DARPA knows that. What I mean is this: It is very hard to optimize the block-based fuzzing technique for automation. But they solve two completely different types of problems. AFL-like fuzzers excel at files for one reason: Files don't do computation. SPIKE-like fuzzers excel at protocols because they are there to handle challenge responses, size-fields, checksums, encryption, and other things common in network protocols. There's also minor differences in how they handle mutation. And of course, in many cases a SPIKE-like fuzzer is EASIER to set up and use than something like AFL, with less problem-optimization needed for valuable results. But still, no comparison of a file-fuzzer to a block-based or protocol fuzzer (PEACH/SPIKE/CODENOMICON) is going to be apples to apples. It's more like apples to dragons. -dave _______________________________________________ Dailydave mailing list Dailydave () lists immunityinc com https://lists.immunityinc.com/mailman/listinfo/dailydave
_______________________________________________ Dailydave mailing list Dailydave () lists immunityinc com https://lists.immunityinc.com/mailman/listinfo/dailydave
Current thread:
- The difference between block-based fuzzing and AFL Dave Aitel (Sep 13)
- Re: The difference between block-based fuzzing and AFL Andrew Ruef (Sep 13)
- Re: The difference between block-based fuzzing and AFL Ryan Stortz (Sep 13)
- Re: The difference between block-based fuzzing and AFL Michal Zalewski (Sep 15)