Re: The difference between block-based fuzzing and AFL

["Andrew Ruef" <munin () mimisbrunnr net>]

The benefit of a tool like AFL is that it’s black-box: you don’t need a grammar, you don’t need a complicated, rich and 
deep specification of a protocol like RPC that encapsulates checksums, encryption, etc. 

 

AFL (and fuzzers like it) have a strategy to work around their lack of knowledge/a deep specification, though: just 
recompile your application to skip checksums and turn off encryption. 

 

Augh! It’s so cheesy! The indignity! You spend months of your life writing hand-crafted protocol emulators that deal 
with every corner case of a 500 page specification to fuzz the crap out of something and some dude with a 
reinforcement-learning-based mutation-fuzzer written in 200 lines of C code that doesn’t know anything about any 
protocol comes up behind you and says “yeah just comment out the checksum, bro.” 

 

It seems to work, though? The bugs you find with or without the checksums/encryption/”computation” should exist with or 
without that, so why not analyze the software without? To synthesize the bugs into something real you’ll need to do 
some by-hand work to make the checksums work out or add encryption, but it seems unlikely that a bug you find will 
require input that can’t be checksummed or something. It would be bad if your surgery on the program to remove 
checksums / encryption also removed other checks like length checks or a filter, or otherwise damaged the generality of 
your analysis. The fix that AFL like tools offer for this is basically “don’t do that.” Maybe that is a thing that, in 
your mind, buckets AFL in some kind of “specialized tool” category. 

 

And in fact, AFL has found bugs in straight up RPC protocols like CapnProto. 

 

From: dailydave-bounces () lists immunityinc com [mailto:dailydave-bounces () lists immunityinc com] On Behalf Of Dave 
Aitel
Sent: Tuesday, September 13, 2016 8:34 AM
To: dailydave () lists immunityinc com
Subject: [Dailydave] The difference between block-based fuzzing and AFL

 

So let's take a quick break from thinking about how messed up Wassenaar is or what random annoying thing the EFF or 
ACLU said about 0day today and talk about fuzzers. AFL has everyone's mind share, but I you have to point out that it 
is still a VERY specialized tool. 

 

The process of taking a file, sending it into some processing unit, and then figuring out if it crashes, sounds easy 
and generic. But in practice you have to carefully optimize how you do it to get any kind of speed and effectiveness 
out of it. 

 

This is another thing about the Cyber Grand Challenge: I think they optimized the problem set in a way using that 
limited system call VM for AFL-like fuzzers. I'm just going to assume none of the problem sets were a complex RPC-like 
protocol, because we would have seen zero people solve them and DARPA knows that.

 

What I mean is this: It is very hard to optimize the block-based fuzzing technique for automation. But they solve two 
completely different types of problems. 

 

AFL-like fuzzers excel at files for one reason: Files don't do computation. SPIKE-like fuzzers excel at protocols 
because they are there to handle challenge responses, size-fields, checksums, encryption, and other things common in 
network protocols. There's also minor differences in how they handle mutation. And of course, in many cases a 
SPIKE-like fuzzer is EASIER to set up and use than something like AFL, with less problem-optimization needed for 
valuable results.

 

But still, no comparison of a file-fuzzer to a block-based or protocol fuzzer (PEACH/SPIKE/CODENOMICON) is going to be 
apples to apples. It's more like apples to dragons.

 

-dave

 

_______________________________________________
Dailydave mailing list
Dailydave () lists immunityinc com
https://lists.immunityinc.com/mailman/listinfo/dailydave