Fizil AFL
Runs on Windows ✅ There’s a fork
Runs on Unix ❌ ✅
Fast ❌ ✅
Bunnies! ❌
Process models In Process, Out of Process Fork Server, Out of Process
Instrumentation guided Soon? ✅
Automatic instrumentation .NET Assemblies Clang, GCC, Python
Rich suite of fuzzing strategies Getting there! ✅
Automatically disables crash reporting ✅ ❌
Rich tooling ❌ ✅
Proven track record ❌ ✅
Stable ❌ ✅
License Apache 2.0 Apache 2.0
Slide 26
Slide 26 text
No content
Slide 27
Slide 27 text
No content
Slide 28
Slide 28 text
No content
Slide 29
Slide 29 text
https://unsplash.com/search/bug?photo=emTCWiq2txk
Slide 30
Slide 30 text
F u z z i n g
https://commons.wikimedia.org/wiki/File:Rabbit_american_fuzzy_lop_buck_white.jpg
Best Known
Example
Released
Informal
Spec
Formal
Spec
Execute
QuickCheck
1999
“Reversing a list twice should result in the
same list”
prop_RevRev xs = reverse (reverse xs) == xs
where types = xs::[Int]
Main> quickCheck prop_RevRev
OK, passed 100 tests.
Slide 70
Slide 70 text
Best Known
Example
Released
Informal
Spec
Formal
Spec
Execute
QuickCheck
1999
“Reversing a list twice should result in the
same list”
prop_RevRev xs = reverse (reverse xs) == xs
where types = xs::[Int]
Main> quickCheck prop_RevRev
OK, passed 100 tests.
Slide 71
Slide 71 text
Best Known
Example
Released
Informal
Spec
Formal
Spec
Execute
QuickCheck
1999
“Reversing a list twice should result in the
same list”
prop_RevRev xs = reverse (reverse xs) == xs
where types = xs::[Int]
Main> quickCheck prop_RevRev
OK, passed 100 tests.
AFL
2007
“System under test shouldn’t crash no matter what I
pass to it”
if (WIFSIGNALED(status) #$ !stop_soon) {
kill_signal = WTERMSIG(status);
return FAULT_CRASH;
}
./afl-fuzz -i testcase_dir -o findings_dir -- \
/path/to/tested/program [&&.program's
cmdline&&.]
Slide 72
Slide 72 text
Best Known
Example
Released
Informal
Spec
Formal
Spec
Execute
QuickCheck
1999
“Reversing a list twice should result in the
same list”
prop_RevRev xs = reverse (reverse xs) == xs
where types = xs::[Int]
Main> quickCheck prop_RevRev
OK, passed 100 tests.
AFL
2007
“System under test shouldn’t crash no matter what I
pass to it”
if (WIFSIGNALED(status) #$ !stop_soon) {
kill_signal = WTERMSIG(status);
return FAULT_CRASH;
}
./afl-fuzz -i testcase_dir -o findings_dir -- \
/path/to/tested/program [&&.program's
cmdline&&.]
Thought Experiment:
W h a t I f
A u t o m a t e d T e s t s
W e r e P e r f e c t ?
Slide 76
Slide 76 text
No content
Slide 77
Slide 77 text
No content
Slide 78
Slide 78 text
W h a t I f S e c u r i t y
A n a l y s i s T o o l s
W e r e P e r f e c t ?
Slide 79
Slide 79 text
–DNI James Clapper
“Something like 90 percent of cyber intrusions start with
phishing… Somebody always falls for it.”
https://twitter.com/ODNIgov/status/776070411482193920
Manual Testing
Examples Exploratory testing, Binary analysis
Effort Very high
Killer App
Finding cases where code technically correct but fails at human-
computer interaction
Major Disadvantage Often misused
Slide 83
Slide 83 text
Dynamic Analysis
Examples QuickCheck, AFL, sqlmap
Effort Low
Killer App More like an app killer, amiright?
Major Disadvantage Tends to find a few specific (though important!) bugs
Slide 84
Slide 84 text
Static Analysis
Examples FxCop, FindBugs, Coverity, Veracode
Effort Very low
Killer App Cheaper than air. Just do it.
Major Disadvantage Limited to finding a few hundred important kinds of bugs
Slide 85
Slide 85 text
Formal Verification / Symbolic Execution
Examples VCC, TLA+, Cryptol
Effort High effort but correspondingly high return
Killer App MiTLS, Hyper-V Memory Manager
Major Disadvantage Hard to find people with skill set
Slide 86
Slide 86 text
Program Synthesis
Examples Nothing off the shelf, really, but Agda and Z3 help
Effort PhD-level research
Killer App Elimination of incidental complexity
Major Disadvantage Doesn’t really exist in general form
Slide 87
Slide 87 text
No content
Slide 88
Slide 88 text
No content
Slide 89
Slide 89 text
How Amazon Web Services Uses Formal Methods
“Formal methods are a big success at AWS, helping us
prevent subtle but serious bugs from reaching
production, bugs we would not have found through any
other technique. They have helped us devise aggressive
optimizations to complex algorithms without sacrificing
quality.”
http://research.microsoft.com/en-us/um/people/lamport/tla/amazon.html
Slide 90
Slide 90 text
No content
Slide 91
Slide 91 text
No content
Slide 92
Slide 92 text
“Finding and Understanding Bugs in C Compilers,”
Yang et al.
https://www.flux.utah.edu/paper/yang-pldi11
Slide 93
Slide 93 text
No content
Slide 94
Slide 94 text
===================================
Technical "whitepaper" for afl-fuzz
===================================
This document provides a quick overview of the guts of American Fuzzy Lop.
See README for the general instruction manual; and for a discussion of
motivations and design goals behind AFL, see historical_notes.txt.
0) Design statement
-------------------
American Fuzzy Lop does its best not to focus on any singular principle of
operation and not be a proof-of-concept for any specific theory. The tool can
be thought of as a collection of hacks that have been tested in practice,
found to be surprisingly effective, and have been implemented in the simplest,
most robust way I could think of at the time.
Many of the resulting features are made possible thanks to the availability of
lightweight instrumentation that served as a foundation for the tool, but this
mechanism should be thought of merely as a means to an end. The only true
governing principles are speed, reliability, and ease of use.
1) Coverage measurements
------------------------
The instrumentation injected into compiled programs captures branch (edge)
coverage, along with coarse branch-taken hit counts. The code injected at
branch points is essentially equivalent to:
cur_location = ;
shared_mem[cur_location ^ prev_location]++;
http://lcamtuf.coredump.cx/afl/technical_details.txt
Slide 95
Slide 95 text
M e m o r y
Slide 96
Slide 96 text
{ "a" : "bc" }
Slide 97
Slide 97 text
let jsonNetResult =
try JsonConvert.DeserializeObject(str) |> ignore
Success
with
| :? JsonReaderException as jre -> jre.Message |> Error
| :? JsonSerializationException as jse -> jse.Message |> Error
| :? System.FormatException as fe ->
if fe.Message.StartsWith("Invalid hex character”) // hard coded in Json.NET
then fe.Message |> Error
else reraise()
⃪ T
est
⬑ Special case error stuff
Slide 98
Slide 98 text
use proc = new Process()
proc.StartInfo.FileName <- executablePath
inputMethod.BeforeStart proc testCase.Data
proc.StartInfo.UseShellExecute <- false
proc.StartInfo.RedirectStandardOutput <- true
proc.StartInfo.RedirectStandardError <- true
proc.StartInfo.EnvironmentVariables.Add(SharedMemory.environmentVariableName, sharedMemoryName)
let output = new System.Text.StringBuilder()
let err = new System.Text.StringBuilder()
proc.OutputDataReceived.Add(fun args -> output.Append(args.Data) |> ignore)
proc.ErrorDataReceived.Add (fun args -> err.Append(args.Data) |> ignore)
proc.Start() |> ignore
inputMethod.AfterStart proc testCase.Data
proc.BeginOutputReadLine()
proc.BeginErrorReadLine()
proc.WaitForExit()
let exitCode = proc.ExitCode
let crashed = exitCode = WinApi.ClrUnhandledExceptionCode
⃪ Set up
⃪ Read results
⃪ Important bit
Slide 99
Slide 99 text
No content
Slide 100
Slide 100 text
/// An ordered list of functions to use when starting with a single piece of
/// example data and producing new examples to try
let private allStrategies = [
bitFlip 1
bitFlip 2
bitFlip 4
byteFlip 1
byteFlip 2
byteFlip 4
arith8
arith16
arith32
interest8
interest16
]
Slide 101
Slide 101 text
let totalBits = bytes.Length * 8
let testCases = seq {
for bit = 0 to totalBits - flipBits do
let newBytes = Array.copy bytes
let firstByte = bit / 8
let firstByteMask, secondByteMask = bitMasks(bit, flipBits)
let newFirstByte = bytes.[firstByte] ^^^ firstByteMask
newBytes.[firstByte] <- newFirstByte
let secondByte = firstByte + 1
if secondByteMask <> 0uy && secondByte < bytes.Length
then
let newSecondByte = bytes.[secondByte] ^^^ secondByteMask
newBytes.[secondByte] <- newSecondByte
yield newBytes
}
Fuzz one byte →
^^^ means xor
↓
Standard Accepts, Json.NET Rejects
Value
88888888888888888888888888888888888888888888888888
88888888888888888888888888888888888888888888888888
88888888888888888888888888888888888888888888888888
88888888888888888888888888888888888888888888888888
88888888888888888888888888888888888888888888888888
Standard Says No limit
Json.NET MaximumJavascriptIntegerCharacterLength = 380;
Slide 122
Slide 122 text
Standard Rejects, Json.NET Accepts
Value [,,,]
Standard Says
A JSON value MUST be an object, array, number, or string, or one
of
the following three literal names:
false null true
Json.NET [null, null, null, null]
Slide 123
Slide 123 text
I m p l e m e n t a t i o n
D e t a i l s
Slide 124
Slide 124 text
let private insertTraceInstruction(ilProcessor: ILProcessor, before: Instruction, state) =
let compileTimeRandom = state.Random.Next(0, UInt16.MaxValue |> Convert.ToInt32)
let ldArg = ilProcessor.Create(OpCodes.Ldc_I4, compileTimeRandom)
let callTrace = ilProcessor.Create(OpCodes.Call, state.Trace)
ilProcessor.InsertBefore(before, ldArg)
ilProcessor.InsertAfter (ldArg, callTrace)
This margin is too narrow to contain a try/finally example, so see:
https://goo.gl/W4y7JH
–ECMA-335, Common Language Infrastructure (CLI),
Partition I
“If marked BeforeFieldInit then the type’s initializer
method is executed at, or sometime before, first access to
any static field defined for that type.”
Slide 131
Slide 131 text
f ( x ) = f ( x )
t i m e ( f ( x ) ) ! = t i m e ( f ( x ) )
✅
❌
Slide 132
Slide 132 text
U n i c o d e
Original JSON
{ "a": "bc" }
ASCII Bytes
7B 20 22 61 22 20 3A 20 22 62 63 22 20 7D
UTF-8 with Byte Order Mark
EF BB BF 7B 20 22 61 22 20 3A 20 22 62 63 22 20 7D
UTF-16 BE with BOM
FE FF 00 7B 00 20 00 22 00 61 00 22 00 20 00 3A 00 20 00 22
00 62 00 63 00 22 00 20 00 7D
Slide 133
Slide 133 text
T h a n k Y o u !
Presentation Review
Cassandra Faris
Chad James
Damian Synadinos
Doug Mair
Tommy Graves
Source Code Inspiration
Michał Zalewski
Nicolas Seriot
Everyone Who Works on dnSpy
& Mono.Cecil
Slide 134
Slide 134 text
No content
Slide 135
Slide 135 text
C r a i g S t u n t z
@craigstuntz
[email protected]
http://www.craigstuntz.com
http://www.meetup.com/Papers-We-Love-Columbus/
https://speakerdeck.com/craigstuntz