Tweet
Tweet

Slide 1

Slide 1 text

High Speed Bug Discovery with Fuzzing Craig Stuntz ∈ Improving

Slide 2

Slide 2 text

Slides https://speakerdeck.com/craigstuntz

Slide 3

Slide 3 text

Fuzzing Is Bad …and you should feel bad about using it

Slide 4

Slide 4 text

How Do We Design Software Security? - Usability Studies (Does it work for human beings?) - Threat Modeling - Formal Verification - Static Analysis - Auditing - Pentesting - Throw Random Crap At It And See If It Dies

Slide 5

Slide 5 text

Offensive Software

Slide 6

Slide 6 text

“ Unfortunately, C and C++ are mostly taught the old way, as if programming in them isn’t like walking in a minefield. Nor have the books about C and C++ caught up with the current reality. These things must change. -John Regehr https://blog.regehr.org/archives/1520

Slide 7

Slide 7 text

“Yet, despite the crippling and obvious limitations of fuzzing and the virtues of symbolic execution, there is one jarring discord: I'm fairly certain that probably around 70% of all remote code execution vulnerabilities disclosed in the past few years trace back to fairly “dumb” fuzzing tools, with the pattern showing little change over time. -Michał Zalewski https://lcamtuf.blogspot.com/2015/02/symbolic-execution-in-vuln-research.html

Slide 8

Slide 8 text

let private bitMasks(bitIndex: int, bitsToFlip: int) = match bitIndex % 8, bitsToFlip with | 0, 1 !→ 0b00000001uy, 0b00000000uy | 1, 1 !→ 0b00000010uy, 0b00000000uy | 2, 1 !→ 0b00000100uy, 0b00000000uy | 3, 1 !→ 0b00001000uy, 0b00000000uy | 4, 1 !→ 0b00010000uy, 0b00000000uy | 5, 1 !→ 0b00100000uy, 0b00000000uy | 6, 1 !→ 0b01000000uy, 0b00000000uy | 7, 1 !→ 0b10000000uy, 0b00000000uy | 0, 2 !→ 0b00000011uy, 0b00000000uy | 1, 2 !→ 0b00000110uy, 0b00000000uy | 2, 2 !→ 0b00001100uy, 0b00000000uy | 3, 2 !→ 0b00011000uy, 0b00000000uy | 4, 2 !→ 0b00110000uy, 0b00000000uy | 5, 2 !→ 0b01100000uy, 0b00000000uy | 6, 2 !→ 0b11000000uy, 0b00000000uy | 7, 2 !→ 0b10000000uy, 0b00000001uy | 0, 4 !→ 0b00001111uy, 0b00000000uy | 1, 4 !→ 0b00011110uy, 0b00000000uy | 2, 4 !→ 0b00111100uy, 0b00000000uy | 3, 4 !→ 0b01111000uy, 0b00000000uy | 4, 4 !→ 0b11110000uy, 0b00000000uy | 5, 4 !→ 0b11100000uy, 0b00000001uy | 6, 4 !→ 0b11000000uy, 0b00000011uy | 7, 4 !→ 0b10000000uy, 0b00000111uy | bit, _ !→ failwithf "Unsupported bit %d or bitsToFlip %d" bit bitsToFlip

Slide 9

Slide 9 text

Spoilers! Why should I care? (because it’s surprisingly effective at finding bugs in software) What is it? (a simple, property-based randomized testing technique) When should I use it? (integration testing complex systems with infinite input values) How do I get started? (I’ll suggest a bunch of tools) Should I write my own? (yes, and I have stories!)

Slide 10

Slide 10 text

Why?

Slide 11

Slide 11 text

400 Crashes, 106 Distinct Security Bugs in Adobe Flash Player https://security.googleblog.com/2011/08/fuzzing-at-scale.html

Slide 12

Slide 12 text

325 C Compiler bugs in GCC, Clang, & Others https://www.flux.utah.edu/paper/yang-pldi11

Slide 13

Slide 13 text

“…our most prolific bug-finding tool.” Robert Guo http://queue.acm.org/detail.cfm?ref=rss&id=3059007

Slide 14

Slide 14 text

“fuzzers…can be integrated into a CI environment for permanent protection henceforth” Guido Vranken https://guidovranken.wordpress.com/2017/06/21/the-openvpn-post-audit-bug-bonanza/

Slide 15

Slide 15 text

DARPA Cyber Grand Challenge Mayhem, Xandra, Mechanical Phish, Galactica http://blogs.grammatech.com/the-cyber-grand-challenge

Slide 16

Slide 16 text

https://commons.wikimedia.org/wiki/File:Rabbit_american_fuzzy_lop_buck_white.jpg What is it?

Slide 17

Slide 17 text

Prevent Regressions Bug Discovery Help with Code Design Meets Specifications Fuzzing Integration testing Unit testing Formal verification Exploratory testing

Slide 18

Slide 18 text

How Many Cases Should We Test? One Only the Most Interesting Every Possible Case Unit Testing Fuzzing Formal Verification

Slide 19

Slide 19 text

Property -Based Random Testing QuickCheck Chaos Monkey Fuzzing Scientist

Slide 20

Slide 20 text

System Under Test Corpus (Collection of Examples) Property Magic!

Slide 21

Slide 21 text

Corpus A few handwritten examples Fuzzing databases Harvest from test suites, defect reports Harvest from public Internet

Slide 22

Slide 22 text

System Under Test A function Entire application Part of OS kernel

Slide 23

Slide 23 text

Properties Does it crash? Does it hang? Is the output “valid”? Does execution trip an address or memory sanitizer? Does the output match some other system?

Slide 24

Slide 24 text

Magic Mutation of corpus Coverage guidance Lots of test runs

Slide 25

Slide 25 text

Possible Inputs Random Inputs Interesting Inputs Random Inputs with Profile Guidance

Slide 26

Slide 26 text

Getting Started with afl - Compile system under test with instrumentation - Place corpus input(s) in a folder - Invoke afl - Wait for bugs https://fuzzing-project.org/tutorial3.html

Slide 27

Slide 27 text

Compile with Instrumentation $ ./configure CC="afl-gcc" \ CXX="afl-g++" \ --disable-shared; \ make

Slide 28

Slide 28 text

Place Corpus in Folder $ mkdir in $ cd in $ cat > foo.json { "a": "bc" } ^D $ cd ..

Slide 29

Slide 29 text

Invoke afl $ afl-fuzz -i in -o out \ my_json_parser @@ folder containing corpus “@@“ means “the current test case” system under test findings go here

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

some_project ├── in └─┬ out ├── queue ├── crashes └── hangs

Slide 32

Slide 32 text

afl In a Nutshell ⃗

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

afl Fuzz Strategies Walking bit flips (try flipping each bit in input individually) Walking byte flips (try flipping each contiguous set of 8 bits) Simple arithmetic (increment or decrement bytes in the file by certain small values) Known integers (or dictionaries) (replace bytes with “problematic” 8, 16, and 32 bit integers like 0 and FF) Profile-guided stacked tweaks and test case splicing (magic!) https://lcamtuf.blogspot.com/2014/08/binary-fuzzing-strategies-what-works.html

Slide 35

Slide 35 text

Walking Bit Flip Original 01010101 Flip bit 0 01010100 Flip bit 1 01010111 Flip bit 2 01010001 Walking 2 Bit Flip Original 01010101 Flip bits 0,1 01010110 Flip bits 2,1 01010011 Flip bits 3,2 01001101

Slide 36

Slide 36 text

Trace Execution Paths A B D C E F ?

Slide 37

Slide 37 text

When Should I Use It? When is it useful? https://it.wikipedia.org/wiki/File:Coniglio_ariete.JPG

Slide 38

Slide 38 text

Unit Tests Fuzzing Useful For Preventing Regressions, Design Finding New Bugs Tests Functions Any Level Test Examples Hand-selected values Corpus + Mutation Execution Time Milliseconds Weeks Magic? No Yes

Slide 39

Slide 39 text

Instrumentation for profile guidance

Slide 40

Slide 40 text

$ ./configure CC="afl-gcc" \ CXX="afl-g++" \ --disable-shared; \ make

Slide 41

Slide 41 text

$ pip install python-afl ——— afl.init()

Slide 42

Slide 42 text

Properties a.k.a. Specifications https://lorinhochstein.wordpress.com/2014/06/04/crossing-the-river-with-tla/

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

STJSON A JSON Parser in Swift 3 compliant with RFC 7159 STJSON was written along with the article Parsing JSON is a Minefield. Basic usage: var p = STJSONParser(data: data) do { let o = p.parse() } catch let e { print(e) } Instantiation with options: var p = STJSON(data:data, maxParserDepth:1024, options:[.useUnicodeReplacementCharacter]) https://github.com/nst/STJSON https://github.com/CraigStuntz/Fizil/tree/master/StJson

Slide 46

Slide 46 text

Can You Fuzz the Domain?

Slide 47

Slide 47 text

Dumb Fuzzer Mangling Byte Arrays

Slide 48

Slide 48 text

public byte[] ResizePng( byte[] image) {

Slide 49

Slide 49 text

public boolean SomeFunction( SomeEnum firstArg, int secondArg) { ✗

Slide 50

Slide 50 text

Smart Fuzzing MongoDB Expression Grammar http://queue.acm.org/detail.cfm?ref=rss&id=3059007

Slide 51

Slide 51 text

public Assembly Compile( AbstractNode syntaxTree) {

Slide 52

Slide 52 text

Worth the Wait? Results Can Take Weeks

Slide 53

Slide 53 text

Isn’t it just for Security? https://www.flickr.com/photos/wocintechchat/25721078480/

Slide 54

Slide 54 text

How do I get started?

Slide 55

Slide 55 text

How to Get Started with Fuzzing 1. Find a program to test 2. Find a fuzzer 3. Find a corpus 4. Choose a property 5. Let it run!

Slide 56

Slide 56 text

Fuzzers https://commons.wikimedia.org/wiki/File:Holland_Lop_with_Broken_Orange_Coloring.jpg

Slide 57

Slide 57 text

libfuzzer Fuzz testing for LLVM compilers http://llvm.org/docs/LibFuzzer.html

Slide 58

Slide 58 text

$ cargo install cargo-fuzz $ cargo fuzz init

Slide 59

Slide 59 text

$ gzip -c /bin/bash > sample.gz $ while true do radamsa sample.gz > fuzzed.gz gzip -dc fuzzed.gz > /dev/null test $? -gt 127 && break done ← Fuzz the corpus ← Execute S.O.T. ← Check a property ← Repeat a lot! https://github.com/aoh/radamsa

Slide 60

Slide 60 text

ClusterFuzz Submit a fuzzer, win a bounty https://security.googleblog.com/2016/08/guided-in-process-fuzzing-of-chrome.html

Slide 61

Slide 61 text

afl

Slide 62

Slide 62 text

OSS-Fuzz Submit your project https://github.com/google/oss-fuzz

Slide 63

Slide 63 text

“Project Springfield” https://www.microsoft.com/en-us/security-risk-detection/

Slide 64

Slide 64 text

burp, ZAP

Slide 65

Slide 65 text

Corpus https://en.wikipedia.org/wiki/File:Long_Room_Interior,_Trinity_College_Dublin,_Ire

Slide 66

Slide 66 text

“ We didn't call it fuzzing back in the 1950s, but it was our standard practice to test programs by inputting decks of punch cards taken from the trash. -Gerald M. Weinberg http://secretsofconsulting.blogspot.com/2017/02/fuzz-testing-and-fuzz-history.html

Slide 67

Slide 67 text

No content

Slide 68

Slide 68 text

Fuzzing SQLite with afl Start with a single test case: create table t1(one smallint); insert into t1 values(1); select * from t1; Add a list of reserved words from documentation Then extract SQL statements from SQLite unit tests (550 files at around 220 bytes each) https://lcamtuf.blogspot.com/2015/04/finding-bugs-in-sqlite-easy-way.html

Slide 69

Slide 69 text

Properties

Slide 70

Slide 70 text

Don’t Crash or Hang

Slide 71

Slide 71 text

Sanitizers and Canaries https://docs.google.com/presentation/d/19OSgb1N9Ezef39Blb-5lkzycq7-tMtAvy825FofyrmY

Slide 72

Slide 72 text

Validators

Slide 73

Slide 73 text

End to End Input Output Same as Input? f(input) f-1(input)

Slide 74

Slide 74 text

Legacy Code https://commons.wikimedia.org/wiki/File:Blue-punch-card-front-horiz.png

Slide 75

Slide 75 text

One of The Things …Is Not Like the Others!

Slide 76

Slide 76 text

Should I write my own?

Slide 77

Slide 77 text

Heck Yeah! https://github.com/CraigStuntz/Fizil

Slide 78

Slide 78 text

Interesting Stuff I Learned While Writing a Fuzzer - Deceptive simplicity of fuzzing - F# bitwise operations - How to instrument .NET code - dnSpy is awesome - Same input -> Same code -> Different paths - Strong naming is painful - Unicode is also painful - Speed is everything

Slide 79

Slide 79 text

let jsonNetResult = try JsonConvert.DeserializeObject(str) |> ignore Success with | :? JsonReaderException as jre -> jre.Message |> Error | :? JsonSerializationException as jse -> jse.Message |> Error | :? System.FormatException as fe -> if fe.Message.StartsWith("Invalid hex character”) // hard coded in Json.NET then fe.Message |> Error else reraise() ⃪ T est ⬑ Special case error stuff

Slide 80

Slide 80 text

use proc = new Process() proc.StartInfo.FileName <- executablePath inputMethod.BeforeStart proc testCase.Data proc.StartInfo.UseShellExecute <- false proc.StartInfo.RedirectStandardOutput <- true proc.StartInfo.RedirectStandardError <- true proc.StartInfo.EnvironmentVariables.Add(SharedMemory.environmentVariableName, sharedMemoryName) let output = new System.Text.StringBuilder() let err = new System.Text.StringBuilder() proc.OutputDataReceived.Add(fun args -> output.Append(args.Data) |> ignore) proc.ErrorDataReceived.Add (fun args -> err.Append(args.Data) |> ignore) proc.Start() |> ignore inputMethod.AfterStart proc testCase.Data proc.BeginOutputReadLine() proc.BeginErrorReadLine() proc.WaitForExit() let exitCode = proc.ExitCode let crashed = exitCode = WinApi.ClrUnhandledExceptionCode ⃪ Set up ⃪ Read results ⃪ Important bit

Slide 81

Slide 81 text

/// An ordered list of functions to use when starting with a single piece of /// example data and producing new examples to try let private allStrategies(dictionaryValues: byte[][])= [ bitFlip 1 bitFlip 2 bitFlip 4 byteFlip 1 byteFlip 2 byteFlip 4 arith8 arith16 arith32 dictionary dictionaryValues interest8 interest16 ]

Slide 82

Slide 82 text

let totalBits = bytes.Length * 8 let testCases = seq { for bit = 0 to totalBits - flipBits do let newBytes = Array.copy bytes let firstByte = bit / 8 let firstByteMask, secondByteMask = bitMasks(bit, flipBits) let newFirstByte = bytes.[firstByte] ^^^ firstByteMask newBytes.[firstByte] <- newFirstByte let secondByte = firstByte + 1 if secondByteMask <> 0uy && secondByte < bytes.Length then let newSecondByte = bytes.[secondByte] ^^^ secondByteMask newBytes.[secondByte] <- newSecondByte yield newBytes } Fuzz one byte → ^^^ means xor ↓

Slide 83

Slide 83 text

No content

Slide 84

Slide 84 text

No content

Slide 85

Slide 85 text

private static void F(string arg) { Console.WriteLine("f"); Console.Error.WriteLine("Error!"); Environment.Exit(1); }

Slide 86

Slide 86 text

private static void F(string arg) { instrument.Trace(29875); Console.WriteLine("f"); Console.Error.WriteLine("Error!"); Environment.Exit(1); } ← Random number

Slide 87

Slide 87 text

private static void F(string arg) { #if MANUAL_INSTRUMENTATION instrument.Trace(29875); #endif Console.WriteLine("f"); Console.Error.WriteLine("Error!"); Environment.Exit(1); }

Slide 88

Slide 88 text

let stringify (ob: obj) : string = JsonConvert.SerializeObject(ob) // Method: System.String\u0020Program::stringify(System.Object) .body stringify { arg_02_0 [generated] arg_07_0 [generated] nop() arg_02_0 = ldloc(ob) arg_07_0 = call(JsonConvert::SerializeObject, arg_02_0) ret(arg_07_0) } // Method: System.String\u0020Program::stringify(System.Object) .body stringify { arg_05_0 [generated] arg_0C_0 [generated] arg_11_0 [generated] arg_05_0 = ldc.i4(23831) call(Instrument::Trace, arg_05_0) nop() arg_0C_0 = ldloc(ob) arg_11_0 = call(JsonConvert::SerializeObject, arg_0C_0) ret(arg_11_0) }

Slide 89

Slide 89 text

No content

Slide 90

Slide 90 text

let private insertTraceInstruction(ilProcessor: ILProcessor, before: Instruction, state) = let compileTimeRandom = state.Random.Next(0, UInt16.MaxValue |> Convert.ToInt32) let ldArg = ilProcessor.Create(OpCodes.Ldc_I4, compileTimeRandom) let callTrace = ilProcessor.Create(OpCodes.Call, state.Trace) ilProcessor.InsertBefore(before, ldArg) ilProcessor.InsertAfter (ldArg, callTrace) This margin is too narrow to contain a try/finally example, so see: https://goo.gl/W4y7JH

Slide 91

Slide 91 text

No content

Slide 92

Slide 92 text

http://www.json.org/

Slide 93

Slide 93 text

https://tools.ietf.org/html/rfc4627

Slide 94

Slide 94 text

http://www.ecma-international.org/ecma-262/5.1/#sec-15.12

Slide 95

Slide 95 text

http://www.ecma-international.org/publications/standards/Ecma-404.htm

Slide 96

Slide 96 text

https://tools.ietf.org/html/rfc7158

Slide 97

Slide 97 text

https://tools.ietf.org/html/rfc7159

Slide 98

Slide 98 text

https://github.com/nst/STJSON

Slide 99

Slide 99 text

https://github.com/CraigStuntz/Fizil/blob/master/StJson/StJsonParser.fs

Slide 100

Slide 100 text

{ "a" : "bc" }

Slide 101

Slide 101 text

No content

Slide 102

Slide 102 text

No content

Slide 103

Slide 103 text

Standard Accepts, Json.NET Rejects Value 88888888888888888888888888888888888888888888888888 88888888888888888888888888888888888888888888888888 88888888888888888888888888888888888888888888888888 88888888888888888888888888888888888888888888888888 88888888888888888888888888888888888888888888888888 Standard Says No limit Json.NET MaximumJavascriptIntegerCharacterLength = 380;

Slide 104

Slide 104 text

Standard Rejects, Json.NET Accepts Value [,,,] Standard Says A JSON value MUST be an object, array, number, or string, or one of the following three literal names: false null true Json.NET [null, null, null, null]

Slide 105

Slide 105 text

No content

Slide 106

Slide 106 text

let private removeStrongName (assemblyDefinition : AssemblyDefinition) = let name = assemblyDefinition.Name; name.HasPublicKey <- false; name.PublicKey <- Array.empty; assemblyDefinition.Modules |> Seq.iter ( fun moduleDefinition -> moduleDefinition.Attributes <- moduleDefinition.Attributes &&& ~~~ModuleAttributes.StrongNameSigned) let aptca = assemblyDefinition.CustomAttributes.FirstOrDefault( fun attr -> attr.AttributeType.FullName = typeof.FullName) assemblyDefinition.CustomAttributes.Remove aptca |> ignore assembly.MainModule.AssemblyReferences |> Seq.filter (fun reference -> Set.contains reference.Name assembliesToInstrument) |> Seq.iter (fun reference -> reference.PublicKeyToken <- null )

Slide 107

Slide 107 text

“ “If marked BeforeFieldInit then the type’s initializer method is executed at, or sometime before, first access to any static field defined for that type.” -ECMA-335, Common Language Infrastructure (CLI), Partition I

Slide 108

Slide 108 text

Unicode Original JSON { "a": "bc" } ASCII Bytes 7B 20 22 61 22 20 3A 20 22 62 63 22 20 7D UTF-8 with Byte Order Mark EF BB BF 7B 20 22 61 22 20 3A 20 22 62 63 22 20 7D UTF-16 BE with BOM FE FF 00 7B 00 20 00 22 00 61 00 22 00 20 00 3A 00 20 00 22 00 62 00 63 00 22 00 20 00 7D

Slide 109

Slide 109 text

No content

Slide 110

Slide 110 text

Resources Awesome Fuzzing: A Curated List of Fuzzing Resources https://github.com/secfigo/Awesome-Fuzzing OWASP Fuzzing Page https://www.owasp.org/index.php/Fuzzing MongoDB’s JavaScript Fuzzer http://queue.acm.org/detail.cfm?ref=rss&id=3059007 afl technical details http://lcamtuf.coredump.cx/afl/technical_details.txt afl Help Email List [email protected] Fizil https://github.com/CraigStuntz/Fizil WTF, ACM?

Slide 111

Slide 111 text

Thank You! - Michał Zalewski, for afl documentation - Rehearsal audiences, employees of - Dynamit - Improving - Ineffable Solutions

Slide 112

Slide 112 text

Craig Stuntz [email protected] www.craigstuntz.com @craigstuntz http://www.meetup.com/Papers-We-Love-Columbus/