High Speed Bug Discovery with Fuzzing

High Speed Bug Discovery with Fuzzing

Unit testing is helpful at preventing regressions and guiding design, but it doesn't do a great job of helping you with exploratory testing. How can you find hidden defects in your code without a lot of manual analysis? Fuzzing is a simple but surprisingly effective technique which has been responsible for finding nearly all of the security vulnerabilities uncovered in Flash over the past five years. But it's not just limited to finding security defects! The technique was very successfully used to stabilize the Microsoft document importers for Open Office and check C++ compiler standards compliance. You'll leave this talk knowing when to use fuzzing to test your application, which tools you should use, how to implement a fuzzer from scratch, and when other techniques are a better choice.

56e5c49368a2e0ab999848a8d9e3c116?s=128

Craig Stuntz

May 05, 2017
Tweet

Transcript

  1. 3.
  2. 4.
  3. 5.
  4. 7.

    Spoilers! Why should I care? (because it’s surprisingly effective at

    finding bugs in software) What is it? (a simple, property-based randomized testing technique) When should I use it? (integration testing complex systems with infinite input values) How do I get started? (I’ll suggest a bunch of tools) Should I write my own? (yes, and I have stories!)
  5. 8.
  6. 9.

    400 Crashes, 106 Distinct Security Bugs in Adobe Flash Player

    https://security.googleblog.com/2011/08/fuzzing-at-scale.html
  7. 19.

    Prevent Regressions Bug Discovery Help with Code Design Meets Specifications

    Integration testing Unit testing Formal verification Exploratory testing
  8. 20.

    Prevent Regressions Bug Discovery Help with Code Design Meets Specifications

    Fuzzing Integration testing Unit testing Formal verification Exploratory testing
  9. 21.

    How Many Cases Should We Test? One Only the Most

    Interesting Every Possible Case Unit Testing Fuzzing Formal Verification
  10. 23.
  11. 26.
  12. 27.
  13. 28.

    Corpus A few handwritten examples Fuzzing databases Harvest from test

    suites, defect reports Harvest from public Internet
  14. 30.

    Properties Does it crash? Does it hang? Is the output

    “valid”? Does execution trip an address or memory sanitizer? Does the output match some other system?
  15. 33.

    Getting Started with afl - Compile system under test with

    instrumentation - Place corpus input(s) in a folder - Invoke afl - Wait for bugs https://fuzzing-project.org/tutorial3.html
  16. 35.

    Place Corpus in Folder $ mkdir in $ cd in

    $ cat > foo.json { "a": "bc" } ^D $ cd ..
  17. 36.

    Invoke afl $ afl-fuzz -i in -o out \ my_json_parser

    @@ folder containing corpus “@@“ means “the current test case” system under test findings go here
  18. 37.
  19. 38.
  20. 41.
  21. 42.

    afl Fuzz Strategies Walking bit flips (try flipping each bit

    in input individually) Walking byte flips (try flipping each contiguous set of 8 bits) Simple arithmetic (increment or decrement bytes in the file by certain small values) Known integers (replace bytes with “problematic” 8, 16, and 32 bit integers like 0 and FF) Profile-guided stacked tweaks and test case splicing (magic!) https://lcamtuf.blogspot.com/2014/08/binary-fuzzing-strategies-what-works.html
  22. 43.

    Walking Bit Flip Original 01010101 Flip bit 0 01010100 Flip

    bit 1 01010111 Flip bit 2 01010001 <etc.> Walking 2 Bit Flip Original 01010101 Flip bits 0,1 01010110 Flip bits 2,1 01010011 Flip bits 3,2 01001101 <etc.>
  23. 48.

    Unit Tests Fuzzing Useful For Preventing Regressions, Design Finding New

    Bugs Tests Functions Any Level Test Examples Hand-selected values Corpus + Mutation Execution Time Milliseconds Weeks Magic? No Yes
  24. 53.
  25. 54.
  26. 55.

    STJSON A JSON Parser in Swift 3 compliant with RFC

    7159 STJSON was written along with the article Parsing JSON is a Minefield. Basic usage: var p = STJSONParser(data: data) do { let o = p.parse() } catch let e { print(e) } Instantiation with options: var p = STJSON(data:data, maxParserDepth:1024, options:[.useUnicodeReplacementCharacter]) https://github.com/nst/STJSON https://github.com/CraigStuntz/Fizil/tree/master/StJson
  27. 65.

    How to Get Started with Fuzzing 1. Find a program

    to test 2. Find a fuzzer 3. Find a corpus 4. Choose a property 5. Let it run!
  28. 69.

    $ gzip -c /bin/bash > sample.gz $ while true do

    radamsa sample.gz > fuzzed.gz gzip -dc fuzzed.gz > /dev/null test $? -gt 127 && break done ← Fuzz the corpus ← Execute S.O.T. ← Check a property ← Repeat a lot! https://github.com/aoh/radamsa
  29. 71.

    afl

  30. 73.
  31. 75.

    “ We didn't call it fuzzing back in the 1950s,

    but it was our standard practice to test programs by inputting decks of punch cards taken from the trash. -Gerald M. Weinberg http://secretsofconsulting.blogspot.com/2017/02/fuzz-testing-and-fuzz-history.html
  32. 76.
  33. 77.

    Fuzzing SQLite with afl Start with a single test case:

    create table t1(one smallint); insert into t1 values(1); select * from t1; Add a list of reserved words from documentation Then extract SQL statements from SQLite unit tests (550 files at around 220 bytes each) https://lcamtuf.blogspot.com/2015/04/finding-bugs-in-sqlite-easy-way.html
  34. 87.

    Interesting Stuff I Learned While Writing a Fuzzer - F#

    bitwise operations - How to instrument .NET code - dnSpy is awesome - Same input -> Same code -> Different paths - Strong naming is painful - Unicode is also painful - MemoryMappedFile performance is straight-up awful
  35. 88.

    let jsonNetResult = try JsonConvert.DeserializeObject<obj>(str) |> ignore Success with |

    :? JsonReaderException as jre -> jre.Message |> Error | :? JsonSerializationException as jse -> jse.Message |> Error | :? System.FormatException as fe -> if fe.Message.StartsWith("Invalid hex character”) // hard coded in Json.NET then fe.Message |> Error else reraise() ⃪ T est ⬑ Special case error stuff
  36. 89.

    use proc = new Process() proc.StartInfo.FileName <- executablePath inputMethod.BeforeStart proc

    testCase.Data proc.StartInfo.UseShellExecute <- false proc.StartInfo.RedirectStandardOutput <- true proc.StartInfo.RedirectStandardError <- true proc.StartInfo.EnvironmentVariables.Add(SharedMemory.environmentVariableName, sharedMemoryName) let output = new System.Text.StringBuilder() let err = new System.Text.StringBuilder() proc.OutputDataReceived.Add(fun args -> output.Append(args.Data) |> ignore) proc.ErrorDataReceived.Add (fun args -> err.Append(args.Data) |> ignore) proc.Start() |> ignore inputMethod.AfterStart proc testCase.Data proc.BeginOutputReadLine() proc.BeginErrorReadLine() proc.WaitForExit() let exitCode = proc.ExitCode let crashed = exitCode = WinApi.ClrUnhandledExceptionCode ⃪ Set up ⃪ Read results ⃪ Important bit
  37. 90.

    /// An ordered list of functions to use when starting

    with a single piece of /// example data and producing new examples to try let private allStrategies = [ bitFlip 1 bitFlip 2 bitFlip 4 byteFlip 1 byteFlip 2 byteFlip 4 arith8 arith16 arith32 interest8 interest16 ]
  38. 91.

    let totalBits = bytes.Length * 8 let testCases = seq

    { for bit = 0 to totalBits - flipBits do let newBytes = Array.copy bytes let firstByte = bit / 8 let firstByteMask, secondByteMask = bitMasks(bit, flipBits) let newFirstByte = bytes.[firstByte] ^^^ firstByteMask newBytes.[firstByte] <- newFirstByte let secondByte = firstByte + 1 if secondByteMask <> 0uy && secondByte < bytes.Length then let newSecondByte = bytes.[secondByte] ^^^ secondByteMask newBytes.[secondByte] <- newSecondByte yield newBytes } Fuzz one byte → ^^^ means xor ↓
  39. 92.
  40. 93.
  41. 96.

    private static void F(string arg) { #if MANUAL_INSTRUMENTATION instrument.Trace(29875); #endif

    Console.WriteLine("f"); Console.Error.WriteLine("Error!"); Environment.Exit(1); }
  42. 98.

    let stringify (ob: obj) : string = JsonConvert.SerializeObject(ob) // Method:

    System.String\u0020Program::stringify(System.Object) .body stringify { arg_02_0 [generated] arg_07_0 [generated] nop() arg_02_0 = ldloc(ob) arg_07_0 = call(JsonConvert::SerializeObject, arg_02_0) ret(arg_07_0) }
  43. 99.

    let stringify (ob: obj) : string = JsonConvert.SerializeObject(ob) // Method:

    System.String\u0020Program::stringify(System.Object) .body stringify { arg_02_0 [generated] arg_07_0 [generated] nop() arg_02_0 = ldloc(ob) arg_07_0 = call(JsonConvert::SerializeObject, arg_02_0) ret(arg_07_0) } // Method: System.String\u0020Program::stringify(System.Object) .body stringify { arg_05_0 [generated] arg_0C_0 [generated] arg_11_0 [generated] arg_05_0 = ldc.i4(23831) call(Instrument::Trace, arg_05_0) nop() arg_0C_0 = ldloc(ob) arg_11_0 = call(JsonConvert::SerializeObject, arg_0C_0) ret(arg_11_0) }
  44. 100.
  45. 101.

    let private insertTraceInstruction(ilProcessor: ILProcessor, before: Instruction, state) = let compileTimeRandom

    = state.Random.Next(0, UInt16.MaxValue |> Convert.ToInt32) let ldArg = ilProcessor.Create(OpCodes.Ldc_I4, compileTimeRandom) let callTrace = ilProcessor.Create(OpCodes.Call, state.Trace) ilProcessor.InsertBefore(before, ldArg) ilProcessor.InsertAfter (ldArg, callTrace) This margin is too narrow to contain a try/finally example, so see: https://goo.gl/W4y7JH
  46. 102.

    let private insertTraceInstruction(ilProcessor: ILProcessor, before: Instruction, state) = let compileTimeRandom

    = state.Random.Next(0, UInt16.MaxValue |> Convert.ToInt32) let ldArg = ilProcessor.Create(OpCodes.Ldc_I4, compileTimeRandom) let callTrace = ilProcessor.Create(OpCodes.Call, state.Trace) ilProcessor.InsertBefore(before, ldArg) ilProcessor.InsertAfter (ldArg, callTrace) This margin is too narrow to contain a try/finally example, so see: https://goo.gl/W4y7JH
  47. 103.

    let private insertTraceInstruction(ilProcessor: ILProcessor, before: Instruction, state) = let compileTimeRandom

    = state.Random.Next(0, UInt16.MaxValue |> Convert.ToInt32) let ldArg = ilProcessor.Create(OpCodes.Ldc_I4, compileTimeRandom) let callTrace = ilProcessor.Create(OpCodes.Call, state.Trace) ilProcessor.InsertBefore(before, ldArg) ilProcessor.InsertAfter (ldArg, callTrace) This margin is too narrow to contain a try/finally example, so see: https://goo.gl/W4y7JH
  48. 104.

    let private insertTraceInstruction(ilProcessor: ILProcessor, before: Instruction, state) = let compileTimeRandom

    = state.Random.Next(0, UInt16.MaxValue |> Convert.ToInt32) let ldArg = ilProcessor.Create(OpCodes.Ldc_I4, compileTimeRandom) let callTrace = ilProcessor.Create(OpCodes.Call, state.Trace) ilProcessor.InsertBefore(before, ldArg) ilProcessor.InsertAfter (ldArg, callTrace) This margin is too narrow to contain a try/finally example, so see: https://goo.gl/W4y7JH
  49. 105.
  50. 115.
  51. 116.
  52. 118.

    Standard Rejects, Json.NET Accepts Value [,,,] Standard Says A JSON

    value MUST be an object, array, number, or string, or one of the following three literal names: false null true Json.NET [null, null, null, null]
  53. 119.
  54. 120.

    let private removeStrongName (assemblyDefinition : AssemblyDefinition) = let name =

    assemblyDefinition.Name; name.HasPublicKey <- false; name.PublicKey <- Array.empty; assemblyDefinition.Modules |> Seq.iter ( fun moduleDefinition -> moduleDefinition.Attributes <- moduleDefinition.Attributes &&& ~~~ModuleAttributes.StrongNameSigned) let aptca = assemblyDefinition.CustomAttributes.FirstOrDefault( fun attr -> attr.AttributeType.FullName = typeof<System.Security.AllowPartiallyTrustedCallersAttribute>.FullName) assemblyDefinition.CustomAttributes.Remove aptca |> ignore assembly.MainModule.AssemblyReferences |> Seq.filter (fun reference -> Set.contains reference.Name assembliesToInstrument) |> Seq.iter (fun reference -> reference.PublicKeyToken <- null )
  55. 121.

    “ “If marked BeforeFieldInit then the type’s initializer method is

    executed at, or sometime before, first access to any static field defined for that type.” -ECMA-335, Common Language Infrastructure (CLI), Partition I
  56. 122.

    Unicode Original JSON { "a": "bc" } ASCII Bytes 7B

    20 22 61 22 20 3A 20 22 62 63 22 20 7D UTF-8 with Byte Order Mark EF BB BF 7B 20 22 61 22 20 3A 20 22 62 63 22 20 7D UTF-16 BE with BOM FE FF 00 7B 00 20 00 22 00 61 00 22 00 20 00 3A 00 20 00 22 00 62 00 63 00 22 00 20 00 7D
  57. 123.
  58. 125.

    Thank You! - Michał Zalewski, for afl documentation - Rehearsal

    audiences, employees of - Dynamit - Improving - Ineffable Solutions