Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Systematic Assessment of Fuzzers using Mutation Analysis

Systematic Assessment of Fuzzers using Mutation Analysis

USENIX Security 2023

Rahul Gopinath

August 11, 2023
Tweet

More Decks by Rahul Gopinath

Other Decks in Research

Transcript

  1. Systematic Assessment of Fuzzers
    using Mutation Analysis
    Philipp Görz1 @[email protected]
    Björn Mathis1 @bjrnmath
    Keno Hassler1
    Emre Güler2 @emrexgueler
    Thorsten Holz1 @thorstenholz
    Andreas Zeller1 @andreaszeller
    Rahul Gopinath3 @[email protected]
    1 CISPA Helmholtz Center for Information Security, Germany 2 Ruhr-University Bochum, Germany 3 University of Sydney, Australia

    View full-size slide

  2. Fuzz Testing / Fuzzing
    https://lcamtuf.coredump.cx/afl/

    View full-size slide

  3. Evaluating Fuzzers
    Com
    parable
    Unbiased
    Custom
    Subjects
    Guaranteed Faults

    3 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  4. Evaluating Fuzzers - Coverage?
    https://github.com/gcovr/gcovr

    View full-size slide

  5. Evaluating Fuzzers
    Com
    parable
    Unbiased
    Custom
    Subjects
    Guaranteed Faults
    5 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  6. Evaluating Fuzzers
    Com
    parable
    Unbiased
    Custom
    Subjects
    Guaranteed Faults
    Coverage
    5 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  7. Evaluating Fuzzers
    Com
    parable
    Unbiased
    Custom
    Subjects
    Guaranteed Faults
    Coverage ✔
    5 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  8. Evaluating Fuzzers
    Com
    parable
    Unbiased
    Custom
    Subjects
    Guaranteed Faults
    Coverage ✔ — — —
    5 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  9. Evaluating Fuzzers - Finding New Bugs?
    https://www.cve.org/
    6 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  10. Evaluating Fuzzers
    Com
    parable
    Unbiased
    Custom
    Subjects
    Guaranteed Faults
    Coverage ✔ — — —
    7 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  11. Evaluating Fuzzers
    Com
    parable
    Unbiased
    Custom
    Subjects
    Guaranteed Faults
    Coverage ✔ — — —
    New Bugs
    7 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  12. Evaluating Fuzzers
    Com
    parable
    Unbiased
    Custom
    Subjects
    Guaranteed Faults
    Coverage ✔ — — —
    New Bugs ✘
    7 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  13. Evaluating Fuzzers
    Com
    parable
    Unbiased
    Custom
    Subjects
    Guaranteed Faults
    Coverage ✔ — — —
    New Bugs ✘ ✘
    7 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  14. Evaluating Fuzzers
    Com
    parable
    Unbiased
    Custom
    Subjects
    Guaranteed Faults
    Coverage ✔ — — —
    New Bugs ✘ ✘ ✔
    7 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  15. Evaluating Fuzzers
    Com
    parable
    Unbiased
    Custom
    Subjects
    Guaranteed Faults
    Coverage ✔ — — —
    New Bugs ✘ ✘ ✔ —
    7 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  16. Evaluating Fuzzers - Refinding Known Bugs?

    View full-size slide

  17. Evaluating Fuzzers - Refinding Known Bugs?
    https://hexhive.epfl.ch/magma/

    View full-size slide

  18. Evaluating Fuzzers
    Com
    parable
    Unbiased
    Custom
    Subjects
    Guaranteed Faults
    Coverage ✔ — — —
    New Bugs ✘ ✘ ✔ —
    9 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  19. Evaluating Fuzzers
    Com
    parable
    Unbiased
    Custom
    Subjects
    Guaranteed Faults
    Coverage ✔ — — —
    New Bugs ✘ ✘ ✔ —
    Known Bugs
    9 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  20. Evaluating Fuzzers
    Com
    parable
    Unbiased
    Custom
    Subjects
    Guaranteed Faults
    Coverage ✔ — — —
    New Bugs ✘ ✘ ✔ —
    Known Bugs ✔
    9 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  21. Evaluating Fuzzers
    Com
    parable
    Unbiased
    Custom
    Subjects
    Guaranteed Faults
    Coverage ✔ — — —
    New Bugs ✘ ✘ ✔ —
    Known Bugs ✔ ✘
    9 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  22. Evaluating Fuzzers
    Com
    parable
    Unbiased
    Custom
    Subjects
    Guaranteed Faults
    Coverage ✔ — — —
    New Bugs ✘ ✘ ✔ —
    Known Bugs ✔ ✘ ✘
    9 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  23. Evaluating Fuzzers
    Com
    parable
    Unbiased
    Custom
    Subjects
    Guaranteed Faults
    Coverage ✔ — — —
    New Bugs ✘ ✘ ✔ —
    Known Bugs ✔ ✘ ✘ ✔
    9 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  24. Mutation Testing / Mutation Analysis
    Fuzzing Your Test Suite
    10 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  25. Mutation Testing / Mutation Analysis
    unsigned int len = message_length(msg);
    if (len < MAX_BUF_LEN) {
    copy_message(msg);
    } else {
    // Invalid length , handle error
    }
    11 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  26. Mutation Testing / Mutation Analysis
    ① unsigned int len = message_length(msg);
    if (len < MAX_BUF_LEN) {
    copy_message(msg);
    } else {
    // Invalid length , handle error
    }
    12 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  27. Mutation Testing / Mutation Analysis
    ① unsigned int len = message_length(msg);
    if (len ② < >= MAX_BUF_LEN) {
    copy_message(msg);
    } else {
    // Invalid length , handle error
    }
    13 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  28. Mutation Testing / Mutation Analysis
    ① unsigned int len = message_length(msg);
    if (len ② < >= MAX_BUF_LEN ③ + 16) {
    copy_message(msg);
    } else {
    // Invalid length , handle error
    }
    14 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  29. Mutation Testing / Mutation Analysis
    ① unsigned int len = message_length(msg);
    if (len ② < >= MAX_BUF_LEN ③ + 16) {
    copy_message(msg);
    } else {
    // Invalid length , handle error
    }

    View full-size slide

  30. Mutation Testing / Mutation Analysis
    ① unsigned int len = message_length(msg);
    if (len ② < >= MAX_BUF_LEN ③ + 16) {
    copy_message(msg);
    } else {
    // Invalid length , handle error
    }

    View full-size slide

  31. Mutation Testing / Mutation Analysis
    ① unsigned int len = message_length(msg);
    if (len ② < >= MAX_BUF_LEN ③ + 16) {
    copy_message(msg);
    } else {
    // Invalid length , handle error
    }


    View full-size slide

  32. Mutation Testing / Mutation Analysis
    ① unsigned int len = message_length(msg);
    if (len ② < >= MAX_BUF_LEN ③ + 16) {
    copy_message(msg);
    } else {
    // Invalid length , handle error
    }


    View full-size slide

  33. Mutation Testing / Mutation Analysis
    ① unsigned int len = message_length(msg);
    if (len ② < >= MAX_BUF_LEN ③ + 16) {
    copy_message(msg);
    } else {
    // Invalid length , handle error
    }



    View full-size slide

  34. Mutation Testing / Mutation Analysis
    ① unsigned int len = message_length(msg);
    if (len ② < >= MAX_BUF_LEN ③ + 16) {
    copy_message(msg);
    } else {
    // Invalid length , handle error
    }


    ✔ ?
    14 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  35. Evaluating Fuzzers
    Com
    parable
    Unbiased
    Custom
    Subjects
    Guaranteed Faults
    Coverage ✔ — — —
    New Bugs ✘ ✘ ✔ —
    Known Bugs ✔ ✘ ✘ ✔
    15 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  36. Evaluating Fuzzers
    Com
    parable
    Unbiased
    Custom
    Subjects
    Guaranteed Faults
    Coverage ✔ — — —
    New Bugs ✘ ✘ ✔ —
    Known Bugs ✔ ✘ ✘ ✔
    Mutation Testing
    15 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  37. Evaluating Fuzzers
    Com
    parable
    Unbiased
    Custom
    Subjects
    Guaranteed Faults
    Coverage ✔ — — —
    New Bugs ✘ ✘ ✔ —
    Known Bugs ✔ ✘ ✘ ✔
    Mutation Testing ✔
    15 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  38. Evaluating Fuzzers
    Com
    parable
    Unbiased
    Custom
    Subjects
    Guaranteed Faults
    Coverage ✔ — — —
    New Bugs ✘ ✘ ✔ —
    Known Bugs ✔ ✘ ✘ ✔
    Mutation Testing ✔ ✔
    15 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  39. Evaluating Fuzzers
    Com
    parable
    Unbiased
    Custom
    Subjects
    Guaranteed Faults
    Coverage ✔ — — —
    New Bugs ✘ ✘ ✔ —
    Known Bugs ✔ ✘ ✘ ✔
    Mutation Testing ✔ ✔ ✔
    15 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  40. Evaluating Fuzzers
    Com
    parable
    Unbiased
    Custom
    Subjects
    Guaranteed Faults
    Coverage ✔ — — —
    New Bugs ✘ ✘ ✔ —
    Known Bugs ✔ ✘ ✘ ✔
    Mutation Testing ✔ ✔ ✔ ✘
    15 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  41. What’s the Problem?
    16 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  42. What’s the Problem?
    • Computationally Expensive!
    16 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  43. What’s the Problem?
    • Computationally Expensive!
    • Mutation Testing: Execute Test Generator (Fuzzer) for each Mutation
    16 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  44. What’s the Problem?
    • Computationally Expensive!
    • Mutation Testing: Execute Test Generator (Fuzzer) for each Mutation
    • Fuzzing: The More Executions the Better
    16 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  45. Contributions
    17 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  46. Contributions
    • Reduce Computational Costs
    17 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  47. Contributions
    • Reduce Computational Costs
    • Split Phases
    17 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  48. Contributions
    • Reduce Computational Costs
    • Split Phases
    • Coverage Fuzzing
    17 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  49. Contributions
    • Reduce Computational Costs
    • Split Phases
    • Coverage Fuzzing
    • Mutation Fuzzing
    17 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  50. Contributions
    • Reduce Computational Costs
    • Split Phases
    • Coverage Fuzzing
    • Mutation Fuzzing
    • Supermutants
    17 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  51. Contributions
    • Reduce Computational Costs
    • Split Phases
    • Coverage Fuzzing
    • Mutation Fuzzing
    • Supermutants
    • Evaluate Multiple Mutations
    with one Fuzzing Run
    17 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  52. Contributions
    • Reduce Computational Costs
    • Split Phases
    • Coverage Fuzzing
    • Mutation Fuzzing
    • Supermutants
    • Evaluate Multiple Mutations
    with one Fuzzing Run
    • Mutation Operators
    17 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  53. Contributions
    • Reduce Computational Costs
    • Split Phases
    • Coverage Fuzzing
    • Mutation Fuzzing
    • Supermutants
    • Evaluate Multiple Mutations
    with one Fuzzing Run
    • Mutation Operators
    • Traditional Operators
    17 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  54. Contributions
    • Reduce Computational Costs
    • Split Phases
    • Coverage Fuzzing
    • Mutation Fuzzing
    • Supermutants
    • Evaluate Multiple Mutations
    with one Fuzzing Run
    • Mutation Operators
    • Traditional Operators
    • Security Specific Operators
    17 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  55. Results
    • Coverage Accounts for most Mutants Detected
    18 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  56. Results
    • Coverage Accounts for most Mutants Detected
    • ASAN Moderately Increases Number of Killed Mutants
    18 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  57. Results
    • Coverage Accounts for most Mutants Detected
    • ASAN Moderately Increases Number of Killed Mutants
    • Mutations are Coupled to Real Faults
    18 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  58. Code
    Code is Publicly Available!
    Interested? Talk to Us!
    SBFT’24?!
    github.com/CISPA-SysSec/mua_fuzzer_bench

    View full-size slide

  59. Mutation Testing / Mutation Analysis
    Fuzzing Your Test Suite
    10 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis
    Evaluating Fuzzers
    Com
    parable
    Unbiased
    Custom
    Subjects
    Guaranteed Faults
    Coverage ✔ — — —
    New Bugs ✘ ✘ ✔ —
    Known Bugs ✔ ✘ ✘ ✔
    Mutation Testing ✔ ✔ ✔ ✘
    12 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis
    Contributions
    • Reduce Computational Costs
    • Split Phases
    • Coverage Fuzzing
    • Mutation Fuzzing
    • Supermutants
    • Evaluate Multiple Mutations
    with one Fuzzing Run
    • Mutation Operators
    • Traditional Operators
    • Security Specific Operators
    14 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis
    Code
    Code is Publicly Available!
    Interested? Talk to Us!
    SBFT’24?!
    github.com/CISPA-SysSec/mua_fuzzer_bench
    20 USENIX — Systematic Assessment of Fuzzers using Mutation Analysis

    View full-size slide

  60. Systematic Assessment of Fuzzers
    using Mutation Analysis
    Philipp Görz1 @[email protected]
    Björn Mathis1 @bjrnmath
    Keno Hassler1
    Emre Güler2 @emrexgueler
    Thorsten Holz1 @thorstenholz
    Andreas Zeller1 @andreaszeller
    Rahul Gopinath3 @[email protected]
    1 CISPA Helmholtz Center for Information Security, Germany 2 Ruhr-University Bochum, Germany 3 University of Sydney, Australia

    View full-size slide

  61. Compilation Procedure
    Mutator
    Base
    Compiler
    Unmutated
    Executable
    Mutated
    Executable
    Fuzzer
    Compiler
    Subject
    (bitcode file)
    Instrumented
    Mutated
    Exectuable
    Mutation
    Finder
    Location
    Executable
    Mutation
    IDs
    Mutation IDs
    for a Supermutant
    Result of
    Subject
    Result of
    Supermutant
    Supermutant
    (bitcode file)

    View full-size slide

  62. Checking Procedure
    Benchmark Manager
    Crashing
    Input
    Mutation
    killed?
    Seeds Unmutated
    Executable
    Mutated
    Executable
    Fuzzer(s)
    Mutation
    covered?
    Instrumented
    Mutated
    Exectuable
    1. Check if Seeds
    (after Phase I)
    already kill mutation(s)
    4. Check if found
    Crashing Input
    kills Mutant
    Run input to check that
    crash does not happen
    in unmutated executable
    Run input to check if
    crash can be confirmed
    3. Fuzz using the fuzzer
    respective executable
    Run input to get
    covered mutations
    2. Use Seeds to start
    Fuzzer (each Fuzzer is
    initialized with
    their respective seeds
    after Phase I)

    View full-size slide

  63. ASan Percentages
    2.7%
    24.7%
    24.7%
    0.0%
    5.5%
    21.9%
    21.9%
    0.0%
    5.6%
    21.1%
    21.1%
    0.0%
    16.2%
    32.3%
    32.3%
    1.8%
    16.3%
    32.1%
    32.1%
    1.8%
    18.0%
    31.5%
    31.5%
    0.9%
    7.0%
    22.2%
    22.2%
    0.6%
    7.5%
    21.9%
    21.9%
    0.5%
    7.5%
    22.4%
    22.4%
    0.6%
    6.7%
    23.3%
    23.3%
    3.0%
    7.4%
    25.0%
    25.0%
    2.5%
    7.3%
    25.4%
    25.4%
    2.0%
    12.4%
    18.4%
    18.4%
    0.6%
    12.6%
    18.3%
    18.3%
    0.6%
    12.1%
    18.5%
    18.5%
    0.6%
    10.4%
    35.8%
    35.8%
    1.7%
    10.4%
    35.0%
    35.0%
    1.8%
    10.0%
    35.4%
    35.4%
    1.2%
    3.7%
    17.1%
    17.1%
    2.9%
    3.6%
    17.2%
    17.2%
    3.0%
    3.0%
    16.8%
    16.8%
    1.1%
    cares_name cares_parse_reply curl guetzli libevent re2 woff2_new
    aflpp honggfuzz libfuzzer
    default asan default asan default asan default asan default asan default asan default asan
    0%
    10%
    20%
    30%
    40%
    0%
    10%
    20%
    30%
    40%
    0%
    10%
    20%
    30%
    40%
    Percentage of Covered Mutations that are Killed
    Found By asan default both

    View full-size slide

  64. Supermutants Computational Reduction
    Subject #Mutants #Supermutants Factor
    Curl 29,118 5,804 5.02
    Guetzli 22,961 13,040 1.76
    Woff2 (New) 40,914 5,930 6.90
    Cares (Name) 4,822 550 8.77
    Cares (Parse Reply) 4,822 1,288 3.74
    libevent 17,234 864 19.95
    re2 21,407 9,670 2.21
    Sum 141,278 37,146 3.80

    View full-size slide

  65. Wallclock Time
    CPU (Years) 4 Servers (Days)
    Seed Collection 1.99 3.50
    Default 14.37 25.22
    Seed + Default 16.36 28.72
    ASAN 15.16 26.61
    24 Hours Runs 7.42 13.02
    Sum 38.95 Years 68.34 Days
    Four servers: Intel Xeon Gold 6230R CPU (52 cores) and 188 GB RAM.
    Note that evaluating a single fuzzer takes 4.09 CPU years with our
    chosen subjects ("Seed + Default" / #Fuzzers).

    View full-size slide

  66. 24 Hour Runs
    Prog Total AFL AFL++ libFuzzer Honggfuzz
    re2 104 0 0 0 0
    Woff2 (New) 104 0 0 0 1
    Curl 104 0 0 1 0
    Guetzli 104 0 0 0 1
    Libevent 104 0 0 0 0
    Cares (Name) 66 0 0 0 0
    Cares (Parse Reply) 104 0 0 0 0
    Mutants killed during 24 hour runs on 104 stubborn mutants for each
    subject using ASAN.

    View full-size slide

  67. Not Independent Mutants
    Program afl aflpp honggfuzz libfuzzer
    Curl 4,850 5,836 4,851 3,852
    Guetzli 10 24 16 0
    Libevent 0 2 0 0
    re2 39 66 37 47
    Woff2 (New) 26 46 56 48
    Cares (Name) 4 0 0 0
    Cares (Parse Reply) 2 4 4 0
    Number of mutants that were covered together with other mutants (i.e.,
    mutants wrongly thought independent).

    View full-size slide