Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Input Languages for 
Effective and Focused Fuzzing

Input Languages for 
Effective and Focused Fuzzing

FuzzCon Europe 2021

Rahul Gopinath

October 21, 2021
Tweet

More Decks by Rahul Gopinath

Other Decks in Research

Transcript

  1. Rahul Gopinath
    https://rahul.gopinath.org
    @_rahulgopinath
    Rafael Dutra
    https://cispa.de/en/people/c01radu/
    1
    Input Languages for

    Effective and Focused Fuzzing

    View full-size slide

  2. Rahul Gopinath
    https://rahul.gopinath.org
    @_rahulgopinath
    Rafael Dutra
    https://cispa.de/en/people/c01radu/
    FuzzCon Europe 2021
    1
    Input Languages for

    Effective and Focused Fuzzing

    View full-size slide

  3. 2
    2
    Feedback-Directed Fuzzers: Challenges
    file.png
    AFL readpng
    PNG
    corpus
    save inputs that increase coverage

    View full-size slide

  4. 3
    3
    Feedback-Directed Fuzzers: Challenges
    file.png
    AFL readpng
    PNG
    corpus
    save inputs that increase coverage
    Most mutations
    make the file
    invalid
    Hard to find
    deeper
    semantic bugs

    View full-size slide

  5. 4
    4
    Feedback-Directed Fuzzers: Challenges
    file.png
    AFL readpng
    PNG
    corpus
    save inputs that increase coverage
    Highly
    dependent on
    initial corpus

    View full-size slide

  6. 5
    5
    Feedback-Directed Fuzzers: Challenges
    file.png
    AFL readpng
    PNG
    corpus
    save inputs that increase coverage
    Require
    feedback from
    the program

    View full-size slide

  7. 6
    6
    What About a Format-specic Fuzzer?

    View full-size slide

  8. 7
    7
    How to Generate a PNG le?

    View full-size slide

  9. 8
    8
    Read the Spec?

    View full-size slide

  10. 9
    9
    How Can We Even Inspect a PNG?

    View full-size slide

  11. 10
    10
    How Can We Even Inspect a PNG?
    010 Editor

    View full-size slide

  12. 11
    11
    Binary Templates
    typedef struct {
    uint32 length;
    char type[4];
    ubyte data[length];
    uint32 crc;
    } PNG_CHUNK;

    View full-size slide

  13. 12
    12
    Binary Templates
    typedef struct {
    uint32 length;
    char type[4];
    ubyte data[length];
    uint32 crc;
    } PNG_CHUNK;

    View full-size slide

  14. 13
    13
    Binary Templates
    typedef struct {
    uint32 length;
    local int64 start = FTell();
    char type[4];
    ubyte data[length];
    local int64 end = FTell();
    local uint32 crc_calc = Checksum(CHECKSUM_CRC32, start, end-start);
    uint32 crc;
    if (crc != crc_calc)
    Warning("Bad CRC %08x; expected: %08x", crc, crc_calc);
    } PNG_CHUNK;

    View full-size slide

  15. 14
    14
    Binary Templates
    typedef struct {
    uint32 length;
    local int64 start = FTell();
    char type[4];
    ubyte data[length];
    local int64 end = FTell();
    local uint32 crc_calc = Checksum(CHECKSUM_CRC32, start, end-start);
    uint32 crc;
    if (crc != crc_calc)
    Warning("Bad CRC %08x; expected: %08x", crc, crc_calc);
    } PNG_CHUNK;

    View full-size slide

  16. 15
    15
    Binary Templates
    typedef struct {
    uint32 length;
    local int64 start = FTell();
    char type[4];
    if (type == "IHDR")
    PNG_CHUNK_IHDR ihdr;
    else if (type == "PLTE")
    PNG_CHUNK_PLTE plte(length);
    /* … */
    local int64 end = FTell();
    local uint32 crc_calc = Checksum(CHECKSUM_CRC32, start, end-start);
    uint32 crc;
    if (crc != crc_calc)
    Warning("Bad CRC %08x; expected: %08x", crc, crc_calc);
    } PNG_CHUNK;

    View full-size slide

  17. 16
    16
    200+ Binary Templates Available

    View full-size slide

  18. 17
    17
    Binary Templates as Generators?

    View full-size slide

  19. 18
    18
    Genera.ng Valid Files
    typedef struct {
    uint32 length;
    local int64 start = FTell();
    char type[4];
    if (type == "IHDR")
    PNG_CHUNK_IHDR ihdr;
    else if (type == "PLTE")
    PNG_CHUNK_PLTE plte(length);
    /* … */
    local int64 end = FTell();
    local uint32 crc_calc = Checksum(CHECKSUM_CRC32, start, end-start);
    uint32 crc;
    if (crc != crc_calc)
    Warning("Bad CRC %08x; expected: %08x", crc, crc_calc);
    } PNG_CHUNK;

    View full-size slide

  20. 19
    19
    Genera.ng Valid Files
    typedef struct {
    uint32 length;
    local int64 start = FTell();
    char type[4];
    if (type == "IHDR")
    PNG_CHUNK_IHDR ihdr;
    else if (type == "PLTE")
    PNG_CHUNK_PLTE plte(length);
    /* … */
    local int64 end = FTell();
    local uint32 crc_calc = Checksum(CHECKSUM_CRC32, start, end-start);
    uint32 crc = { crc_calc };
    if (crc != crc_calc)
    Warning("Bad CRC %08x; expected: %08x", crc, crc_calc);
    } PNG_CHUNK;

    View full-size slide

  21. 20
    20
    Genera.ng Valid Files
    typedef struct {
    uint32 length;
    local int64 start = FTell();
    char type[4];
    if (type == "IHDR")
    PNG_CHUNK_IHDR ihdr;
    else if (type == "PLTE")
    PNG_CHUNK_PLTE plte(length);
    /* … */
    local int64 end = FTell();
    local uint32 crc_calc = Checksum(CHECKSUM_CRC32, start, end-start);
    uint32 crc = { crc_calc };
    if (crc != crc_calc)
    Warning("Bad CRC %08x; expected: %08x", crc, crc_calc);
    } PNG_CHUNK;

    View full-size slide

  22. 21
    21
    Genera.ng Valid Files
    typedef struct {
    uint32 length ;
    local int64 start = FTell();
    char type[4];
    if (type == "IHDR")
    PNG_CHUNK_IHDR ihdr;
    else if (type == "PLTE")
    PNG_CHUNK_PLTE plte(length);
    /* … */
    local int64 end = FTell(); /* ← Fix length here */
    local uint32 crc_calc = Checksum(CHECKSUM_CRC32, start, end-start);
    uint32 crc = { crc_calc };
    if (crc != crc_calc)
    Warning("Bad CRC %08x; expected: %08x", crc, crc_calc);
    } PNG_CHUNK;

    View full-size slide

  23. 22
    22
    Genera.ng Valid Files
    typedef struct {
    uint32 length ;
    local int64 start = FTell();
    char type[4];
    if (type == "IHDR")
    PNG_CHUNK_IHDR ihdr;
    else if (type == "PLTE")
    PNG_CHUNK_PLTE plte(length);
    /* … */
    local int64 end = FTell(); /* ← Fix length here */
    local uint32 crc_calc = Checksum(CHECKSUM_CRC32, start, end-start);
    uint32 crc = { crc_calc };
    if (crc != crc_calc)
    Warning("Bad CRC %08x; expected: %08x", crc, crc_calc);
    } PNG_CHUNK;

    View full-size slide

  24. 23
    23
    Genera.ng Valid Files
    typedef struct {
    uint32 length ;
    local int64 start = FTell();
    char type[4];
    if (type == "IHDR")
    PNG_CHUNK_IHDR ihdr;
    else if (type == "PLTE")
    PNG_CHUNK_PLTE plte(length);
    /* … */
    local int64 end = FTell(); /* ← Fix length here */
    local uint32 crc_calc = Checksum(CHECKSUM_CRC32, start, end-start);
    uint32 crc = { crc_calc };
    if (crc != crc_calc)
    Warning("Bad CRC %08x; expected: %08x", crc, crc_calc);
    } PNG_CHUNK;

    View full-size slide

  25. 24
    24
    Genera.ng Valid Files
    typedef struct {
    uint32 length ;
    local int64 start = FTell();
    char type[4];
    if (type == "IHDR")
    PNG_CHUNK_IHDR ihdr;
    else if (type == "PLTE")
    PNG_CHUNK_PLTE plte(length);
    /* … */
    local int64 end = FTell(); /* ← Fix length here */
    local uint32 crc_calc = Checksum(CHECKSUM_CRC32, start, end-start);
    uint32 crc = { crc_calc };
    if (crc != crc_calc)
    Warning("Bad CRC %08x; expected: %08x", crc, crc_calc);
    } PNG_CHUNK;
    IHDR PLTE IDAT IEND

    View full-size slide

  26. 25
    25
    FormatFuzzer

    View full-size slide

  27. 26
    26
    FormatFuzzer
    decision seed file.png
    generate
    mutate
    parse
    png-fuzzer
    png.bt
    C++

    View full-size slide

  28. 27
    27
    Decision Seeds
    89 P N G 0d 0a 1a 0a
    00 00 00 00
    Decision Seed Generated PNG File

    View full-size slide

  29. 28
    28
    Decision Seeds
    89 P N G 0d 0a 1a 0a 00 00 00 0d I H D R
    00 00 00 01 00 00 00 01 08 02 00 00 00 90 77 53
    de
    00 00 00 00 00 00 00 0c 00 00 00 00 00 00 08 00
    01 00 00 00 00 00 00 00 00
    Decision Seed Generated PNG File

    View full-size slide

  30. 29
    29
    Decision Seeds
    89 P N G 0d 0a 1a 0a 00 00 00 0d I H D R
    00 00 00 01 00 00 00 01 08 02 00 00 00 90 77 53
    de 00 00 00 0c I D A T 78 da 63 f8 cf c0 00
    00 03 01 01 00 f7 03 41 43
    00 00 00 00 00 00 00 0c 00 00 00 00 00 00 08 00
    01 00 00 00 00 00 00 00 00 00 00 01 00 0b 00 29
    24 21 34 00 ff 00 00 08 00 00
    Decision Seed Generated PNG File

    View full-size slide

  31. 30
    30
    Decision Seeds
    89 P N G 0d 0a 1a 0a 00 00 00 0d I H D R
    00 00 00 01 00 00 00 01 08 02 00 00 00 90 77 53
    de 00 00 00 0c I D A T 78 da 63 f8 cf c0 00
    00 03 01 01 00 f7 03 41 43 00 00 00 00 I E N
    D ae 42 60 82
    00 00 00 00 00 00 00 0c 00 00 00 00 00 00 08 00
    01 00 00 00 00 00 00 00 00 00 00 01 00 0b 00 29
    24 21 34 00 ff 00 00 08 00 00 00 00 7f 00 00 00
    00 00 00 00
    Decision Seed Generated PNG File

    View full-size slide

  32. 31
    31
    Smart Muta.ons

    View full-size slide

  33. 32
    32
    Smart Muta.ons Respect Context
    typedef struct {
    uint32 length ;
    local int64 start = FTell();
    char type[4];
    if (type == "IHDR")
    PNG_CHUNK_IHDR ihdr;
    else if (type == "PLTE")
    PNG_CHUNK_PLTE plte(length);
    /* … */
    local int64 end = FTell(); /* ← Fix length here */
    local uint32 crc_calc = Checksum(CHECKSUM_CRC32, start, end-start);
    uint32 crc = { crc_calc };
    if (crc != crc_calc)
    Warning("Bad CRC %08x; expected: %08x", crc, crc_calc);
    } PNG_CHUNK;

    View full-size slide

  34. 33
    33
    Smart Muta.ons Respect Context
    typedef struct {
    uint32 length ;
    local int64 start = FTell();
    char type[4];
    if (type == "IHDR")
    PNG_CHUNK_IHDR ihdr;
    else if (type == "PLTE")
    PNG_CHUNK_PLTE plte(length);
    /* … */
    local int64 end = FTell(); /* ← Fix length here */
    local uint32 crc_calc = Checksum(CHECKSUM_CRC32, start, end-start);
    uint32 crc = { crc_calc };
    if (crc != crc_calc)
    Warning("Bad CRC %08x; expected: %08x", crc, crc_calc);
    } PNG_CHUNK;

    View full-size slide

  35. 34
    34
    How to use FormatFuzzer?

    View full-size slide

  36. 35
    35
    Fuzzing Strategies: Black-box Genera.on
    decision seed file.png
    generate
    mutate
    parse
    png-fuzzer
    /dev/urandom
    readpng
    FFGen

    View full-size slide

  37. 36
    36
    Fuzzing Strategies: Black-box Muta.on
    decision seed
    corpus
    PNG
    corpus
    generate
    mutate
    parse
    png-fuzzer
    FFMut

    View full-size slide

  38. 37
    37
    Fuzzing Strategies: Black-box Muta.on
    decision seed
    corpus
    file.png
    generate
    mutate
    parse
    png-fuzzer
    FFMut
    readpng

    View full-size slide

  39. 38
    38
    Feedback-Directed Fuzzers
    file.png
    AFL
    AFL
    readpng
    PNG
    corpus
    save inputs that increase coverage

    View full-size slide

  40. 39
    39
    Fuzzing Strategies: Generator-based Fuzzing
    AFL
    AFL+FFGen
    readpng
    save inputs that increase coverage
    decision seed
    corpus
    decision seed
    generate
    mutate
    parse
    png-fuzzer file.png

    View full-size slide

  41. 40
    40
    Fuzzing Strategies: Muta.on-based Fuzzing
    file.png
    AFL
    AFL+FFMut
    readpng
    PNG
    corpus
    save inputs that increase coverage
    generate
    mutate
    parse
    png-fuzzer

    View full-size slide

  42. 41
    41
    Evalua.on
    • Effort: most lines in the binary templates
    remain unchanged
    • Speed: ~7000 files/s generated or parsed
    • Success: 97% generations are successful
    • Accuracy: 76% of generated files are valid
    (82% without evil decisions)

    View full-size slide

  43. 42
    42
    Fuzzing Results (Line Coverage %)
    PNG JPG GIF MIDI MP4 ZIP PCAP AVI BMP
    FFGen 22.3 24.2 68.7 12.3 5.6 33.7 11.5 5.6 27.8
    FFMut 22.5 24.1 70.7 10.4 6.9 34.8 7.8 6.7 27.8
    AFL 17.6 29.0 73.3 11.7 10.3 36.1 24.0 9.3 30.7
    AFL+FFGen 23.6 26.7 71.6 11.9 9.0 36.5 21.1 9.1 27.9
    AFL+FFMut 26.0 33.1 73.2 12.2 10.2 37.1 23.4 10.1 30.7
    AFLSmart 18.0 29.4 12.1 10.8 36.2 24.1 10.7

    View full-size slide

  44. 43
    43
    Bugs Found
    • FFmpeg: 8 distinct bugs already fixed by
    FFmpeg developers

    Most are segmentation faults related to allocation
    • TiMidity: 19 distinct memory errors found

    View full-size slide

  45. 44
    44
    FormatFuzzer
    • FormatFuzzer 1.0 released today
    ● Includes integration with AFL++ 2.60c
    • Project page: https://uds-se.github.io/FormatFuzzer/
    • Source: https://github.com/uds-se/FormatFuzzer
    • Preprint: http://arxiv.org/abs/2109.11277

    View full-size slide

  46. 45
    Current Research

    View full-size slide

  47. 45
    Current Research
    Mining Input

    Formats
    Patterns of Failure
    Composable
    Fuzzers
    Specifying
    Constraints
    Learning from Inputs
    Input Coverage
    Fuzzing Digital
    Certificates

    View full-size slide

  48. 46
    Current Research
    Mining Input

    Formats
    Patterns of Failure
    Composable
    Fuzzers
    Specifying
    Constraints
    Learning from Inputs
    Input Coverage
    Fuzzing Digital
    Certificates

    View full-size slide

  49. 47
    Current Research
    Mining Input

    Formats

    View full-size slide

  50. def process_input(input)
    :

    try
    :

    ✘val = parse(input
    )

    res = process(val
    )

    return re
    s

    except SyntaxError
    :

    return Erro
    r

    48
    Parser

    View full-size slide

  51. SYNTAX ERROR
    def process_input(input)
    :

    try
    :

    ✘val = parse(input
    )

    res = process(val
    )

    return re
    s

    except SyntaxError
    :

    return Erro
    r

    49
    The Core

    View full-size slide

  52. def process_input(input)
    :

    try
    :

    ✔val = parse(input
    )

    res = process(val
    )

    return re
    s

    except SyntaxError
    :

    return Erro
    r

    {

    "store":
    {

    "book":
    [

    {

    "category":"reference"
    ,

    "author":"Nigel Rees"
    ,

    "title":"Sayings of the Century"
    ,

    "price":8.9
    5

    }
    ,

    {

    "category":"fiction"
    ,

    "author":"Evelyn Waugh"
    ,

    "title":"Sword of Honour"
    ,

    "price":12.9
    9

    }
    ,

    {

    "category":"fiction"
    ,

    "author":"J. R. R. Tolkien"
    ,

    "title":"The Lord of the Rings"
    ,

    "isbn":"0-395-19395-8"
    ,

    "price":22.9
    9

    }

    ]
    ,

    "bicycle":
    {

    "color":"red"
    ,

    "price":19.9
    5

    }

    }

    }
    50
    Input Format

    View full-size slide

  53. def process_input(input)
    :

    try
    :

    ✔val = parse(input
    )

    res = process(val
    )

    return re
    s

    except SyntaxError
    :

    return Erro
    r

    ::= 

    ::= 

    | 

    | 

    | 

    | `true`


    | `false
    `

    | `null`
    ::= `{``}
    `

    | `{}`
    ::= `,`>

    |
    ::= `:`
    ::= `[``]
    `

    | `[]`
    ::= `,`>

    |
    ::= `"` `"
    `

    | `""`
    ::= >

    |
    ::= [A-Za-z0-9]
    ::=
    ::= >

    |
    ::= [0-9]
    51
    Input Grammar

    View full-size slide

  54. 52
    Grammar
    JSON grammar
    ::= >


    ::= 

    | 

    | 

    | 

    | `true
    `

    | `false
    `

    | `null
    `

    ::= `{``}`| `{}`
    ::= `,` |
    ::= `:`
    ::= `[``]` | `[]`
    ::= `,` |
    ::= `"` `"` | `""`
    ::= |
    ::= [A-Za-z0-9]
    ::=
    ::= |
    ::= [0-9]

    View full-size slide

  55. 52
    Grammar
    JSON grammar
    key
    ::= >


    ::= 

    | 

    | 

    | 

    | `true
    `

    | `false
    `

    | `null
    `

    ::= `{``}`| `{}`
    ::= `,` |
    ::= `:`
    ::= `[``]` | `[]`
    ::= `,` |
    ::= `"` `"` | `""`
    ::= |
    ::= [A-Za-z0-9]
    ::=
    ::= |
    ::= [0-9]

    View full-size slide

  56. 52
    Grammar
    JSON grammar
    De
    f
    inition for
    key
    ::= >


    ::= 

    | 

    | 

    | 

    | `true
    `

    | `false
    `

    | `null
    `

    ::= `{``}`| `{}`
    ::= `,` |
    ::= `:`
    ::= `[``]` | `[]`
    ::= `,` |
    ::= `"` `"` | `""`
    ::= |
    ::= [A-Za-z0-9]
    ::=
    ::= |
    ::= [0-9]

    View full-size slide

  57. 53
    Grammar
    JSON grammar
    ::= >


    ::= 

    | 

    | 

    | 

    | `true
    `

    | `false
    `

    | `null
    `

    ::= `{``}`| `{}`
    ::= `,` |
    ::= `:`
    ::= `[``]` | `[]`
    ::= `,` |
    ::= `"` `"` | `""`
    ::= |
    ::= [A-Za-z0-9]
    ::=
    ::= |
    ::= [0-9]

    View full-size slide

  58. 53
    Grammar
    JSON grammar
    ::= >


    ::= 

    | 

    | 

    | 

    | `true
    `

    | `false
    `

    | `null
    `

    ::= `{``}`| `{}`
    ::= `,` |
    ::= `:`
    ::= `[``]` | `[]`
    ::= `,` |
    ::= `"` `"` | `""`
    ::= |
    ::= [A-Za-z0-9]
    ::=
    ::= |
    ::= [0-9]
    Expansion Rule

    View full-size slide

  59. 53
    Grammar
    JSON grammar
    ::= >


    ::= 

    | 

    | 

    | 

    | `true
    `

    | `false
    `

    | `null
    `

    ::= `{``}`| `{}`
    ::= `,` |
    ::= `:`
    ::= `[``]` | `[]`
    ::= `,` |
    ::= `"` `"` | `""`
    ::= |
    ::= [A-Za-z0-9]
    ::=
    ::= |
    ::= [0-9]
    Expansion Rule
    Terminal Symbol

    View full-size slide

  60. 53
    Grammar
    JSON grammar
    ::= >


    ::= 

    | 

    | 

    | 

    | `true
    `

    | `false
    `

    | `null
    `

    ::= `{``}`| `{}`
    ::= `,` |
    ::= `:`
    ::= `[``]` | `[]`
    ::= `,` |
    ::= `"` `"` | `""`
    ::= |
    ::= [A-Za-z0-9]
    ::=
    ::= |
    ::= [0-9]
    Expansion Rule
    Terminal Symbol
    Nonterminal Symbol

    View full-size slide

  61. 53
    Grammar
    JSON grammar
    ::= >


    ::= 

    | 

    | 

    | 

    | `true
    `

    | `false
    `

    | `null
    `

    ::= `{``}`| `{}`
    ::= `,` |
    ::= `:`
    ::= `[``]` | `[]`
    ::= `,` |
    ::= `"` `"` | `""`
    ::= |
    ::= [A-Za-z0-9]
    ::=
    ::= |
    ::= [0-9]
    Expansion Rule
    Terminal Symbol
    Nonterminal Symbol

    View full-size slide

  62. ::= 

    ::= 

    | 

    | 

    | 

    | `true` | `false` | `null`
    ::= `{``}`| `{}`
    ::= `,` |
    ::= `:`
    ::= `[``]` | `[]`
    ::= `,` |
    ::= `"` `"` | `""`
    ::= |
    ::= [A-Za-z0-9]
    ::=
    ::= |
    ::= [0-9] Derivation Tree
    54

    View full-size slide

  63. ::= 

    ::= 

    | 

    | 

    | 

    | `true` | `false` | `null`
    ::= `{``}`| `{}`
    ::= `,` |
    ::= `:`
    ::= `[``]` | `[]`
    ::= `,` |
    ::= `"` `"` | `""`
    ::= |
    ::= [A-Za-z0-9]
    ::=
    ::= |
    ::= [0-9] Derivation Tree
    {"":true}
    54

    View full-size slide

  64. {

    '' : [['']]
    ,

    '' : [['']
    ,

    ['']
    ,

    ['']
    ,

    ['']
    ,

    ['true'], ['false'], ['null']]
    ,

    '' : [['{', '','}']
    ,

    ['{}']]
    ,

    '' : [[',',',']
    ,

    ['']]
    ,

    '' : [['',':', '']]
    ,

    '' : [['[', '', ']']
    ,

    ['[]']]
    ,

    '' : [[',',',']
    ,

    ['']]
    ,

    '' : [['"', '', '"']
    ,

    ['""']]
    ,

    '' : [['','']
    ,

    ['']]
    ,

    '' : [['']]
    ,

    '' : [['','']
    ,

    ['']]
    ,

    '' : [[c] for c in string.characters
    ]

    '' : [[c] for c in string.digits]

    }
    Fuzzer
    55

    View full-size slide

  65. Parser
    56
    {

    '' : [['']]
    ,

    '' : [['']
    ,

    ['']
    ,

    ['']
    ,

    ['']
    ,

    ['true'], ['false'], ['null']]
    ,

    '' : [['{', '','}']
    ,

    ['{}']]
    ,

    '' : [[',',',']
    ,

    ['']]
    ,

    '' : [['',':', '']]
    ,

    '' : [['[', '', ']']
    ,

    ['[]']]
    ,

    '' : [[',',',']
    ,

    ['']]
    ,

    '' : [['"', '', '"']
    ,

    ['""']]
    ,

    '' : [['','']
    ,

    ['']]
    ,

    '' : [['']]
    ,

    '' : [['','']
    ,

    ['']]
    ,

    '' : [[c] for c in string.characters
    ]

    '' : [[c] for c in string.digits]

    }

    View full-size slide

  66. 57
    {

    '' : [['']]
    ,

    '' : [['']
    ,

    ['']
    ,

    ['']
    ,

    ['']
    ,

    ['true'], ['false'], ['null']]
    ,

    '' : [['{', '','}']
    ,

    ['{}']]
    ,

    '' : [[',',',']
    ,

    ['']]
    ,

    '' : [['',':', '']]
    ,

    '' : [['[', '', ']']
    ,

    ['[]']]
    ,

    '' : [[',',',']
    ,

    ['']]
    ,

    '' : [['"', '', '"']
    ,

    ['""']]
    ,

    '' : [['','']
    ,

    ['']]
    ,

    '' : [['']]
    ,

    '' : [['','']
    ,

    ['']]
    ,

    '' : [[c] for c in string.characters
    ]

    '' : [[c] for c in string.digits]

    } https://www.fuzzingbook.org/html/LangFuzzer.html

    View full-size slide

  67. 58
    Where to Get the Grammar From?

    View full-size slide

  68. 59
    Almost Everyone Uses Handwritten Parsers
    https://notes.eatonphil.com/parser-generators-vs-handwritten-parsers-survey-2021.html

    View full-size slide

  69. 61
    "Be liberal in what you accept, and conservative in what you send"
    Postel's Law

    View full-size slide

  70. 62
    "Be liberal in what you accept, and conservative in what you send"

    Postel's Law
    The Specification
    Where to Get the Grammar From?

    View full-size slide

  71. 62
    "Be liberal in what you accept, and conservative in what you send"

    Postel's Law
    The Specification
    The Implementation
    Extra "Features"
    Where to Get the Grammar From?

    View full-size slide

  72. 63
    Where to Get an Accurate Grammar?

    View full-size slide

  73. 64
    Where to Get an Accurate Grammar?
    Hand-written parsers already encode the grammar

    View full-size slide

  74. def json_raw(stm)
    :

    while True
    :

    stm.skipspaces(
    )

    c = stm.peek(
    )

    if c == 't'
    :

    return json_fixed(stm, 'true'
    )

    elif c == 'f'
    :

    return json_fixed(stm, 'false'
    )

    elif c == 'n':


    return json_fixed(stm, 'null'
    )

    elif c == '"':


    return json_string(stm
    )

    elif c == '{':


    return json_dict(stm
    )

    elif c == '[':


    return json_list(stm
    )

    elif c in NUMSTART
    :

    return json_number(stm
    )

    raise JSONError(E_MALF, stm, stm.pos)
    https://github.com/phensley/microjson 65
    Source to Control Flow

    View full-size slide

  75. def json_raw(stm)
    :

    while True
    :

    stm.skipspaces(
    )

    c = stm.peek(
    )

    if c == 't'
    :

    return json_fixed(stm, 'true'
    )

    elif c == 'f'
    :

    return json_fixed(stm, 'false'
    )

    elif c == 'n':


    return json_fixed(stm, 'null'
    )

    elif c == '"':


    return json_string(stm
    )

    elif c == '{':


    return json_dict(stm
    )

    elif c == '[':


    return json_list(stm
    )

    elif c in NUMSTART
    :

    return json_number(stm
    )

    raise JSONError(E_MALF, stm, stm.pos)
    https://github.com/phensley/microjson 65
    Control Flow Graph
    Source to Control Flow

    View full-size slide

  76. Grammar Mining From Control Flow
    Sequence
    A
    B
    C
    [F]
    Selection
    cond
    A B
    [F]
    F
    T
    Iteration
    cond
    B
    [F]
    66

    View full-size slide

  77. :=
    Grammar Mining From Control Flow
    Sequence
    A
    B
    C
    [F]
    Selection
    cond
    A B
    [F]
    F
    T
    Iteration
    cond
    B
    [F]
    66

    View full-size slide

  78. := >

    |
    :=
    Grammar Mining From Control Flow
    Sequence
    A
    B
    C
    [F]
    Selection
    cond
    A B
    [F]
    F
    T
    Iteration
    cond
    B
    [F]
    66

    View full-size slide

  79. := >

    |
    := := >

    |
    Grammar Mining From Control Flow
    Sequence
    A
    B
    C
    [F]
    Selection
    cond
    A B
    [F]
    F
    T
    Iteration
    cond
    B
    [F]
    66

    View full-size slide

  80. 67
    https://github.com/vrthra/mimid/
    Gopinath, Mathis, and Zeller. Mining Input Grammars from Dynamic Control Flow. ESEC/FSE 2020.
    Mimid

    View full-size slide

  81. Recall
    Subjects Mimid
    calc.py 100.0 %
    mathexpr.py 87.5 %
    cgidecode.py 100.0 %
    urlparse.py 100.0 %
    microjson.py 98.7 %
    parseclisp.py 99.3 %
    jsonparser.c 100.0 %
    tiny.c 100.0 %
    mjs.c 95.4 %
    Inputs generated by inferred grammar
    that were accepted by the program
    Subjects Mimid
    calc.py 100.0 %
    mathexpr.py 92.7 %
    cgidecode.py 100.0 %
    urlparse.py 96.4 %
    microjson.py 93.0 %
    parseclisp.py 80.6 %
    jsonparser.c 83.8 %
    tiny.c 92.8 %
    mjs.c 95.9 %
    Inputs generated by golden grammar that
    were accepted by the inferred grammar parser
    Precision
    Evaluation: Accuracy
    68

    View full-size slide

  82. 69
    Sample Free Generators

    View full-size slide

  83. 69
    Sample Free Generators
    A
    [ 2
    ,
    B
    9
    )
    4 ]
    A ∉ [,+,-,1,2,3,4,5,6,7,8,9,0
    B ∉ +,-,1,2,3,4,5,6,7,8,9,0,)
    ) ∉ +,-,1,2,3,4,5,6,7,8,9,0
    [2,94]

    View full-size slide

  84. Grammar Miner
    Program Under Test
    Sample Free Generator
    Grammar Fuzzer
    Inputs
    70

    View full-size slide

  85. 71
    Extract The Program Input Format Automatically
    What Does This Mean For You?
    • Fuzzer agnostic


    • Easy to use


    • Open Source
    https://github.com/vrthra/mimid

    View full-size slide

  86. 72
    Current Research
    Mining Input

    Formats

    View full-size slide

  87. 73
    Current Research
    Mining Input

    Formats
    Patterns of Failure
    Composable

    Fuzzers
    Specifying
    Constraints
    Learning from Inputs
    Input Coverage
    Fuzzing Digital
    Certificates

    View full-size slide

  88. Patterns of Failure
    74
    Current Research

    View full-size slide

  89. def process_input(input)
    :

    try
    :

    val = parse(input
    )

    res = process(val
    )

    return re
    s

    except SyntaxError
    :

    return Erro
    r

    75

    View full-size slide

  90. def process_input(input)
    :

    try
    :

    val = parse(input
    )

    res = process(val
    )

    return re
    s

    except SyntaxError
    :

    return Erro
    r

    {"type":"PathNode","matrix":
    {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948
    628471,"dy":371.5686591257294},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,
    "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":
    [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415
    6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":
    [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971
    7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":
    [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]}
    ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":
    [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],
    "shadow":false},{"type":"PathNode","matrix":
    {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328
    55618,"dy":133.20628077515605},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star
    tX":126.25,"startY":127.50445838342671,"closed":true,"segments":
    [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127
    .50445838342671,"r":
    [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439
    5]},
    {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049
    55,"r":
    [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},
    {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":
    [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337
    9]},
    {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834
    2671,"r":
    [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}
    ],"shadow":false},{"type":"TextNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children":
    [],"
    fi
    llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20},
    {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children":
    [],"
    fi
    llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20},
    {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":
    [],"arrowSize":10,"path":{"type":"PathNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":464,"dy":-3},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"

    75

    View full-size slide

  91. {"type":"PathNode","matrix":
    {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948
    628471,"dy":371.5686591257294},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,
    "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":
    [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415
    6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":
    [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971
    7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":
    [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]}
    ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":
    [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],
    "shadow":false},{"type":"PathNode","matrix":
    {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328
    55618,"dy":133.20628077515605},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star
    tX":126.25,"startY":127.50445838342671,"closed":true,"segments":
    [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127
    .50445838342671,"r":
    [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439
    5]},
    {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049
    55,"r":
    [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},
    {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":
    [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337
    9]},
    {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834
    2671,"r":
    [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}
    ],"shadow":false},{"type":"TextNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children":
    [],"
    fi
    llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20},
    {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children":
    [],"
    fi
    llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20},
    {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":
    [],"arrowSize":10,"path":{"type":"PathNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":464,"dy":-3},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"
    What is the smallest failure inducing input?
    76
    Delta Debugging

    View full-size slide

  92. {"type":"PathNode","matrix":
    {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948
    628471,"dy":371.5686591257294},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,
    "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":
    [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415
    6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":
    [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971
    7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":
    [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]}
    ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":
    [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],
    "shadow":false},{"type":"PathNode","matrix":
    {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328
    55618,"dy":133.20628077515605},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star
    tX":126.25,"startY":127.50445838342671,"closed":true,"segments":
    [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127
    .50445838342671,"r":
    [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439
    5]},
    {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049
    55,"r":
    [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},
    {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":
    [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337
    9]},
    {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834
    2671,"r":
    [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}
    ],"shadow":false},{"type":"TextNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children":
    [],"
    fi
    llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20},
    {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children":
    [],"
    fi
    llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20},
    {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":
    [],"arrowSize":10,"path":{"type":"PathNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":464,"dy":-3},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3," 77
    Grammar Based Delta Debugging

    View full-size slide

  93. {"type":"PathNode","matrix":
    {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948
    628471,"dy":371.5686591257294},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,
    "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":
    [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415
    6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":
    [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971
    7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":
    [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]}
    ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":
    [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],
    "shadow":false},{"type":"PathNode","matrix":
    {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328
    55618,"dy":133.20628077515605},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star
    tX":126.25,"startY":127.50445838342671,"closed":true,"segments":
    [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127
    .50445838342671,"r":
    [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439
    5]},
    {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049
    55,"r":
    [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},
    {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":
    [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337
    9]},
    {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834
    2671,"r":
    [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}
    ],"shadow":false},{"type":"TextNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children":
    [],"
    fi
    llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20},
    {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children":
    [],"
    fi
    llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20},
    {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":
    [],"arrowSize":10,"path":{"type":"PathNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":464,"dy":-3},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"
    ::= 

    ::= 

    | 

    | 

    | 

    | `true`


    | `false
    `

    | `null
    `

    ::= `"` `"
    `

    | `""
    `

    ::=


    |
    ::= `{``}
    `

    | `{}`
    ::= `,`>

    |
    ::= `:`
    ::= `[``]
    `

    | `[]`
    ::= `,`>

    |
    ::= >

    ::=


    |
    77
    Grammar Based Delta Debugging

    View full-size slide

  94. {"":[]}
    78
    Grammar Based Delta Debugging
    {"type":"PathNode","matrix":
    {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948
    628471,"dy":371.5686591257294},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,
    "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":
    [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415
    6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":
    [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971
    7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":
    [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]}
    ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":
    [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],
    "shadow":false},{"type":"PathNode","matrix":
    {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328
    55618,"dy":133.20628077515605},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star
    tX":126.25,"startY":127.50445838342671,"closed":true,"segments":
    [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127
    .50445838342671,"r":
    [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439
    5]},
    {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049
    55,"r":
    [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},
    {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":
    [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337
    9]},
    {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834
    2671,"r":
    [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}
    ],"shadow":false},{"type":"TextNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children":
    [],"
    fi
    llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20},
    {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children":
    [],"
    fi
    llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20},
    {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":
    [],"arrowSize":10,"path":{"type":"PathNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":464,"dy":-3},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"
    Test Minimization

    View full-size slide

  95. {"":[]} ✘
    78
    Grammar Based Delta Debugging
    {"type":"PathNode","matrix":
    {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948
    628471,"dy":371.5686591257294},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,
    "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":
    [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415
    6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":
    [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971
    7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":
    [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]}
    ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":
    [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],
    "shadow":false},{"type":"PathNode","matrix":
    {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328
    55618,"dy":133.20628077515605},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star
    tX":126.25,"startY":127.50445838342671,"closed":true,"segments":
    [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127
    .50445838342671,"r":
    [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439
    5]},
    {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049
    55,"r":
    [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},
    {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":
    [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337
    9]},
    {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834
    2671,"r":
    [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}
    ],"shadow":false},{"type":"TextNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children":
    [],"
    fi
    llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20},
    {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children":
    [],"
    fi
    llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20},
    {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":
    [],"arrowSize":10,"path":{"type":"PathNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":464,"dy":-3},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"
    Test Minimization

    View full-size slide

  96. 79
    Grammar Based Delta Debugging
    {"type":"PathNode","matrix":
    {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948
    628471,"dy":371.5686591257294},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,
    "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":
    [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415
    6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":
    [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971
    7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":
    [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]}
    ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":
    [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],
    "shadow":false},{"type":"PathNode","matrix":
    {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328
    55618,"dy":133.20628077515605},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star
    tX":126.25,"startY":127.50445838342671,"closed":true,"segments":
    [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127
    .50445838342671,"r":
    [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439
    5]},
    {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049
    55,"r":
    [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},
    {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":
    [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337
    9]},
    {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834
    2671,"r":
    [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}
    ],"shadow":false},{"type":"TextNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children":
    [],"
    fi
    llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20},
    {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children":
    [],"
    fi
    llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20},
    {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":
    [],"arrowSize":10,"path":{"type":"PathNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":464,"dy":-3},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"
    Why?
    {"":[]}

    View full-size slide

  97. 79
    Grammar Based Delta Debugging
    {"type":"PathNode","matrix":
    {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948
    628471,"dy":371.5686591257294},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,
    "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":
    [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415
    6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":
    [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971
    7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":
    [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]}
    ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":
    [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],
    "shadow":false},{"type":"PathNode","matrix":
    {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328
    55618,"dy":133.20628077515605},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star
    tX":126.25,"startY":127.50445838342671,"closed":true,"segments":
    [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127
    .50445838342671,"r":
    [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439
    5]},
    {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049
    55,"r":
    [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},
    {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":
    [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337
    9]},
    {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834
    2671,"r":
    [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}
    ],"shadow":false},{"type":"TextNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children":
    [],"
    fi
    llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20},
    {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children":
    [],"
    fi
    llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20},
    {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":
    [],"arrowSize":10,"path":{"type":"PathNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":464,"dy":-3},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"
    Why?
    [12345]
    {"":[]}

    View full-size slide

  98. 79
    Grammar Based Delta Debugging
    {"type":"PathNode","matrix":
    {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948
    628471,"dy":371.5686591257294},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,
    "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":
    [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415
    6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":
    [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971
    7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":
    [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]}
    ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":
    [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],
    "shadow":false},{"type":"PathNode","matrix":
    {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328
    55618,"dy":133.20628077515605},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star
    tX":126.25,"startY":127.50445838342671,"closed":true,"segments":
    [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127
    .50445838342671,"r":
    [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439
    5]},
    {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049
    55,"r":
    [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},
    {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":
    [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337
    9]},
    {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834
    2671,"r":
    [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}
    ],"shadow":false},{"type":"TextNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children":
    [],"
    fi
    llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20},
    {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children":
    [],"
    fi
    llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20},
    {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":
    [],"arrowSize":10,"path":{"type":"PathNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":464,"dy":-3},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"
    Why?
    [12345]
    {"":[]}
    {"":0}

    View full-size slide

  99. 79
    Grammar Based Delta Debugging
    {"type":"PathNode","matrix":
    {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948
    628471,"dy":371.5686591257294},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,
    "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":
    [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415
    6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":
    [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971
    7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":
    [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]}
    ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":
    [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],
    "shadow":false},{"type":"PathNode","matrix":
    {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328
    55618,"dy":133.20628077515605},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star
    tX":126.25,"startY":127.50445838342671,"closed":true,"segments":
    [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127
    .50445838342671,"r":
    [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439
    5]},
    {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049
    55,"r":
    [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},
    {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":
    [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337
    9]},
    {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834
    2671,"r":
    [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}
    ],"shadow":false},{"type":"TextNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children":
    [],"
    fi
    llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20},
    {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children":
    [],"
    fi
    llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20},
    {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":
    [],"arrowSize":10,"path":{"type":"PathNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":464,"dy":-3},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"
    Why?
    [12345]
    {"":[]}
    {"":0}
    {"x":[]}

    View full-size slide

  100. 80
    DDSET
    Gopinath, Kampmann, Havrikov, Soremekun, and Zeller. Abstracting Failure Inducing Inputs. ISSTA 2020.
    https://github.com/vrthra/ddset

    View full-size slide

  101. {"": []}
    DDSET:
    81
    ::= 

    ::= 

    | 

    | 

    | 

    | `true`


    | `false
    `

    | `null
    `

    ::= `"` `"
    `

    | `""
    `

    ::=


    |
    ::= `{``}
    `

    | `{}`
    ::= `,`>

    |
    ::= `:`
    ::= `[``]
    `

    | `[]`
    ::= `,`>

    |
    ::= >

    ::=


    |

    View full-size slide

  102. {"": []}
    82
    DDSET:
    ::= 

    ::= 

    | 

    | 

    | 

    | `true`


    | `false
    `

    | `null
    `

    ::= `"` `"
    `

    | `""
    `

    ::=


    |
    ::= `{``}
    `

    | `{}`
    ::= `,`>

    |
    ::= `:`
    ::= `[``]
    `

    | `[]`
    ::= `,`>

    |
    ::= >

    ::=


    |

    View full-size slide

  103. {"": []}
    {"7897A": []}
    {"klnm,.qer;dfs?P":[]}
    {"123KOUIJ!qR30578950":[]} 82
    DDSET:
    ::= 

    ::= 

    | 

    | 

    | 

    | `true`


    | `false
    `

    | `null
    `

    ::= `"` `"
    `

    | `""
    `

    ::=


    |
    ::= `{``}
    `

    | `{}`
    ::= `,`>

    |
    ::= `:`
    ::= `[``]
    `

    | `[]`
    ::= `,`>

    |
    ::= >

    ::=


    |

    View full-size slide

  104. {"": []}
    83
    DDSET:
    ::= 

    ::= 

    | 

    | 

    | 

    | `true`


    | `false
    `

    | `null
    `

    ::= `"` `"
    `

    | `""
    `

    ::=


    |
    ::= `{``}
    `

    | `{}`
    ::= `,`>

    |
    ::= `:`
    ::= `[``]
    `

    | `[]`
    ::= `,`>

    |
    ::= >

    ::=


    |

    View full-size slide

  105. {"": []}
    {"": true}
    {"":[1,2,445,"x"]}
    {"":{"PQ":[true, false, 223,"a"]}} 83
    DDSET:
    ::= 

    ::= 

    | 

    | 

    | 

    | `true`


    | `false
    `

    | `null
    `

    ::= `"` `"
    `

    | `""
    `

    ::=


    |
    ::= `{``}
    `

    | `{}`
    ::= `,`>

    |
    ::= `:`
    ::= `[``]
    `

    | `[]`
    ::= `,`>

    |
    ::= >

    ::=


    |

    View full-size slide

  106. {"": []}
    Abstraction
    {"": }
    Abstract Input
    84
    DDSET:
    ::= 

    ::= 

    | 

    | 

    | 

    | `true`


    | `false
    `

    | `null
    `

    ::= `"` `"
    `

    | `""
    `

    ::=


    |
    ::= `{``}
    `

    | `{}`
    ::= `,`>

    |
    ::= `:`
    ::= `[``]
    `

    | `[]`
    ::= `,`>

    |
    ::= >

    ::=


    |

    View full-size slide

  107. {"": }
    Abstract Input
    {"": []}
    Minimized Input
    85
    ::= 

    ::= 

    | 

    | 

    | 

    | `true`


    | `false
    `

    | `null
    `

    ::= `"` `"
    `

    | `""
    `

    ::=


    |
    ::= `{``}
    `

    | `{}`
    ::= `,`>

    |
    ::= `:`
    ::= `[``]
    `

    | `[]`
    ::= `,`>

    |
    ::= >

    ::=


    |

    View full-size slide

  108. Issue 2842 from Closure
    var A = class extends (class {}){};
    Issue 2937 from Closure
    {while ((l_0)){ if ((l_0)) {break;;var l_0; continue }0}}
    var {baz:{} = baz => {}} = baz => {};
    Issue 385 from Rhino
    const [y,y] = [];
    Issue 386 from Rhino
    86

    View full-size slide

  109. Issue 2842 from Closure
    var A = class extends (class {}){};
    Issue 2937 from Closure
    {while ((l_0)){ if ((l_0)) {break;;var l_0; continue }0}}
    var {baz:{} = baz => {}} = baz => {};
    Issue 385 from Rhino
    const [y,y] = [];
    Issue 386 from Rhino
    = class extends (class {}){}
    var {<$Id1>:{} = <$Id1> => {}} ;
    {while ((<$Id1>)){ if ((<$Id1>)) {break;;var <$Id1>; continue }0}}
    const [<$Id1>,<$Id1>] = []
    86

    View full-size slide

  110. {"": }
    Abstract Input
    {"": []}
    Minimized Input
    87
    ::= 

    ::= 

    | 

    | 

    | 

    | `true`


    | `false
    `

    | `null
    `

    ::= `"` `"
    `

    | `""
    `

    ::=


    |
    ::= `{``}
    `

    | `{}`
    ::= `,`>

    |
    ::= `:`
    ::= `[``]
    `

    | `[]`
    ::= `,`>

    |
    ::= >

    ::=


    |
    Evocative Grammar

    View full-size slide

  111. {"": }
    Abstract Input
    {"": []}
    Minimized Input
    87
    ::= 

    ::= 

    | 

    | 

    | 

    | `true`


    | `false
    `

    | `null
    `

    ::= `"` `"
    `

    | `""
    `

    ::=


    |
    ::= `{``}
    `

    | `{}`
    ::= `,`>

    |
    ::= `:`
    ::= `[``]
    `

    | `[]`
    ::= `,`>

    |
    ::= >

    ::=


    |



    where is "":
    Evocative Pattern
    Evocative Grammar

    View full-size slide

  112. {"": }
    Abstract Input
    {"": []}
    Minimized Input
    87
    ::= 

    ::= 

    | 

    | 

    | 

    | `true`


    | `false
    `

    | `null
    `

    ::= `"` `"
    `

    | `""
    `

    ::=


    |
    ::= `{``}
    `

    | `{}`
    ::= `,`>

    |
    ::= `:`
    ::= `[``]
    `

    | `[]`
    ::= `,`>

    |
    ::= >

    ::=


    |
    ::= 

    ::= 

    | >

    | 

    | 

    ::= `{``}`

    ::=


    | `,`>

    | `,`

    ::= `:`>

    | `:`

    ::= `[``]`

    ::= >

    | `,`>

    | `,`>

    ::= `""
    `

    ::= 

    ::= 

    | >

    | 

    | 

    | `true` | `false` | `null`

    ::= `{``}` | `{}`

    ::= | `,`

    ::= `:`

    ::= `[``]` | `[]`

    ::= | `,`

    ::= `"` `"` | `""`

    ::= 

    ::= [A-Za-z0-9]

    ::= 

    ::= | 

    ::= [0-9]



    where is "":
    Evocative Pattern
    Evocative Grammar

    View full-size slide

  113. 88
    generate()
    ::= 

    ::= 

    | >

    | 

    | 

    ::= `{``}`

    ::=


    | `,`>

    | `,`

    ::= `:`>

    | `:`

    ::= `[``]`

    ::= >

    | `,`>

    | `,`>

    ::= `""
    `

    ::= 

    ::= 

    | >

    | 

    | 

    | `true` | `false` | `null`

    ::= `{``}` | `{}`

    ::= | `,`

    ::= `:`

    ::= `[``]` | `[]`

    ::= | `,`

    ::= `"` `"` | `""`

    ::= 

    ::= [A-Za-z0-9]

    ::= 

    ::= | 

    ::= [0-9]



    where is "":
    Evocative Grammar

    View full-size slide

  114. 88
    {"": 100}
    {"": [343,{},44998]}
    [{"": {"xxy":44998, {"b":[1,2,3]}}},[],[]]
    {"_": {"ket":[], {"":[],"y",[[],[1,2,3,455,6]]}}}
    {".":[{3243435656:"xy,zzzpqiu"},[{"":[112]},{"d":[[]]},{}]]}
    [{"": [1,2,3,4]}]
    {"pqr": {"": [1,2,3,4]}, "abc":[]}
    [[1132],{"xx":[{6:"dafjli;y,zzzdfaiu"},[{"__":[1{}{}]},{"":[[444456]]},{}]]}
    generate()








    ::= 

    ::= 

    | >

    | 

    | 

    ::= `{``}`

    ::=


    | `,`>

    | `,`

    ::= `:`>

    | `:`

    ::= `[``]`

    ::= >

    | `,`>

    | `,`>

    ::= `""
    `

    ::= 

    ::= 

    | >

    | 

    | 

    | `true` | `false` | `null`

    ::= `{``}` | `{}`

    ::= | `,`

    ::= `:`

    ::= `[``]` | `[]`

    ::= | `,`

    ::= `"` `"` | `""`

    ::= 

    ::= [A-Za-z0-9]

    ::= 

    ::= | 

    ::= [0-9]



    where is "":
    Evocative Grammar

    View full-size slide

  115. 89



    where is "":
    1. We can produce any and all instances of the
    failure inducing pattern
    .

    2. We can recognize any input that contains the
    failure inducing pattern
    .

    3. The grammar will reject any input that doesn't
    contain the failure inducing pattern.
    ::= 

    ::= 

    | >

    | 

    | 

    ::= `{``}`

    ::=


    | `,`>

    | `,`

    ::= `:`>

    | `:`

    ::= `[``]`

    ::= >

    | `,`>

    | `,`>

    ::= `""
    `

    ::= 

    ::= 

    | >

    | 

    | 

    | `true` | `false` | `null`

    ::= `{``}` | `{}`

    ::= | `,`

    ::= `:`

    ::= `[``]` | `[]`

    ::= | `,`

    ::= `"` `"` | `""`

    ::= 

    ::= [A-Za-z0-9]

    ::= 

    ::= | 

    ::= [0-9]
    Evocative Grammar
    (Statistical Guarantees based on the accuracy of the evocative pattern)

    View full-size slide

  116. 90
    Evocative Patterns
    What Does This Mean For You?
    • Automatically Mined


    • Produce Specialized Fuzzers


    • Open Source
    https://github.com/vrthra/ddset

    View full-size slide

  117. Patterns of Failure
    91
    Current Research

    View full-size slide

  118. 92
    Current Research
    Mining Input

    Formats
    Patterns of Failure
    Composable

    Fuzzers
    Specifying
    Constraints
    Learning from Inputs
    Input Coverage
    Fuzzing Digital
    Certificates

    View full-size slide

  119. 93
    Current Research
    Composable

    Fuzzers

    View full-size slide




  120. where is "":
    if json.has_key("")
    :

    raise Exception()
    94
    Evocative Pattern

    View full-size slide




  121. where is "":
    if json.has_key("")
    :

    raise Exception()
    95
    Composing Evocative Patterns

    View full-size slide




  122. where is "":
    if json.has_key("")
    :

    raise Exception()
    if json.has_key_value(null)
    :

    raise Exception()
    95
    Composing Evocative Patterns

    View full-size slide




  123. where is "":
    if json.has_key("")
    :

    raise Exception()
    if json.has_key_value(null)
    :

    raise Exception()



    where is : null
    95
    Composing Evocative Patterns

    View full-size slide

  124. if json.has_key("") and json.has_key_value(null)
    :

    raise Exception()
    96
    Composing Evocative Patterns

    View full-size slide

  125. if json.has_key("") and json.has_key_value(null)
    :

    raise Exception()



    where is "":

    is : null
    96
    Composing Evocative Patterns

    View full-size slide

  126. if json.has_key("") and not json.has_key_value(null)
    :

    raise Exception()
    97
    Composing Evocative Patterns

    View full-size slide

  127. if json.has_key("") and not json.has_key_value(null)
    :

    raise Exception()



    where is "":


    is : null
    97
    Composing Evocative Patterns

    View full-size slide

  128. if json.has_key("")
    :

    raise Exception(
    )

    if json.has_key_value(null)
    :

    raise Exception()
    98
    Composing Evocative Patterns

    View full-size slide

  129. if json.has_key("")
    :

    raise Exception(
    )

    if json.has_key_value(null)
    :

    raise Exception()



    where is "":


    is : null
    98
    Composing Evocative Patterns

    View full-size slide

  130. := >

    := >

    | >

    := '[' ']
    '

    := '{' '}
    '

    := >

    | ','>

    | ','>

    := >

    | ','>

    | ','>

    := ':'>

    | ':'>

    := >

    := 'false' | 'true
    '

    | | >

    | >

    := '[]' | '[' ']
    '

    := '{}' | '{' '}
    '

    := >

    | ','>

    := >

    | ','>

    := ':'>

    := 'false' | 'true
    '

    | | >

    | |



    where is "":


    is : null
    generate()
    99

    View full-size slide

  131. := >

    := >

    | >

    := '[' ']
    '

    := '{' '}
    '

    := >

    | ','>

    | ','>

    := >

    | ','>

    | ','>

    := ':'>

    | ':'>

    := >

    := 'false' | 'true
    '

    | | >

    | >

    := '[]' | '[' ']
    '

    := '{}' | '{' '}
    '

    := >

    | ','>

    := >

    | ','>

    := ':'>

    := 'false' | 'true
    '

    | | >

    | |



    where is "":


    is : null
    {"": 100}
    {"": [343,{},44998]}
    [{"": {"xxy":44998, {"b":[1,2,3]}}},[],[]]
    {"_": {"ket":[], {"":[],"y",[[],[1,2,3,455,6]]}}}
    {".":[{3243435656:"xy,zzzpqiu"},[{"":[112]},{"d":[[]]},{}]]}
    [{"": [1,2,3,4]}]
    {"pqr": {"": [1,2,3,4]}, "abc":[]}
    [[1132],{"xx":[{6:"dafjli;y,zzzdfaiu"},[{"__":[1{}{}]},{"":[[444456]]},{}]]}
    generate()








    99

    View full-size slide

  132. Issue 386 from Rhino
    var A = class extends (class {}){};
    Issue 2937 from Closure
    const [y,y] = [];
    var {baz:{} = baz => {}} = baz => {};
    Issue 385 from Rhino
    {while ((l_0)){ if ((l_0)) {break;;var l_0; continue }0}}
    Issue 2842 from Closure
    = class extends (class {}){}
    var {<$Id1>:{} = <$Id1> => {}} ;
    const [<$Id1>,<$Id1>] = []
    {while ((<$Id1>)){ if ((<$Id1>)) {break;;var <$Id1>; continue }0}}
    100

    View full-size slide

  133. where


    is = class extends (class {}){}


    is {while ((<$Id1>)){ if ((<$Id1>)) {break;;var <$Id1>; continue }0}}


    is var {<$Id2>:{} = <$Id2> => {}} ;


    is const [<$Id3>,<$Id3>] = []
    Evocative Expressions
    101

    View full-size slide

  134. where


    is = class extends (class {}){}


    is {while ((<$Id1>)){ if ((<$Id1>)) {break;;var <$Id1>; continue }0}}


    is var {<$Id2>:{} = <$Id2> => {}} ;


    is const [<$Id3>,<$Id3>] = []
    Evocative Expressions

    101

    View full-size slide

  135. where


    is = class extends (class {}){}


    is {while ((<$Id1>)){ if ((<$Id1>)) {break;;var <$Id1>; continue }0}}


    is var {<$Id2>:{} = <$Id2> => {}} ;


    is const [<$Id3>,<$Id3>] = []
    Evocative Expressions


    101

    View full-size slide

  136. where


    is = class extends (class {}){}


    is {while ((<$Id1>)){ if ((<$Id1>)) {break;;var <$Id1>; continue }0}}


    is var {<$Id2>:{} = <$Id2> => {}} ;


    is const [<$Id3>,<$Id3>] = []
    Evocative Expressions



    101

    View full-size slide

  137. where


    is = class extends (class {}){}


    is {while ((<$Id1>)){ if ((<$Id1>)) {break;;var <$Id1>; continue }0}}


    is var {<$Id2>:{} = <$Id2> => {}} ;


    is const [<$Id3>,<$Id3>] = []
    Evocative Expressions




    101

    View full-size slide

  138. Evocative Expressions for Focused Fuzzing
    def assign_admin_rights()
    :

    self.db_rights.add(MODIFY_DB
    )

    self.fs_rights = R
    W

    self.timeout = 6
    0

    self.deploy = Tru
    e

    def assign_guest_rights()
    :

    self.db_rights = [QUERY_DB
    ]

    self.fs_rights = Non
    e

    self.timeout = Non
    e

    self.deploy = Fals
    e

    def modify_db(stmt)
    :

    if ADMIN in self.db_rights
    :

    process(stmt
    )

    else
    :

    raise Error(
    )

    def query_db(stmt)
    :

    process(stmt
    )

    View full-size slide

  139. 103
    Evocative Expressions for Focused Fuzzing
    def assign_admin_rights()
    :

    self.db_rights.add(MODIFY_DB
    )

    self.fs_rights = R
    W

    self.timeout = 6
    0

    self.deploy = Tru
    e

    def assign_guest_rights()
    :

    self.db_rights = [QUERY_DB
    ]

    self.fs_rights = Non
    e

    self.timeout = Non
    e

    self.deploy = Fals
    e

    def modify_db(stmt)
    :

    if ADMIN in self.db_rights
    :

    process(stmt
    )

    else
    :

    raise Error(
    )

    def query_db(stmt)
    :

    process(stmt
    )

    {"role" : "admin"}

    View full-size slide

  140. 103



    where is "role": "admin"
    Evocative Expressions for Focused Fuzzing
    def assign_admin_rights()
    :

    self.db_rights.add(MODIFY_DB
    )

    self.fs_rights = R
    W

    self.timeout = 6
    0

    self.deploy = Tru
    e

    def assign_guest_rights()
    :

    self.db_rights = [QUERY_DB
    ]

    self.fs_rights = Non
    e

    self.timeout = Non
    e

    self.deploy = Fals
    e

    def modify_db(stmt)
    :

    if ADMIN in self.db_rights
    :

    process(stmt
    )

    else
    :

    raise Error(
    )

    def query_db(stmt)
    :

    process(stmt
    )

    {"role" : "admin"}

    View full-size slide

  141. 104
    Evocative Expressions for Focused Fuzzing
    def assign_admin_rights()
    :

    self.db_rights.add(MODIFY_DB
    )

    self.fs_rights = R
    W

    self.timeout = 6
    0

    self.deploy = Tru
    e

    def assign_guest_rights()
    :

    self.db_rights = [QUERY_DB
    ]

    self.fs_rights = Non
    e

    self.timeout = Non
    e

    self.deploy = Fals
    e

    def modify_db(stmt)
    :

    if ADMIN in self.db_rights
    :

    process(stmt
    )

    else
    :

    raise Error(
    )

    def query_db(stmt)
    :

    process(stmt
    )

    {"method":"remove_table","args":["orders", "inventory"]}



    where is "method":"remove_table","args":

    View full-size slide

  142. 105
    Evocative Expressions for Focused Fuzzing
    def assign_admin_rights()
    :

    self.db_rights.add(MODIFY_DB
    )

    self.fs_rights = R
    W

    self.timeout = 6
    0

    self.deploy = Tru
    e

    def assign_guest_rights()
    :

    self.db_rights = [QUERY_DB
    ]

    self.fs_rights = Non
    e

    self.timeout = Non
    e

    self.deploy = Fals
    e

    def modify_db(stmt)
    :

    if ADMIN in self.db_rights
    :

    process(stmt
    )

    else
    :

    raise Error(
    )

    def query_db(stmt)
    :

    process(stmt
    )




    where is "role": "admin"


    is {"method":"remove_table","args":}

    View full-size slide

  143. 105
    Evocative Expressions for Focused Fuzzing
    def assign_admin_rights()
    :

    self.db_rights.add(MODIFY_DB
    )

    self.fs_rights = R
    W

    self.timeout = 6
    0

    self.deploy = Tru
    e

    def assign_guest_rights()
    :

    self.db_rights = [QUERY_DB
    ]

    self.fs_rights = Non
    e

    self.timeout = Non
    e

    self.deploy = Fals
    e

    def modify_db(stmt)
    :

    if ADMIN in self.db_rights
    :

    process(stmt
    )

    else
    :

    raise Error(
    )

    def query_db(stmt)
    :

    process(stmt
    )




    where is "role": "admin"


    is {"method":"remove_table","args":}
    {"method":"remove_table","args":["orders", "inventory"], "role":"admin"}
    {"role":"admin", "method":"remove_table","args":["orders", "inventory"]}
    {"method":"remove_table","role":"admin","args":["orders", "inventory"]}
    {"method":"remove_table","args":["orders","inventory",{"role":"admin"}]}

    View full-size slide

  144. 106
    Evocative Expressions for Focused Fuzzing
    def assign_admin_rights()
    :

    self.db_rights.add(MODIFY_DB
    )

    self.fs_rights = R
    W

    self.timeout = 6
    0

    self.deploy = Tru
    e

    def assign_guest_rights()
    :

    self.db_rights = [QUERY_DB
    ]

    self.fs_rights = Non
    e

    self.timeout = Non
    e

    self.deploy = Fals
    e

    def modify_db(stmt)
    :

    if ADMIN in self.db_rights
    :

    process(stmt
    )

    else
    :

    raise Error(
    )

    def query_db(stmt)
    :

    process(stmt
    )




    where is "role": "admin"


    is {"method":"remove_table","args":}

    View full-size slide

  145. 106
    Evocative Expressions for Focused Fuzzing
    def assign_admin_rights()
    :

    self.db_rights.add(MODIFY_DB
    )

    self.fs_rights = R
    W

    self.timeout = 6
    0

    self.deploy = Tru
    e

    def assign_guest_rights()
    :

    self.db_rights = [QUERY_DB
    ]

    self.fs_rights = Non
    e

    self.timeout = Non
    e

    self.deploy = Fals
    e

    def modify_db(stmt)
    :

    if ADMIN in self.db_rights
    :

    process(stmt
    )

    else
    :

    raise Error(
    )

    def query_db(stmt)
    :

    process(stmt
    )




    where is "role": "admin"


    is {"method":"remove_table","args":}
    {"method":"remove_table","args":["orders", "inventory"]}
    {"role":"guest", "method":"remove_table","args":{"role":"guest"}}
    {"method":"remove_table","role":"guest","args":["orders", "inventory"]}
    {"method":"remove_table","args":["orders","inventory",{"role":"guest"}]}

    View full-size slide

  146. Evocative Expressions: Data Structures
    Algebraic Data Types
    107
    ::=
    struct mystruct
    {

    stype m1
    ;

    stype m2
    ;

    };
    union myunion
    {

    utype m1
    ;

    utype m2
    ;

    };
    ::=
    |
    Data Structures
    Context Free Grammar

    View full-size slide

  147. 108
    Composable Fuzzers
    REST
    with a REST vulnerability
    and a SQL Injection
    SQL
    but doesn't Induce BugB
    BugB
    or cover function A
    FnA
    I want an XML fuzzer
    XML

    View full-size slide

  148. XMLREST
    108
    Composable Fuzzers
    REST
    with a REST vulnerability
    and a SQL Injection
    SQL
    but doesn't Induce BugB
    BugB
    or cover function A
    FnA
    I want an XML fuzzer
    XML

    View full-size slide

  149. XMLREST XMLSQL
    108
    Composable Fuzzers
    REST
    with a REST vulnerability
    and a SQL Injection
    SQL
    but doesn't Induce BugB
    BugB
    or cover function A
    FnA
    I want an XML fuzzer
    XML

    View full-size slide

  150. XMLREST XMLBugB
    XMLSQL
    108
    Composable Fuzzers
    REST
    with a REST vulnerability
    and a SQL Injection
    SQL
    but doesn't Induce BugB
    BugB
    or cover function A
    FnA
    I want an XML fuzzer
    XML

    View full-size slide

  151. XMLREST XMLFnA
    XMLBugB
    XMLSQL
    108
    Composable Fuzzers
    REST
    with a REST vulnerability
    and a SQL Injection
    SQL
    but doesn't Induce BugB
    BugB
    or cover function A
    FnA
    I want an XML fuzzer
    XML

    View full-size slide

  152. XMLREST XMLFnA
    XMLBugB
    XMLSQL
    109
    Composable Fuzzers
    REST
    with a REST vulnerability
    and a SQL Injection
    SQL
    but doesn't Induce BugB
    BugB
    or cover function A
    FnA
    & & not[ ]
    |
    I want an XML fuzzer
    XML

    View full-size slide

  153. 110
    Evocative Expressions = Composable Fuzzers
    I want a fuzzer that targets a REST server wit
    h

    • SQL Injections A,B, and
    C

    • But does not go through the input sanitizer code
    I want a fuzzer that targets a C compiler wit
    h

    • No undefined behaviors in the produced input
    s

    • But contain at least one function pointer declaration
    I want a fuzzer that targets a database wit
    h

    • Each input containing previously fixed bugs A and
    B

    • But does not induce a known bug
    C

    • And does not cover the function X in the database source code
    I want a fuzzer that targets a JSON parser wit
    h

    • Each input containing at least one known quirk from other parsers
    I want a fuzzer that targets my applicatio
    n

    • Each input exercising the code I just fixe
    d

    • And also other known bug pattern
    s

    • But does not consume resource A

    View full-size slide

  154. 110
    Evocative Expressions = Composable Fuzzers
    I want a fuzzer that targets a REST server wit
    h

    • SQL Injections A,B, and
    C

    • But does not go through the input sanitizer code
    I want a fuzzer that targets a C compiler wit
    h

    • No undefined behaviors in the produced input
    s

    • But contain at least one function pointer declaration
    I want a fuzzer that targets a database wit
    h

    • Each input containing previously fixed bugs A and
    B

    • But does not induce a known bug
    C

    • And does not cover the function X in the database source code
    I want a fuzzer that targets a JSON parser wit
    h

    • Each input containing at least one known quirk from other parsers
    I want a fuzzer that targets my applicatio
    n

    • Each input exercising the code I just fixe
    d

    • And also other known bug pattern
    s

    • But does not consume resource A
    All w
    ithout w
    riting
    a
    single
    line
    of code

    View full-size slide

  155. Gopinath, Nemati, Zeller. Input Algebras. ICSE 2021.
    Evocative Expressions
    https://rahul.gopinath.org/posts/
    111

    View full-size slide

  156. 112
    Evocative Expressions
    What Does This Mean For You?
    • Mix and match specialized fuzzers


    • Use historical bugs


    • An ecosystem of targeted fuzzers


    • Open Source
    https://github.com/vrthra/Ewoks

    View full-size slide

  157. 113
    Current Research
    Composable

    Fuzzers

    View full-size slide

  158. 114
    Current Research
    Mining Input

    Formats
    Patterns of Failure
    Composable

    Fuzzers
    Specifying
    Constraints
    Learning from Inputs
    Input Coverage
    Fuzzing Digital
    Certificates

    View full-size slide

  159. 115
    Current Research
    Mining Input

    Formats
    Patterns of Failure
    Composable

    Fuzzers
    Specifying
    Constraints
    Learning from Inputs
    Input Coverage
    Fuzzing Digital
    Certificates

    View full-size slide