Input Languages for Effective and Focused Fuzzing

FuzzCon Europe 2021

Transcript

1. Rahul Gopinath
   https://rahul.gopinath.org
   @_rahulgopinath
   Rafael Dutra
   https://cispa.de/en/people/c01radu/
   1
   Input Languages for
   

   Effective and Focused Fuzzing
   

   View Slide

2. Rahul Gopinath
   https://rahul.gopinath.org
   @_rahulgopinath
   Rafael Dutra
   https://cispa.de/en/people/c01radu/
   FuzzCon Europe 2021
   1
   Input Languages for
   

   Effective and Focused Fuzzing
   

   View Slide

3. 2
   2
   Feedback-Directed Fuzzers: Challenges
   file.png
   AFL readpng
   PNG
   corpus
   save inputs that increase coverage
   

   View Slide

4. 3
   3
   Feedback-Directed Fuzzers: Challenges
   file.png
   AFL readpng
   PNG
   corpus
   save inputs that increase coverage
   Most mutations
   make the file
   invalid
   Hard to find
   deeper
   semantic bugs
   

   View Slide

5. 4
   4
   Feedback-Directed Fuzzers: Challenges
   file.png
   AFL readpng
   PNG
   corpus
   save inputs that increase coverage
   Highly
   dependent on
   initial corpus
   

   View Slide

6. 5
   5
   Feedback-Directed Fuzzers: Challenges
   file.png
   AFL readpng
   PNG
   corpus
   save inputs that increase coverage
   Require
   feedback from
   the program
   

   View Slide

7. 6
   6
   What About a Format-specic Fuzzer?
   

   View Slide

8. 7
   7
   How to Generate a PNG le?
   

   View Slide

9. 8
   8
   Read the Spec?
   

   View Slide

10. 9
    9
    How Can We Even Inspect a PNG?
    

    View Slide

11. 10
    10
    How Can We Even Inspect a PNG?
    010 Editor
    

    View Slide

12. 11
    11
    Binary Templates
    typedef struct {
    uint32 length;
    char type[4];
    ubyte data[length];
    uint32 crc;
    } PNG_CHUNK;
    

    View Slide

13. 12
    12
    Binary Templates
    typedef struct {
    uint32 length;
    char type[4];
    ubyte data[length];
    uint32 crc;
    } PNG_CHUNK;
    

    View Slide

14. 13
    13
    Binary Templates
    typedef struct {
    uint32 length;
    local int64 start = FTell();
    char type[4];
    ubyte data[length];
    local int64 end = FTell();
    local uint32 crc_calc = Checksum(CHECKSUM_CRC32, start, end-start);
    uint32 crc;
    if (crc != crc_calc)
    Warning("Bad CRC %08x; expected: %08x", crc, crc_calc);
    } PNG_CHUNK;
    

    View Slide

15. 14
    14
    Binary Templates
    typedef struct {
    uint32 length;
    local int64 start = FTell();
    char type[4];
    ubyte data[length];
    local int64 end = FTell();
    local uint32 crc_calc = Checksum(CHECKSUM_CRC32, start, end-start);
    uint32 crc;
    if (crc != crc_calc)
    Warning("Bad CRC %08x; expected: %08x", crc, crc_calc);
    } PNG_CHUNK;
    

    View Slide

16. 15
    15
    Binary Templates
    typedef struct {
    uint32 length;
    local int64 start = FTell();
    char type[4];
    if (type == "IHDR")
    PNG_CHUNK_IHDR ihdr;
    else if (type == "PLTE")
    PNG_CHUNK_PLTE plte(length);
    /* … */
    local int64 end = FTell();
    local uint32 crc_calc = Checksum(CHECKSUM_CRC32, start, end-start);
    uint32 crc;
    if (crc != crc_calc)
    Warning("Bad CRC %08x; expected: %08x", crc, crc_calc);
    } PNG_CHUNK;
    

    View Slide

17. 16
    16
    200+ Binary Templates Available
    

    View Slide

18. 17
    17
    Binary Templates as Generators?
    

    View Slide

19. 18
    18
    Genera.ng Valid Files
    typedef struct {
    uint32 length;
    local int64 start = FTell();
    char type[4];
    if (type == "IHDR")
    PNG_CHUNK_IHDR ihdr;
    else if (type == "PLTE")
    PNG_CHUNK_PLTE plte(length);
    /* … */
    local int64 end = FTell();
    local uint32 crc_calc = Checksum(CHECKSUM_CRC32, start, end-start);
    uint32 crc;
    if (crc != crc_calc)
    Warning("Bad CRC %08x; expected: %08x", crc, crc_calc);
    } PNG_CHUNK;
    

    View Slide

20. 19
    19
    Genera.ng Valid Files
    typedef struct {
    uint32 length;
    local int64 start = FTell();
    char type[4];
    if (type == "IHDR")
    PNG_CHUNK_IHDR ihdr;
    else if (type == "PLTE")
    PNG_CHUNK_PLTE plte(length);
    /* … */
    local int64 end = FTell();
    local uint32 crc_calc = Checksum(CHECKSUM_CRC32, start, end-start);
    uint32 crc = { crc_calc };
    if (crc != crc_calc)
    Warning("Bad CRC %08x; expected: %08x", crc, crc_calc);
    } PNG_CHUNK;
    

    View Slide

21. 20
    20
    Genera.ng Valid Files
    typedef struct {
    uint32 length;
    local int64 start = FTell();
    char type[4];
    if (type == "IHDR")
    PNG_CHUNK_IHDR ihdr;
    else if (type == "PLTE")
    PNG_CHUNK_PLTE plte(length);
    /* … */
    local int64 end = FTell();
    local uint32 crc_calc = Checksum(CHECKSUM_CRC32, start, end-start);
    uint32 crc = { crc_calc };
    if (crc != crc_calc)
    Warning("Bad CRC %08x; expected: %08x", crc, crc_calc);
    } PNG_CHUNK;
    

    View Slide

22. 21
    21
    Genera.ng Valid Files
    typedef struct {
    uint32 length ;
    local int64 start = FTell();
    char type[4];
    if (type == "IHDR")
    PNG_CHUNK_IHDR ihdr;
    else if (type == "PLTE")
    PNG_CHUNK_PLTE plte(length);
    /* … */
    local int64 end = FTell(); /* ← Fix length here */
    local uint32 crc_calc = Checksum(CHECKSUM_CRC32, start, end-start);
    uint32 crc = { crc_calc };
    if (crc != crc_calc)
    Warning("Bad CRC %08x; expected: %08x", crc, crc_calc);
    } PNG_CHUNK;
    

    View Slide

23. 22
    22
    Genera.ng Valid Files
    typedef struct {
    uint32 length ;
    local int64 start = FTell();
    char type[4];
    if (type == "IHDR")
    PNG_CHUNK_IHDR ihdr;
    else if (type == "PLTE")
    PNG_CHUNK_PLTE plte(length);
    /* … */
    local int64 end = FTell(); /* ← Fix length here */
    local uint32 crc_calc = Checksum(CHECKSUM_CRC32, start, end-start);
    uint32 crc = { crc_calc };
    if (crc != crc_calc)
    Warning("Bad CRC %08x; expected: %08x", crc, crc_calc);
    } PNG_CHUNK;
    

    View Slide

24. 23
    23
    Genera.ng Valid Files
    typedef struct {
    uint32 length ;
    local int64 start = FTell();
    char type[4];
    if (type == "IHDR")
    PNG_CHUNK_IHDR ihdr;
    else if (type == "PLTE")
    PNG_CHUNK_PLTE plte(length);
    /* … */
    local int64 end = FTell(); /* ← Fix length here */
    local uint32 crc_calc = Checksum(CHECKSUM_CRC32, start, end-start);
    uint32 crc = { crc_calc };
    if (crc != crc_calc)
    Warning("Bad CRC %08x; expected: %08x", crc, crc_calc);
    } PNG_CHUNK;
    

    View Slide

25. 24
    24
    Genera.ng Valid Files
    typedef struct {
    uint32 length ;
    local int64 start = FTell();
    char type[4];
    if (type == "IHDR")
    PNG_CHUNK_IHDR ihdr;
    else if (type == "PLTE")
    PNG_CHUNK_PLTE plte(length);
    /* … */
    local int64 end = FTell(); /* ← Fix length here */
    local uint32 crc_calc = Checksum(CHECKSUM_CRC32, start, end-start);
    uint32 crc = { crc_calc };
    if (crc != crc_calc)
    Warning("Bad CRC %08x; expected: %08x", crc, crc_calc);
    } PNG_CHUNK;
    IHDR PLTE IDAT IEND
    

    View Slide

26. 25
    25
    FormatFuzzer
    

    View Slide

27. 26
    26
    FormatFuzzer
    decision seed file.png
    generate
    mutate
    parse
    png-fuzzer
    png.bt
    C++
    

    View Slide

28. 27
    27
    Decision Seeds
    89 P N G 0d 0a 1a 0a
    00 00 00 00
    Decision Seed Generated PNG File
    

    View Slide

29. 28
    28
    Decision Seeds
    89 P N G 0d 0a 1a 0a 00 00 00 0d I H D R
    00 00 00 01 00 00 00 01 08 02 00 00 00 90 77 53
    de
    00 00 00 00 00 00 00 0c 00 00 00 00 00 00 08 00
    01 00 00 00 00 00 00 00 00
    Decision Seed Generated PNG File
    

    View Slide

30. 29
    29
    Decision Seeds
    89 P N G 0d 0a 1a 0a 00 00 00 0d I H D R
    00 00 00 01 00 00 00 01 08 02 00 00 00 90 77 53
    de 00 00 00 0c I D A T 78 da 63 f8 cf c0 00
    00 03 01 01 00 f7 03 41 43
    00 00 00 00 00 00 00 0c 00 00 00 00 00 00 08 00
    01 00 00 00 00 00 00 00 00 00 00 01 00 0b 00 29
    24 21 34 00 ff 00 00 08 00 00
    Decision Seed Generated PNG File
    

    View Slide

31. 30
    30
    Decision Seeds
    89 P N G 0d 0a 1a 0a 00 00 00 0d I H D R
    00 00 00 01 00 00 00 01 08 02 00 00 00 90 77 53
    de 00 00 00 0c I D A T 78 da 63 f8 cf c0 00
    00 03 01 01 00 f7 03 41 43 00 00 00 00 I E N
    D ae 42 60 82
    00 00 00 00 00 00 00 0c 00 00 00 00 00 00 08 00
    01 00 00 00 00 00 00 00 00 00 00 01 00 0b 00 29
    24 21 34 00 ff 00 00 08 00 00 00 00 7f 00 00 00
    00 00 00 00
    Decision Seed Generated PNG File
    

    View Slide

32. 31
    31
    Smart Muta.ons
    

    View Slide

33. 32
    32
    Smart Muta.ons Respect Context
    typedef struct {
    uint32 length ;
    local int64 start = FTell();
    char type[4];
    if (type == "IHDR")
    PNG_CHUNK_IHDR ihdr;
    else if (type == "PLTE")
    PNG_CHUNK_PLTE plte(length);
    /* … */
    local int64 end = FTell(); /* ← Fix length here */
    local uint32 crc_calc = Checksum(CHECKSUM_CRC32, start, end-start);
    uint32 crc = { crc_calc };
    if (crc != crc_calc)
    Warning("Bad CRC %08x; expected: %08x", crc, crc_calc);
    } PNG_CHUNK;
    

    View Slide

34. 33
    33
    Smart Muta.ons Respect Context
    typedef struct {
    uint32 length ;
    local int64 start = FTell();
    char type[4];
    if (type == "IHDR")
    PNG_CHUNK_IHDR ihdr;
    else if (type == "PLTE")
    PNG_CHUNK_PLTE plte(length);
    /* … */
    local int64 end = FTell(); /* ← Fix length here */
    local uint32 crc_calc = Checksum(CHECKSUM_CRC32, start, end-start);
    uint32 crc = { crc_calc };
    if (crc != crc_calc)
    Warning("Bad CRC %08x; expected: %08x", crc, crc_calc);
    } PNG_CHUNK;
    

    View Slide

35. 34
    34
    How to use FormatFuzzer?
    

    View Slide

36. 35
    35
    Fuzzing Strategies: Black-box Genera.on
    decision seed file.png
    generate
    mutate
    parse
    png-fuzzer
    /dev/urandom
    readpng
    FFGen
    

    View Slide

37. 36
    36
    Fuzzing Strategies: Black-box Muta.on
    decision seed
    corpus
    PNG
    corpus
    generate
    mutate
    parse
    png-fuzzer
    FFMut
    

    View Slide

38. 37
    37
    Fuzzing Strategies: Black-box Muta.on
    decision seed
    corpus
    file.png
    generate
    mutate
    parse
    png-fuzzer
    FFMut
    readpng
    

    View Slide

39. 38
    38
    Feedback-Directed Fuzzers
    file.png
    AFL
    AFL
    readpng
    PNG
    corpus
    save inputs that increase coverage
    

    View Slide

40. 39
    39
    Fuzzing Strategies: Generator-based Fuzzing
    AFL
    AFL+FFGen
    readpng
    save inputs that increase coverage
    decision seed
    corpus
    decision seed
    generate
    mutate
    parse
    png-fuzzer file.png
    

    View Slide

41. 40
    40
    Fuzzing Strategies: Muta.on-based Fuzzing
    file.png
    AFL
    AFL+FFMut
    readpng
    PNG
    corpus
    save inputs that increase coverage
    generate
    mutate
    parse
    png-fuzzer
    

    View Slide

42. 41
    41
    Evalua.on
    • Effort: most lines in the binary templates
    remain unchanged
    • Speed: ~7000 files/s generated or parsed
    • Success: 97% generations are successful
    • Accuracy: 76% of generated files are valid
    (82% without evil decisions)
    

    View Slide

43. 42
    42
    Fuzzing Results (Line Coverage %)
    PNG JPG GIF MIDI MP4 ZIP PCAP AVI BMP
    FFGen 22.3 24.2 68.7 12.3 5.6 33.7 11.5 5.6 27.8
    FFMut 22.5 24.1 70.7 10.4 6.9 34.8 7.8 6.7 27.8
    AFL 17.6 29.0 73.3 11.7 10.3 36.1 24.0 9.3 30.7
    AFL+FFGen 23.6 26.7 71.6 11.9 9.0 36.5 21.1 9.1 27.9
    AFL+FFMut 26.0 33.1 73.2 12.2 10.2 37.1 23.4 10.1 30.7
    AFLSmart 18.0 29.4 12.1 10.8 36.2 24.1 10.7
    

    View Slide

44. 43
    43
    Bugs Found
    • FFmpeg: 8 distinct bugs already fixed by
    FFmpeg developers
    ●
    Most are segmentation faults related to allocation
    • TiMidity: 19 distinct memory errors found
    

    View Slide

45. 44
    44
    FormatFuzzer
    • FormatFuzzer 1.0 released today
    ● Includes integration with AFL++ 2.60c
    • Project page: https://uds-se.github.io/FormatFuzzer/
    • Source: https://github.com/uds-se/FormatFuzzer
    • Preprint: http://arxiv.org/abs/2109.11277
    

    View Slide

46. 45
    Current Research
    

    View Slide

47. 45
    Current Research
    Mining Input

    Formats
    Patterns of Failure
    Composable
    Fuzzers
    Specifying
    Constraints
    Learning from Inputs
    Input Coverage
    Fuzzing Digital
    Certificates
    

    View Slide

48. 46
    Current Research
    Mining Input

    Formats
    Patterns of Failure
    Composable
    Fuzzers
    Specifying
    Constraints
    Learning from Inputs
    Input Coverage
    Fuzzing Digital
    Certificates
    

    View Slide

49. 47
    Current Research
    Mining Input

    Formats
    

    View Slide

50. def process_input(input)
    :
    
    try
    :
    
    ✘val = parse(input
    )
    
    res = process(val
    )
    
    return re
    s
    
    except SyntaxError
    :
    
    return Erro
    r
    
    48
    Parser
    

    View Slide

51. SYNTAX ERROR
    def process_input(input)
    :
    
    try
    :
    
    ✘val = parse(input
    )
    
    res = process(val
    )
    
    return re
    s
    
    except SyntaxError
    :
    
    return Erro
    r
    
    49
    The Core
    

    View Slide

52. def process_input(input)
    :
    
    try
    :
    
    ✔val = parse(input
    )
    
    res = process(val
    )
    
    return re
    s
    
    except SyntaxError
    :
    
    return Erro
    r
    
    {
    
    "store":
    {
    
    "book":
    [
    
    {
    
    "category":"reference"
    ,
    
    "author":"Nigel Rees"
    ,
    
    "title":"Sayings of the Century"
    ,
    
    "price":8.9
    5
    
    }
    ,
    
    {
    
    "category":"fiction"
    ,
    
    "author":"Evelyn Waugh"
    ,
    
    "title":"Sword of Honour"
    ,
    
    "price":12.9
    9
    
    }
    ,
    
    {
    
    "category":"fiction"
    ,
    
    "author":"J. R. R. Tolkien"
    ,
    
    "title":"The Lord of the Rings"
    ,
    
    "isbn":"0-395-19395-8"
    ,
    
    "price":22.9
    9
    
    }
    
    ]
    ,
    
    "bicycle":
    {
    
    "color":"red"
    ,
    
    "price":19.9
    5
    
    }
    
    }
    
    }
    50
    Input Format
    

    View Slide

53. def process_input(input)
    :
    
    try
    :
    
    ✔val = parse(input
    )
    
    res = process(val
    )
    
    return re
    s
    
    except SyntaxError
    :
    
    return Erro
    r
    
    ::= 

    ::= 

    | 

    | 

    | 

    | `true`
    
    
    | `false
    `
    
    | `null`
    ::= `{``}
    `
    
    | `{}`
    ::= `,`>
    
    |
    ::= `:`
    ::= `[``]
    `
    
    | `[]`
    ::= `,`>
    
    |
    ::= `"` `"
    `
    
    | `""`
    ::= >
    
    |
    ::= [A-Za-z0-9]
    ::=
    ::= >
    
    |
    ::= [0-9]
    51
    Input Grammar
    

    View Slide

54. 52
    Grammar
    JSON grammar
    ::= >
    
    

    ::= 

    | 

    | 

    | 

    | `true
    `
    
    | `false
    `
    
    | `null
    `
    
    ::= `{``}`| `{}`
    ::= `,` |
    ::= `:`
    ::= `[``]` | `[]`
    ::= `,` |
    ::= `"` `"` | `""`
    ::= |
    ::= [A-Za-z0-9]
    ::=
    ::= |
    ::= [0-9]
    

    View Slide

55. 52
    Grammar
    JSON grammar
    key
    ::= >
    
    

    ::= 

    | 

    | 

    | 

    | `true
    `
    
    | `false
    `
    
    | `null
    `
    
    ::= `{``}`| `{}`
    ::= `,` |
    ::= `:`
    ::= `[``]` | `[]`
    ::= `,` |
    ::= `"` `"` | `""`
    ::= |
    ::= [A-Za-z0-9]
    ::=
    ::= |
    ::= [0-9]
    

    View Slide

56. 52
    Grammar
    JSON grammar
    De
    f
    inition for
    key
    ::= >
    
    

    ::= 

    | 

    | 

    | 

    | `true
    `
    
    | `false
    `
    
    | `null
    `
    
    ::= `{``}`| `{}`
    ::= `,` |
    ::= `:`
    ::= `[``]` | `[]`
    ::= `,` |
    ::= `"` `"` | `""`
    ::= |
    ::= [A-Za-z0-9]
    ::=
    ::= |
    ::= [0-9]
    

    View Slide

57. 53
    Grammar
    JSON grammar
    ::= >
    
    

    ::= 

    | 

    | 

    | 

    | `true
    `
    
    | `false
    `
    
    | `null
    `
    
    ::= `{``}`| `{}`
    ::= `,` |
    ::= `:`
    ::= `[``]` | `[]`
    ::= `,` |
    ::= `"` `"` | `""`
    ::= |
    ::= [A-Za-z0-9]
    ::=
    ::= |
    ::= [0-9]
    

    View Slide

58. 53
    Grammar
    JSON grammar
    ::= >
    
    

    ::= 

    | 

    | 

    | 

    | `true
    `
    
    | `false
    `
    
    | `null
    `
    
    ::= `{``}`| `{}`
    ::= `,` |
    ::= `:`
    ::= `[``]` | `[]`
    ::= `,` |
    ::= `"` `"` | `""`
    ::= |
    ::= [A-Za-z0-9]
    ::=
    ::= |
    ::= [0-9]
    Expansion Rule
    

    View Slide

59. 53
    Grammar
    JSON grammar
    ::= >
    
    

    ::= 

    | 

    | 

    | 

    | `true
    `
    
    | `false
    `
    
    | `null
    `
    
    ::= `{``}`| `{}`
    ::= `,` |
    ::= `:`
    ::= `[``]` | `[]`
    ::= `,` |
    ::= `"` `"` | `""`
    ::= |
    ::= [A-Za-z0-9]
    ::=
    ::= |
    ::= [0-9]
    Expansion Rule
    Terminal Symbol
    

    View Slide

60. 53
    Grammar
    JSON grammar
    ::= >
    
    

    ::= 

    | 

    | 

    | 

    | `true
    `
    
    | `false
    `
    
    | `null
    `
    
    ::= `{``}`| `{}`
    ::= `,` |
    ::= `:`
    ::= `[``]` | `[]`
    ::= `,` |
    ::= `"` `"` | `""`
    ::= |
    ::= [A-Za-z0-9]
    ::=
    ::= |
    ::= [0-9]
    Expansion Rule
    Terminal Symbol
    Nonterminal Symbol
    

    View Slide

61. 53
    Grammar
    JSON grammar
    ::= >
    
    

    ::= 

    | 

    | 

    | 

    | `true
    `
    
    | `false
    `
    
    | `null
    `
    
    ::= `{``}`| `{}`
    ::= `,` |
    ::= `:`
    ::= `[``]` | `[]`
    ::= `,` |
    ::= `"` `"` | `""`
    ::= |
    ::= [A-Za-z0-9]
    ::=
    ::= |
    ::= [0-9]
    Expansion Rule
    Terminal Symbol
    Nonterminal Symbol
    

    View Slide

62. ::= 

    ::= 

    | 

    | 

    | 

    | `true` | `false` | `null`
    ::= `{``}`| `{}`
    ::= `,` |
    ::= `:`
    ::= `[``]` | `[]`
    ::= `,` |
    ::= `"` `"` | `""`
    ::= |
    ::= [A-Za-z0-9]
    ::=
    ::= |
    ::= [0-9] Derivation Tree
    54
    

    View Slide

63. ::= 

    ::= 

    | 

    | 

    | 

    | `true` | `false` | `null`
    ::= `{``}`| `{}`
    ::= `,` |
    ::= `:`
    ::= `[``]` | `[]`
    ::= `,` |
    ::= `"` `"` | `""`
    ::= |
    ::= [A-Za-z0-9]
    ::=
    ::= |
    ::= [0-9] Derivation Tree
    {"":true}
    54
    

    View Slide

64. {

    '' : [['']]
    ,
    
    '' : [['']
    ,
    
    ['']
    ,
    
    ['']
    ,
    
    ['']
    ,
    
    ['true'], ['false'], ['null']]
    ,
    
    '' : [['{', '','}']
    ,
    
    ['{}']]
    ,
    
    '' : [[',',',']
    ,
    
    ['']]
    ,
    
    '' : [['',':', '']]
    ,
    
    '' : [['[', '', ']']
    ,
    
    ['[]']]
    ,
    
    '' : [[',',',']
    ,
    
    ['']]
    ,
    
    '' : [['"', '', '"']
    ,
    
    ['""']]
    ,
    
    '' : [['','']
    ,
    
    ['']]
    ,
    
    '' : [['']]
    ,
    
    '' : [['','']
    ,
    
    ['']]
    ,
    
    '' : [[c] for c in string.characters
    ]
    
    '' : [[c] for c in string.digits]

    }
    Fuzzer
    55
    

    View Slide

65. Parser
    56
    {

    '' : [['']]
    ,
    
    '' : [['']
    ,
    
    ['']
    ,
    
    ['']
    ,
    
    ['']
    ,
    
    ['true'], ['false'], ['null']]
    ,
    
    '' : [['{', '','}']
    ,
    
    ['{}']]
    ,
    
    '' : [[',',',']
    ,
    
    ['']]
    ,
    
    '' : [['',':', '']]
    ,
    
    '' : [['[', '', ']']
    ,
    
    ['[]']]
    ,
    
    '' : [[',',',']
    ,
    
    ['']]
    ,
    
    '' : [['"', '', '"']
    ,
    
    ['""']]
    ,
    
    '' : [['','']
    ,
    
    ['']]
    ,
    
    '' : [['']]
    ,
    
    '' : [['','']
    ,
    
    ['']]
    ,
    
    '' : [[c] for c in string.characters
    ]
    
    '' : [[c] for c in string.digits]

    }
    

    View Slide

66. 57
    {

    '' : [['']]
    ,
    
    '' : [['']
    ,
    
    ['']
    ,
    
    ['']
    ,
    
    ['']
    ,
    
    ['true'], ['false'], ['null']]
    ,
    
    '' : [['{', '','}']
    ,
    
    ['{}']]
    ,
    
    '' : [[',',',']
    ,
    
    ['']]
    ,
    
    '' : [['',':', '']]
    ,
    
    '' : [['[', '', ']']
    ,
    
    ['[]']]
    ,
    
    '' : [[',',',']
    ,
    
    ['']]
    ,
    
    '' : [['"', '', '"']
    ,
    
    ['""']]
    ,
    
    '' : [['','']
    ,
    
    ['']]
    ,
    
    '' : [['']]
    ,
    
    '' : [['','']
    ,
    
    ['']]
    ,
    
    '' : [[c] for c in string.characters
    ]
    
    '' : [[c] for c in string.digits]

    } https://www.fuzzingbook.org/html/LangFuzzer.html
    

    View Slide

67. 58
    Where to Get the Grammar From?
    

    View Slide

68. 59
    Almost Everyone Uses Handwritten Parsers
    https://notes.eatonphil.com/parser-generators-vs-handwritten-parsers-survey-2021.html
    

    View Slide

69. View Slide

70. View Slide

71. 61
    "Be liberal in what you accept, and conservative in what you send"
    Postel's Law
    

    View Slide

72. 62
    "Be liberal in what you accept, and conservative in what you send"

    Postel's Law
    The Specification
    Where to Get the Grammar From?
    

    View Slide

73. 62
    "Be liberal in what you accept, and conservative in what you send"

    Postel's Law
    The Specification
    The Implementation
    Extra "Features"
    Where to Get the Grammar From?
    

    View Slide

74. 63
    Where to Get an Accurate Grammar?
    

    View Slide

75. 64
    Where to Get an Accurate Grammar?
    Hand-written parsers already encode the grammar
    

    View Slide

76. def json_raw(stm)
    :
    
    while True
    :
    
    stm.skipspaces(
    )
    
    c = stm.peek(
    )
    
    if c == 't'
    :
    
    return json_fixed(stm, 'true'
    )
    
    elif c == 'f'
    :
    
    return json_fixed(stm, 'false'
    )
    
    elif c == 'n':
    
    
    return json_fixed(stm, 'null'
    )
    
    elif c == '"':
    
    
    return json_string(stm
    )
    
    elif c == '{':
    
    
    return json_dict(stm
    )
    
    elif c == '[':
    
    
    return json_list(stm
    )
    
    elif c in NUMSTART
    :
    
    return json_number(stm
    )
    
    raise JSONError(E_MALF, stm, stm.pos)
    https://github.com/phensley/microjson 65
    Source to Control Flow
    

    View Slide

77. def json_raw(stm)
    :
    
    while True
    :
    
    stm.skipspaces(
    )
    
    c = stm.peek(
    )
    
    if c == 't'
    :
    
    return json_fixed(stm, 'true'
    )
    
    elif c == 'f'
    :
    
    return json_fixed(stm, 'false'
    )
    
    elif c == 'n':
    
    
    return json_fixed(stm, 'null'
    )
    
    elif c == '"':
    
    
    return json_string(stm
    )
    
    elif c == '{':
    
    
    return json_dict(stm
    )
    
    elif c == '[':
    
    
    return json_list(stm
    )
    
    elif c in NUMSTART
    :
    
    return json_number(stm
    )
    
    raise JSONError(E_MALF, stm, stm.pos)
    https://github.com/phensley/microjson 65
    Control Flow Graph
    Source to Control Flow
    

    View Slide

78. Grammar Mining From Control Flow
    Sequence
    A
    B
    C
    [F]
    Selection
    cond
    A B
    [F]
    F
    T
    Iteration
    cond
    B
    [F]
    66
    

    View Slide

79. :=
    Grammar Mining From Control Flow
    Sequence
    A
    B
    C
    [F]
    Selection
    cond
    A B
    [F]
    F
    T
    Iteration
    cond
    B
    [F]
    66
    

    View Slide

80. := >
    
    |
    :=
    Grammar Mining From Control Flow
    Sequence
    A
    B
    C
    [F]
    Selection
    cond
    A B
    [F]
    F
    T
    Iteration
    cond
    B
    [F]
    66
    

    View Slide

81. := >
    
    |
    := := >
    
    |
    Grammar Mining From Control Flow
    Sequence
    A
    B
    C
    [F]
    Selection
    cond
    A B
    [F]
    F
    T
    Iteration
    cond
    B
    [F]
    66
    

    View Slide

82. 67
    https://github.com/vrthra/mimid/
    Gopinath, Mathis, and Zeller. Mining Input Grammars from Dynamic Control Flow. ESEC/FSE 2020.
    Mimid
    

    View Slide

83. Recall
    Subjects Mimid
    calc.py 100.0 %
    mathexpr.py 87.5 %
    cgidecode.py 100.0 %
    urlparse.py 100.0 %
    microjson.py 98.7 %
    parseclisp.py 99.3 %
    jsonparser.c 100.0 %
    tiny.c 100.0 %
    mjs.c 95.4 %
    Inputs generated by inferred grammar
    that were accepted by the program
    Subjects Mimid
    calc.py 100.0 %
    mathexpr.py 92.7 %
    cgidecode.py 100.0 %
    urlparse.py 96.4 %
    microjson.py 93.0 %
    parseclisp.py 80.6 %
    jsonparser.c 83.8 %
    tiny.c 92.8 %
    mjs.c 95.9 %
    Inputs generated by golden grammar that
    were accepted by the inferred grammar parser
    Precision
    Evaluation: Accuracy
    68
    

    View Slide

84. 69
    Sample Free Generators
    

    View Slide

85. 69
    Sample Free Generators
    A
    [ 2
    ,
    B
    9
    )
    4 ]
    A ∉ [,+,-,1,2,3,4,5,6,7,8,9,0
    B ∉ +,-,1,2,3,4,5,6,7,8,9,0,)
    ) ∉ +,-,1,2,3,4,5,6,7,8,9,0
    [2,94]
    

    View Slide

86. Grammar Miner
    Program Under Test
    Sample Free Generator
    Grammar Fuzzer
    Inputs
    70
    

    View Slide

87. 71
    Extract The Program Input Format Automatically
    What Does This Mean For You?
    • Fuzzer agnostic
    
    
    • Easy to use
    
    
    • Open Source
    https://github.com/vrthra/mimid
    

    View Slide

88. 72
    Current Research
    Mining Input

    Formats
    

    View Slide

89. 73
    Current Research
    Mining Input

    Formats
    Patterns of Failure
    Composable

    Fuzzers
    Specifying
    Constraints
    Learning from Inputs
    Input Coverage
    Fuzzing Digital
    Certificates
    

    View Slide

90. Patterns of Failure
    74
    Current Research
    

    View Slide

91. def process_input(input)
    :
    
    try
    :
    
    val = parse(input
    )
    
    res = process(val
    )
    
    return re
    s
    
    except SyntaxError
    :
    
    return Erro
    r
    
    75
    

    View Slide

92. def process_input(input)
    :
    
    try
    :
    
    val = parse(input
    )
    
    res = process(val
    )
    
    return re
    s
    
    except SyntaxError
    :
    
    return Erro
    r
    
    {"type":"PathNode","matrix":
    {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948
    628471,"dy":371.5686591257294},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,
    "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":
    [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415
    6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":
    [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971
    7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":
    [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]}
    ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":
    [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],
    "shadow":false},{"type":"PathNode","matrix":
    {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328
    55618,"dy":133.20628077515605},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star
    tX":126.25,"startY":127.50445838342671,"closed":true,"segments":
    [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127
    .50445838342671,"r":
    [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439
    5]},
    {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049
    55,"r":
    [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},
    {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":
    [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337
    9]},
    {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834
    2671,"r":
    [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}
    ],"shadow":false},{"type":"TextNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children":
    [],"
    fi
    llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20},
    {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children":
    [],"
    fi
    llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20},
    {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":
    [],"arrowSize":10,"path":{"type":"PathNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":464,"dy":-3},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"
    ✘
    75
    

    View Slide

93. {"type":"PathNode","matrix":
    {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948
    628471,"dy":371.5686591257294},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,
    "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":
    [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415
    6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":
    [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971
    7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":
    [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]}
    ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":
    [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],
    "shadow":false},{"type":"PathNode","matrix":
    {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328
    55618,"dy":133.20628077515605},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star
    tX":126.25,"startY":127.50445838342671,"closed":true,"segments":
    [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127
    .50445838342671,"r":
    [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439
    5]},
    {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049
    55,"r":
    [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},
    {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":
    [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337
    9]},
    {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834
    2671,"r":
    [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}
    ],"shadow":false},{"type":"TextNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children":
    [],"
    fi
    llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20},
    {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children":
    [],"
    fi
    llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20},
    {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":
    [],"arrowSize":10,"path":{"type":"PathNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":464,"dy":-3},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"
    What is the smallest failure inducing input?
    76
    Delta Debugging
    

    View Slide

94. {"type":"PathNode","matrix":
    {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948
    628471,"dy":371.5686591257294},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,
    "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":
    [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415
    6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":
    [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971
    7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":
    [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]}
    ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":
    [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],
    "shadow":false},{"type":"PathNode","matrix":
    {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328
    55618,"dy":133.20628077515605},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star
    tX":126.25,"startY":127.50445838342671,"closed":true,"segments":
    [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127
    .50445838342671,"r":
    [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439
    5]},
    {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049
    55,"r":
    [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},
    {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":
    [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337
    9]},
    {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834
    2671,"r":
    [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}
    ],"shadow":false},{"type":"TextNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children":
    [],"
    fi
    llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20},
    {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children":
    [],"
    fi
    llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20},
    {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":
    [],"arrowSize":10,"path":{"type":"PathNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":464,"dy":-3},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3," 77
    Grammar Based Delta Debugging
    

    View Slide

95. {"type":"PathNode","matrix":
    {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948
    628471,"dy":371.5686591257294},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,
    "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":
    [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415
    6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":
    [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971
    7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":
    [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]}
    ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":
    [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],
    "shadow":false},{"type":"PathNode","matrix":
    {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328
    55618,"dy":133.20628077515605},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star
    tX":126.25,"startY":127.50445838342671,"closed":true,"segments":
    [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127
    .50445838342671,"r":
    [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439
    5]},
    {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049
    55,"r":
    [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},
    {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":
    [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337
    9]},
    {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834
    2671,"r":
    [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}
    ],"shadow":false},{"type":"TextNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children":
    [],"
    fi
    llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20},
    {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children":
    [],"
    fi
    llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20},
    {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":
    [],"arrowSize":10,"path":{"type":"PathNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":464,"dy":-3},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"
    ::= 

    ::= 

    | 

    | 

    | 

    | `true`
    
    
    | `false
    `
    
    | `null
    `
    
    ::= `"` `"
    `
    
    | `""
    `
    
    ::=
    
    
    |
    ::= `{``}
    `
    
    | `{}`
    ::= `,`>
    
    |
    ::= `:`
    ::= `[``]
    `
    
    | `[]`
    ::= `,`>
    
    |
    ::= >
    
    ::=
    
    
    |
    77
    Grammar Based Delta Debugging
    

    View Slide

96. {"":[]}
    78
    Grammar Based Delta Debugging
    {"type":"PathNode","matrix":
    {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948
    628471,"dy":371.5686591257294},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,
    "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":
    [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415
    6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":
    [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971
    7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":
    [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]}
    ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":
    [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],
    "shadow":false},{"type":"PathNode","matrix":
    {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328
    55618,"dy":133.20628077515605},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star
    tX":126.25,"startY":127.50445838342671,"closed":true,"segments":
    [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127
    .50445838342671,"r":
    [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439
    5]},
    {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049
    55,"r":
    [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},
    {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":
    [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337
    9]},
    {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834
    2671,"r":
    [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}
    ],"shadow":false},{"type":"TextNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children":
    [],"
    fi
    llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20},
    {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children":
    [],"
    fi
    llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20},
    {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":
    [],"arrowSize":10,"path":{"type":"PathNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":464,"dy":-3},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"
    Test Minimization
    

    View Slide

97. {"":[]} ✘
    78
    Grammar Based Delta Debugging
    {"type":"PathNode","matrix":
    {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948
    628471,"dy":371.5686591257294},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,
    "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":
    [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415
    6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":
    [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971
    7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":
    [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]}
    ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":
    [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],
    "shadow":false},{"type":"PathNode","matrix":
    {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328
    55618,"dy":133.20628077515605},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star
    tX":126.25,"startY":127.50445838342671,"closed":true,"segments":
    [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127
    .50445838342671,"r":
    [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439
    5]},
    {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049
    55,"r":
    [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},
    {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":
    [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337
    9]},
    {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834
    2671,"r":
    [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}
    ],"shadow":false},{"type":"TextNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children":
    [],"
    fi
    llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20},
    {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children":
    [],"
    fi
    llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20},
    {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":
    [],"arrowSize":10,"path":{"type":"PathNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":464,"dy":-3},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"
    Test Minimization
    

    View Slide

98. 79
    Grammar Based Delta Debugging
    {"type":"PathNode","matrix":
    {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948
    628471,"dy":371.5686591257294},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,
    "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":
    [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415
    6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":
    [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971
    7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":
    [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]}
    ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":
    [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],
    "shadow":false},{"type":"PathNode","matrix":
    {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328
    55618,"dy":133.20628077515605},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star
    tX":126.25,"startY":127.50445838342671,"closed":true,"segments":
    [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127
    .50445838342671,"r":
    [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439
    5]},
    {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049
    55,"r":
    [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},
    {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":
    [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337
    9]},
    {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834
    2671,"r":
    [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}
    ],"shadow":false},{"type":"TextNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children":
    [],"
    fi
    llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20},
    {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children":
    [],"
    fi
    llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20},
    {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":
    [],"arrowSize":10,"path":{"type":"PathNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":464,"dy":-3},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"
    Why?
    {"":[]}
    

    View Slide

99. 79
    Grammar Based Delta Debugging
    {"type":"PathNode","matrix":
    {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948
    628471,"dy":371.5686591257294},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,
    "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":
    [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415
    6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":
    [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971
    7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":
    [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]}
    ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":
    [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],
    "shadow":false},{"type":"PathNode","matrix":
    {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328
    55618,"dy":133.20628077515605},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star
    tX":126.25,"startY":127.50445838342671,"closed":true,"segments":
    [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127
    .50445838342671,"r":
    [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439
    5]},
    {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049
    55,"r":
    [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},
    {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":
    [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337
    9]},
    {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834
    2671,"r":
    [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}
    ],"shadow":false},{"type":"TextNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children":
    [],"
    fi
    llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20},
    {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children":
    [],"
    fi
    llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20},
    {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":
    [],"arrowSize":10,"path":{"type":"PathNode","matrix":
    {"m11":1,"m12":0,"m21":0,"m22":1,"dx":464,"dy":-3},"children":
    [],"strokeStyle":"#000000","
    fi
    llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"
    Why?
    [12345]
    {"":[]}
    

    View Slide

100. 79
     Grammar Based Delta Debugging
     {"type":"PathNode","matrix":
     {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948
     628471,"dy":371.5686591257294},"children":
     [],"strokeStyle":"#000000","
     fi
     llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,
     "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":
     [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415
     6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":
     [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971
     7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":
     [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]}
     ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":
     [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],
     "shadow":false},{"type":"PathNode","matrix":
     {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328
     55618,"dy":133.20628077515605},"children":
     [],"strokeStyle":"#000000","
     fi
     llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star
     tX":126.25,"startY":127.50445838342671,"closed":true,"segments":
     [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127
     .50445838342671,"r":
     [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439
     5]},
     {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049
     55,"r":
     [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},
     {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":
     [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337
     9]},
     {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834
     2671,"r":
     [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}
     ],"shadow":false},{"type":"TextNode","matrix":
     {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children":
     [],"
     fi
     llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20},
     {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children":
     [],"
     fi
     llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20},
     {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":
     [],"arrowSize":10,"path":{"type":"PathNode","matrix":
     {"m11":1,"m12":0,"m21":0,"m22":1,"dx":464,"dy":-3},"children":
     [],"strokeStyle":"#000000","
     fi
     llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"
     Why?
     [12345]
     {"":[]}
     {"":0}
     

     View Slide

101. 79
     Grammar Based Delta Debugging
     {"type":"PathNode","matrix":
     {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948
     628471,"dy":371.5686591257294},"children":
     [],"strokeStyle":"#000000","
     fi
     llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5,
     "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r":
     [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415
     6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r":
     [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971
     7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r":
     [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]}
     ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r":
     [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}],
     "shadow":false},{"type":"PathNode","matrix":
     {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328
     55618,"dy":133.20628077515605},"children":
     [],"strokeStyle":"#000000","
     fi
     llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star
     tX":126.25,"startY":127.50445838342671,"closed":true,"segments":
     [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127
     .50445838342671,"r":
     [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439
     5]},
     {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049
     55,"r":
     [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]},
     {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r":
     [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337
     9]},
     {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834
     2671,"r":
     [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]}
     ],"shadow":false},{"type":"TextNode","matrix":
     {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children":
     [],"
     fi
     llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20},
     {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children":
     [],"
     fi
     llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20},
     {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children":
     [],"arrowSize":10,"path":{"type":"PathNode","matrix":
     {"m11":1,"m12":0,"m21":0,"m22":1,"dx":464,"dy":-3},"children":
     [],"strokeStyle":"#000000","
     fi
     llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"
     Why?
     [12345]
     {"":[]}
     {"":0}
     {"x":[]}
     

     View Slide

102. 80
     DDSET
     Gopinath, Kampmann, Havrikov, Soremekun, and Zeller. Abstracting Failure Inducing Inputs. ISSTA 2020.
     https://github.com/vrthra/ddset
     

     View Slide

103. {"": []}
     DDSET:
     81
     ::= 

     ::= 

     | 

     | 

     | 

     | `true`
     
     
     | `false
     `
     
     | `null
     `
     
     ::= `"` `"
     `
     
     | `""
     `
     
     ::=
     
     
     |
     ::= `{``}
     `
     
     | `{}`
     ::= `,`>
     
     |
     ::= `:`
     ::= `[``]
     `
     
     | `[]`
     ::= `,`>
     
     |
     ::= >
     
     ::=
     
     
     |
     

     View Slide

104. {"": []}
     82
     DDSET:
     ::= 

     ::= 

     | 

     | 

     | 

     | `true`
     
     
     | `false
     `
     
     | `null
     `
     
     ::= `"` `"
     `
     
     | `""
     `
     
     ::=
     
     
     |
     ::= `{``}
     `
     
     | `{}`
     ::= `,`>
     
     |
     ::= `:`
     ::= `[``]
     `
     
     | `[]`
     ::= `,`>
     
     |
     ::= >
     
     ::=
     
     
     |
     

     View Slide

105. {"": []}
     {"7897A": []}
     {"klnm,.qer;dfs?P":[]}
     {"123KOUIJ!qR30578950":[]} 82
     DDSET:
     ::= 

     ::= 

     | 

     | 

     | 

     | `true`
     
     
     | `false
     `
     
     | `null
     `
     
     ::= `"` `"
     `
     
     | `""
     `
     
     ::=
     
     
     |
     ::= `{``}
     `
     
     | `{}`
     ::= `,`>
     
     |
     ::= `:`
     ::= `[``]
     `
     
     | `[]`
     ::= `,`>
     
     |
     ::= >
     
     ::=
     
     
     |
     

     View Slide

106. {"": []}
     83
     DDSET:
     ::= 

     ::= 

     | 

     | 

     | 

     | `true`
     
     
     | `false
     `
     
     | `null
     `
     
     ::= `"` `"
     `
     
     | `""
     `
     
     ::=
     
     
     |
     ::= `{``}
     `
     
     | `{}`
     ::= `,`>
     
     |
     ::= `:`
     ::= `[``]
     `
     
     | `[]`
     ::= `,`>
     
     |
     ::= >
     
     ::=
     
     
     |
     

     View Slide

107. {"": []}
     {"": true}
     {"":[1,2,445,"x"]}
     {"":{"PQ":[true, false, 223,"a"]}} 83
     DDSET:
     ::= 

     ::= 

     | 

     | 

     | 

     | `true`
     
     
     | `false
     `
     
     | `null
     `
     
     ::= `"` `"
     `
     
     | `""
     `
     
     ::=
     
     
     |
     ::= `{``}
     `
     
     | `{}`
     ::= `,`>
     
     |
     ::= `:`
     ::= `[``]
     `
     
     | `[]`
     ::= `,`>
     
     |
     ::= >
     
     ::=
     
     
     |
     

     View Slide

108. {"": []}
     Abstraction
     {"": }
     Abstract Input
     84
     DDSET:
     ::= 

     ::= 

     | 

     | 

     | 

     | `true`
     
     
     | `false
     `
     
     | `null
     `
     
     ::= `"` `"
     `
     
     | `""
     `
     
     ::=
     
     
     |
     ::= `{``}
     `
     
     | `{}`
     ::= `,`>
     
     |
     ::= `:`
     ::= `[``]
     `
     
     | `[]`
     ::= `,`>
     
     |
     ::= >
     
     ::=
     
     
     |
     

     View Slide

109. {"": }
     Abstract Input
     {"": []}
     Minimized Input
     85
     ::= 

     ::= 

     | 

     | 

     | 

     | `true`
     
     
     | `false
     `
     
     | `null
     `
     
     ::= `"` `"
     `
     
     | `""
     `
     
     ::=
     
     
     |
     ::= `{``}
     `
     
     | `{}`
     ::= `,`>
     
     |
     ::= `:`
     ::= `[``]
     `
     
     | `[]`
     ::= `,`>
     
     |
     ::= >
     
     ::=
     
     
     |
     

     View Slide

110. Issue 2842 from Closure
     var A = class extends (class {}){};
     Issue 2937 from Closure
     {while ((l_0)){ if ((l_0)) {break;;var l_0; continue }0}}
     var {baz:{} = baz => {}} = baz => {};
     Issue 385 from Rhino
     const [y,y] = [];
     Issue 386 from Rhino
     86
     

     View Slide

111. Issue 2842 from Closure
     var A = class extends (class {}){};
     Issue 2937 from Closure
     {while ((l_0)){ if ((l_0)) {break;;var l_0; continue }0}}
     var {baz:{} = baz => {}} = baz => {};
     Issue 385 from Rhino
     const [y,y] = [];
     Issue 386 from Rhino
     = class extends (class {}){}
     var {<$Id1>:{} = <$Id1> => {}} ;
     {while ((<$Id1>)){ if ((<$Id1>)) {break;;var <$Id1>; continue }0}}
     const [<$Id1>,<$Id1>] = []
     86
     

     View Slide

112. {"": }
     Abstract Input
     {"": []}
     Minimized Input
     87
     ::= 

     ::= 

     | 

     | 

     | 

     | `true`
     
     
     | `false
     `
     
     | `null
     `
     
     ::= `"` `"
     `
     
     | `""
     `
     
     ::=
     
     
     |
     ::= `{``}
     `
     
     | `{}`
     ::= `,`>
     
     |
     ::= `:`
     ::= `[``]
     `
     
     | `[]`
     ::= `,`>
     
     |
     ::= >
     
     ::=
     
     
     |
     Evocative Grammar
     

     View Slide

113. {"": }
     Abstract Input
     {"": []}
     Minimized Input
     87
     ::= 

     ::= 

     | 

     | 

     | 

     | `true`
     
     
     | `false
     `
     
     | `null
     `
     
     ::= `"` `"
     `
     
     | `""
     `
     
     ::=
     
     
     |
     ::= `{``}
     `
     
     | `{}`
     ::= `,`>
     
     |
     ::= `:`
     ::= `[``]
     `
     
     | `[]`
     ::= `,`>
     
     |
     ::= >
     
     ::=
     
     
     |
     
     
     
     where is "":
     Evocative Pattern
     Evocative Grammar
     

     View Slide

114. {"": }
     Abstract Input
     {"": []}
     Minimized Input
     87
     ::= 

     ::= 

     | 

     | 

     | 

     | `true`
     
     
     | `false
     `
     
     | `null
     `
     
     ::= `"` `"
     `
     
     | `""
     `
     
     ::=
     
     
     |
     ::= `{``}
     `
     
     | `{}`
     ::= `,`>
     
     |
     ::= `:`
     ::= `[``]
     `
     
     | `[]`
     ::= `,`>
     
     |
     ::= >
     
     ::=
     
     
     |
     ::= 

     ::= 

     | >
     
     | 

     | 

     ::= `{``}`

     ::=
     
     
     | `,`>
     
     | `,`

     ::= `:`>
     
     | `:`

     ::= `[``]`

     ::= >
     
     | `,`>
     
     | `,`>
     
     ::= `""
     `
     
     ::= 

     ::= 

     | >
     
     | 

     | 

     | `true` | `false` | `null`

     ::= `{``}` | `{}`

     ::= | `,`

     ::= `:`

     ::= `[``]` | `[]`

     ::= | `,`

     ::= `"` `"` | `""`

     ::= 

     ::= [A-Za-z0-9]

     ::= 

     ::= | 

     ::= [0-9]
     
     
     
     where is "":
     Evocative Pattern
     Evocative Grammar
     

     View Slide

115. 88
     generate()
     ::= 

     ::= 

     | >
     
     | 

     | 

     ::= `{``}`

     ::=
     
     
     | `,`>
     
     | `,`

     ::= `:`>
     
     | `:`

     ::= `[``]`

     ::= >
     
     | `,`>
     
     | `,`>
     
     ::= `""
     `
     
     ::= 

     ::= 

     | >
     
     | 

     | 

     | `true` | `false` | `null`

     ::= `{``}` | `{}`

     ::= | `,`

     ::= `:`

     ::= `[``]` | `[]`

     ::= | `,`

     ::= `"` `"` | `""`

     ::= 

     ::= [A-Za-z0-9]

     ::= 

     ::= | 

     ::= [0-9]
     
     
     
     where is "":
     Evocative Grammar
     

     View Slide

116. 88
     {"": 100}
     {"": [343,{},44998]}
     [{"": {"xxy":44998, {"b":[1,2,3]}}},[],[]]
     {"_": {"ket":[], {"":[],"y",[[],[1,2,3,455,6]]}}}
     {".":[{3243435656:"xy,zzzpqiu"},[{"":[112]},{"d":[[]]},{}]]}
     [{"": [1,2,3,4]}]
     {"pqr": {"": [1,2,3,4]}, "abc":[]}
     [[1132],{"xx":[{6:"dafjli;y,zzzdfaiu"},[{"__":[1{}{}]},{"":[[444456]]},{}]]}
     generate()
     ✘
     ✘
     ✘
     ✘
     ✘
     ✘
     ✘
     ✘
     ::= 

     ::= 

     | >
     
     | 

     | 

     ::= `{``}`

     ::=
     
     
     | `,`>
     
     | `,`

     ::= `:`>
     
     | `:`

     ::= `[``]`

     ::= >
     
     | `,`>
     
     | `,`>
     
     ::= `""
     `
     
     ::= 

     ::= 

     | >
     
     | 

     | 

     | `true` | `false` | `null`

     ::= `{``}` | `{}`

     ::= | `,`

     ::= `:`

     ::= `[``]` | `[]`

     ::= | `,`

     ::= `"` `"` | `""`

     ::= 

     ::= [A-Za-z0-9]

     ::= 

     ::= | 

     ::= [0-9]
     
     
     
     where is "":
     Evocative Grammar
     

     View Slide

117. 89
     
     
     
     where is "":
     1. We can produce any and all instances of the
     failure inducing pattern
     .
     
     2. We can recognize any input that contains the
     failure inducing pattern
     .
     
     3. The grammar will reject any input that doesn't
     contain the failure inducing pattern.
     ::= 

     ::= 

     | >
     
     | 

     | 

     ::= `{``}`

     ::=
     
     
     | `,`>
     
     | `,`

     ::= `:`>
     
     | `:`

     ::= `[``]`

     ::= >
     
     | `,`>
     
     | `,`>
     
     ::= `""
     `
     
     ::= 

     ::= 

     | >
     
     | 

     | 

     | `true` | `false` | `null`

     ::= `{``}` | `{}`

     ::= | `,`

     ::= `:`

     ::= `[``]` | `[]`

     ::= | `,`

     ::= `"` `"` | `""`

     ::= 

     ::= [A-Za-z0-9]

     ::= 

     ::= | 

     ::= [0-9]
     Evocative Grammar
     (Statistical Guarantees based on the accuracy of the evocative pattern)
     

     View Slide

118. 90
     Evocative Patterns
     What Does This Mean For You?
     • Automatically Mined
     
     
     • Produce Specialized Fuzzers
     
     
     • Open Source
     https://github.com/vrthra/ddset
     

     View Slide

119. Patterns of Failure
     91
     Current Research
     

     View Slide

120. 92
     Current Research
     Mining Input

     Formats
     Patterns of Failure
     Composable

     Fuzzers
     Specifying
     Constraints
     Learning from Inputs
     Input Coverage
     Fuzzing Digital
     Certificates
     

     View Slide

121. 93
     Current Research
     Composable

     Fuzzers
     

     View Slide

122. 
     
     
     where is "":
     if json.has_key("")
     :
     
     raise Exception()
     94
     Evocative Pattern
     

     View Slide

123. 
     
     
     where is "":
     if json.has_key("")
     :
     
     raise Exception()
     95
     Composing Evocative Patterns
     

     View Slide

124. 
     
     
     where is "":
     if json.has_key("")
     :
     
     raise Exception()
     if json.has_key_value(null)
     :
     
     raise Exception()
     95
     Composing Evocative Patterns
     

     View Slide

125. 
     
     
     where is "":
     if json.has_key("")
     :
     
     raise Exception()
     if json.has_key_value(null)
     :
     
     raise Exception()
     
     
     
     where is : null
     95
     Composing Evocative Patterns
     

     View Slide

126. if json.has_key("") and json.has_key_value(null)
     :
     
     raise Exception()
     96
     Composing Evocative Patterns
     

     View Slide

127. if json.has_key("") and json.has_key_value(null)
     :
     
     raise Exception()
     
     
     
     where is "":
     

     is : null
     96
     Composing Evocative Patterns
     

     View Slide

128. if json.has_key("") and not json.has_key_value(null)
     :
     
     raise Exception()
     97
     Composing Evocative Patterns
     

     View Slide

129. if json.has_key("") and not json.has_key_value(null)
     :
     
     raise Exception()
     
     
     
     where is "":
     
     
     is : null
     97
     Composing Evocative Patterns
     

     View Slide

130. if json.has_key("")
     :
     
     raise Exception(
     )
     
     if json.has_key_value(null)
     :
     
     raise Exception()
     98
     Composing Evocative Patterns
     

     View Slide

131. if json.has_key("")
     :
     
     raise Exception(
     )
     
     if json.has_key_value(null)
     :
     
     raise Exception()
     
     
     
     where is "":
     
     
     is : null
     98
     Composing Evocative Patterns
     

     View Slide

132. := >
     
     := >
     
     | >
     
     := '[' ']
     '
     
     := '{' '}
     '
     
     := >
     
     | ','>
     
     | ','>
     
     := >
     
     | ','>
     
     | ','>
     
     := ':'>
     
     | ':'>
     
     := >
     
     := 'false' | 'true
     '
     
     | | >
     
     | >
     
     := '[]' | '[' ']
     '
     
     := '{}' | '{' '}
     '
     
     := >
     
     | ','>
     
     := >
     
     | ','>
     
     := ':'>
     
     := 'false' | 'true
     '
     
     | | >
     
     | |
     
     
     
     where is "":
     
     
     is : null
     generate()
     99
     

     View Slide

133. := >
     
     := >
     
     | >
     
     := '[' ']
     '
     
     := '{' '}
     '
     
     := >
     
     | ','>
     
     | ','>
     
     := >
     
     | ','>
     
     | ','>
     
     := ':'>
     
     | ':'>
     
     := >
     
     := 'false' | 'true
     '
     
     | | >
     
     | >
     
     := '[]' | '[' ']
     '
     
     := '{}' | '{' '}
     '
     
     := >
     
     | ','>
     
     := >
     
     | ','>
     
     := ':'>
     
     := 'false' | 'true
     '
     
     | | >
     
     | |
     
     
     
     where is "":
     
     
     is : null
     {"": 100}
     {"": [343,{},44998]}
     [{"": {"xxy":44998, {"b":[1,2,3]}}},[],[]]
     {"_": {"ket":[], {"":[],"y",[[],[1,2,3,455,6]]}}}
     {".":[{3243435656:"xy,zzzpqiu"},[{"":[112]},{"d":[[]]},{}]]}
     [{"": [1,2,3,4]}]
     {"pqr": {"": [1,2,3,4]}, "abc":[]}
     [[1132],{"xx":[{6:"dafjli;y,zzzdfaiu"},[{"__":[1{}{}]},{"":[[444456]]},{}]]}
     generate()
     ✘
     ✘
     ✘
     ✘
     ✘
     ✘
     ✘
     ✘
     99
     

     View Slide

134. Issue 386 from Rhino
     var A = class extends (class {}){};
     Issue 2937 from Closure
     const [y,y] = [];
     var {baz:{} = baz => {}} = baz => {};
     Issue 385 from Rhino
     {while ((l_0)){ if ((l_0)) {break;;var l_0; continue }0}}
     Issue 2842 from Closure
     = class extends (class {}){}
     var {<$Id1>:{} = <$Id1> => {}} ;
     const [<$Id1>,<$Id1>] = []
     {while ((<$Id1>)){ if ((<$Id1>)) {break;;var <$Id1>; continue }0}}
     100
     

     View Slide

135. where
     
     
     is = class extends (class {}){}
     
     
     is {while ((<$Id1>)){ if ((<$Id1>)) {break;;var <$Id1>; continue }0}}
     
     
     is var {<$Id2>:{} = <$Id2> => {}} ;
     
     
     is const [<$Id3>,<$Id3>] = []
     Evocative Expressions
     101
     

     View Slide

136. where
     
     
     is = class extends (class {}){}
     
     
     is {while ((<$Id1>)){ if ((<$Id1>)) {break;;var <$Id1>; continue }0}}
     
     
     is var {<$Id2>:{} = <$Id2> => {}} ;
     
     
     is const [<$Id3>,<$Id3>] = []
     Evocative Expressions
     
     101
     

     View Slide

137. where
     
     
     is = class extends (class {}){}
     
     
     is {while ((<$Id1>)){ if ((<$Id1>)) {break;;var <$Id1>; continue }0}}
     
     
     is var {<$Id2>:{} = <$Id2> => {}} ;
     
     
     is const [<$Id3>,<$Id3>] = []
     Evocative Expressions
     
     
     101
     

     View Slide

138. where
     
     
     is = class extends (class {}){}
     
     
     is {while ((<$Id1>)){ if ((<$Id1>)) {break;;var <$Id1>; continue }0}}
     
     
     is var {<$Id2>:{} = <$Id2> => {}} ;
     
     
     is const [<$Id3>,<$Id3>] = []
     Evocative Expressions
     
     
     
     101
     

     View Slide

139. where
     
     
     is = class extends (class {}){}
     
     
     is {while ((<$Id1>)){ if ((<$Id1>)) {break;;var <$Id1>; continue }0}}
     
     
     is var {<$Id2>:{} = <$Id2> => {}} ;
     
     
     is const [<$Id3>,<$Id3>] = []
     Evocative Expressions
     
     
     
     
     101
     

     View Slide

140. Evocative Expressions for Focused Fuzzing
     def assign_admin_rights()
     :
     
     self.db_rights.add(MODIFY_DB
     )
     
     self.fs_rights = R
     W
     
     self.timeout = 6
     0
     
     self.deploy = Tru
     e
     
     def assign_guest_rights()
     :
     
     self.db_rights = [QUERY_DB
     ]
     
     self.fs_rights = Non
     e
     
     self.timeout = Non
     e
     
     self.deploy = Fals
     e
     
     def modify_db(stmt)
     :
     
     if ADMIN in self.db_rights
     :
     
     process(stmt
     )
     
     else
     :
     
     raise Error(
     )
     
     def query_db(stmt)
     :
     
     process(stmt
     )
     
     

     View Slide

141. 103
     Evocative Expressions for Focused Fuzzing
     def assign_admin_rights()
     :
     
     self.db_rights.add(MODIFY_DB
     )
     
     self.fs_rights = R
     W
     
     self.timeout = 6
     0
     
     self.deploy = Tru
     e
     
     def assign_guest_rights()
     :
     
     self.db_rights = [QUERY_DB
     ]
     
     self.fs_rights = Non
     e
     
     self.timeout = Non
     e
     
     self.deploy = Fals
     e
     
     def modify_db(stmt)
     :
     
     if ADMIN in self.db_rights
     :
     
     process(stmt
     )
     
     else
     :
     
     raise Error(
     )
     
     def query_db(stmt)
     :
     
     process(stmt
     )
     
     {"role" : "admin"}
     

     View Slide

142. 103
     
     
     
     where is "role": "admin"
     Evocative Expressions for Focused Fuzzing
     def assign_admin_rights()
     :
     
     self.db_rights.add(MODIFY_DB
     )
     
     self.fs_rights = R
     W
     
     self.timeout = 6
     0
     
     self.deploy = Tru
     e
     
     def assign_guest_rights()
     :
     
     self.db_rights = [QUERY_DB
     ]
     
     self.fs_rights = Non
     e
     
     self.timeout = Non
     e
     
     self.deploy = Fals
     e
     
     def modify_db(stmt)
     :
     
     if ADMIN in self.db_rights
     :
     
     process(stmt
     )
     
     else
     :
     
     raise Error(
     )
     
     def query_db(stmt)
     :
     
     process(stmt
     )
     
     {"role" : "admin"}
     

     View Slide

143. 104
     Evocative Expressions for Focused Fuzzing
     def assign_admin_rights()
     :
     
     self.db_rights.add(MODIFY_DB
     )
     
     self.fs_rights = R
     W
     
     self.timeout = 6
     0
     
     self.deploy = Tru
     e
     
     def assign_guest_rights()
     :
     
     self.db_rights = [QUERY_DB
     ]
     
     self.fs_rights = Non
     e
     
     self.timeout = Non
     e
     
     self.deploy = Fals
     e
     
     def modify_db(stmt)
     :
     
     if ADMIN in self.db_rights
     :
     
     process(stmt
     )
     
     else
     :
     
     raise Error(
     )
     
     def query_db(stmt)
     :
     
     process(stmt
     )
     
     {"method":"remove_table","args":["orders", "inventory"]}
     
     
     
     where is "method":"remove_table","args":
     

     View Slide

144. 105
     Evocative Expressions for Focused Fuzzing
     def assign_admin_rights()
     :
     
     self.db_rights.add(MODIFY_DB
     )
     
     self.fs_rights = R
     W
     
     self.timeout = 6
     0
     
     self.deploy = Tru
     e
     
     def assign_guest_rights()
     :
     
     self.db_rights = [QUERY_DB
     ]
     
     self.fs_rights = Non
     e
     
     self.timeout = Non
     e
     
     self.deploy = Fals
     e
     
     def modify_db(stmt)
     :
     
     if ADMIN in self.db_rights
     :
     
     process(stmt
     )
     
     else
     :
     
     raise Error(
     )
     
     def query_db(stmt)
     :
     
     process(stmt
     )
     
     
     
     
     where is "role": "admin"
     
     
     is {"method":"remove_table","args":}
     

     View Slide

145. 105
     Evocative Expressions for Focused Fuzzing
     def assign_admin_rights()
     :
     
     self.db_rights.add(MODIFY_DB
     )
     
     self.fs_rights = R
     W
     
     self.timeout = 6
     0
     
     self.deploy = Tru
     e
     
     def assign_guest_rights()
     :
     
     self.db_rights = [QUERY_DB
     ]
     
     self.fs_rights = Non
     e
     
     self.timeout = Non
     e
     
     self.deploy = Fals
     e
     
     def modify_db(stmt)
     :
     
     if ADMIN in self.db_rights
     :
     
     process(stmt
     )
     
     else
     :
     
     raise Error(
     )
     
     def query_db(stmt)
     :
     
     process(stmt
     )
     
     
     
     
     where is "role": "admin"
     
     
     is {"method":"remove_table","args":}
     {"method":"remove_table","args":["orders", "inventory"], "role":"admin"}
     {"role":"admin", "method":"remove_table","args":["orders", "inventory"]}
     {"method":"remove_table","role":"admin","args":["orders", "inventory"]}
     {"method":"remove_table","args":["orders","inventory",{"role":"admin"}]}
     

     View Slide

146. 106
     Evocative Expressions for Focused Fuzzing
     def assign_admin_rights()
     :
     
     self.db_rights.add(MODIFY_DB
     )
     
     self.fs_rights = R
     W
     
     self.timeout = 6
     0
     
     self.deploy = Tru
     e
     
     def assign_guest_rights()
     :
     
     self.db_rights = [QUERY_DB
     ]
     
     self.fs_rights = Non
     e
     
     self.timeout = Non
     e
     
     self.deploy = Fals
     e
     
     def modify_db(stmt)
     :
     
     if ADMIN in self.db_rights
     :
     
     process(stmt
     )
     
     else
     :
     
     raise Error(
     )
     
     def query_db(stmt)
     :
     
     process(stmt
     )
     
     
     
     
     where is "role": "admin"
     
     
     is {"method":"remove_table","args":}
     

     View Slide

147. 106
     Evocative Expressions for Focused Fuzzing
     def assign_admin_rights()
     :
     
     self.db_rights.add(MODIFY_DB
     )
     
     self.fs_rights = R
     W
     
     self.timeout = 6
     0
     
     self.deploy = Tru
     e
     
     def assign_guest_rights()
     :
     
     self.db_rights = [QUERY_DB
     ]
     
     self.fs_rights = Non
     e
     
     self.timeout = Non
     e
     
     self.deploy = Fals
     e
     
     def modify_db(stmt)
     :
     
     if ADMIN in self.db_rights
     :
     
     process(stmt
     )
     
     else
     :
     
     raise Error(
     )
     
     def query_db(stmt)
     :
     
     process(stmt
     )
     
     
     
     
     where is "role": "admin"
     
     
     is {"method":"remove_table","args":}
     {"method":"remove_table","args":["orders", "inventory"]}
     {"role":"guest", "method":"remove_table","args":{"role":"guest"}}
     {"method":"remove_table","role":"guest","args":["orders", "inventory"]}
     {"method":"remove_table","args":["orders","inventory",{"role":"guest"}]}
     

     View Slide

148. Evocative Expressions: Data Structures
     Algebraic Data Types
     107
     ::=
     struct mystruct
     {
     
     stype m1
     ;
     
     stype m2
     ;
     
     };
     union myunion
     {
     
     utype m1
     ;
     
     utype m2
     ;
     
     };
     ::=
     |
     Data Structures
     Context Free Grammar
     

     View Slide

149. 108
     Composable Fuzzers
     REST
     with a REST vulnerability
     and a SQL Injection
     SQL
     but doesn't Induce BugB
     BugB
     or cover function A
     FnA
     I want an XML fuzzer
     XML
     

     View Slide

150. XMLREST
     108
     Composable Fuzzers
     REST
     with a REST vulnerability
     and a SQL Injection
     SQL
     but doesn't Induce BugB
     BugB
     or cover function A
     FnA
     I want an XML fuzzer
     XML
     

     View Slide

151. XMLREST XMLSQL
     108
     Composable Fuzzers
     REST
     with a REST vulnerability
     and a SQL Injection
     SQL
     but doesn't Induce BugB
     BugB
     or cover function A
     FnA
     I want an XML fuzzer
     XML
     

     View Slide

152. XMLREST XMLBugB
     XMLSQL
     108
     Composable Fuzzers
     REST
     with a REST vulnerability
     and a SQL Injection
     SQL
     but doesn't Induce BugB
     BugB
     or cover function A
     FnA
     I want an XML fuzzer
     XML
     

     View Slide

153. XMLREST XMLFnA
     XMLBugB
     XMLSQL
     108
     Composable Fuzzers
     REST
     with a REST vulnerability
     and a SQL Injection
     SQL
     but doesn't Induce BugB
     BugB
     or cover function A
     FnA
     I want an XML fuzzer
     XML
     

     View Slide

154. XMLREST XMLFnA
     XMLBugB
     XMLSQL
     109
     Composable Fuzzers
     REST
     with a REST vulnerability
     and a SQL Injection
     SQL
     but doesn't Induce BugB
     BugB
     or cover function A
     FnA
     & & not[ ]
     |
     I want an XML fuzzer
     XML
     

     View Slide

155. 110
     Evocative Expressions = Composable Fuzzers
     I want a fuzzer that targets a REST server wit
     h
     
     • SQL Injections A,B, and
     C
     
     • But does not go through the input sanitizer code
     I want a fuzzer that targets a C compiler wit
     h
     
     • No undefined behaviors in the produced input
     s
     
     • But contain at least one function pointer declaration
     I want a fuzzer that targets a database wit
     h
     
     • Each input containing previously fixed bugs A and
     B
     
     • But does not induce a known bug
     C
     
     • And does not cover the function X in the database source code
     I want a fuzzer that targets a JSON parser wit
     h
     
     • Each input containing at least one known quirk from other parsers
     I want a fuzzer that targets my applicatio
     n
     
     • Each input exercising the code I just fixe
     d
     
     • And also other known bug pattern
     s
     
     • But does not consume resource A
     

     View Slide

156. 110
     Evocative Expressions = Composable Fuzzers
     I want a fuzzer that targets a REST server wit
     h
     
     • SQL Injections A,B, and
     C
     
     • But does not go through the input sanitizer code
     I want a fuzzer that targets a C compiler wit
     h
     
     • No undefined behaviors in the produced input
     s
     
     • But contain at least one function pointer declaration
     I want a fuzzer that targets a database wit
     h
     
     • Each input containing previously fixed bugs A and
     B
     
     • But does not induce a known bug
     C
     
     • And does not cover the function X in the database source code
     I want a fuzzer that targets a JSON parser wit
     h
     
     • Each input containing at least one known quirk from other parsers
     I want a fuzzer that targets my applicatio
     n
     
     • Each input exercising the code I just fixe
     d
     
     • And also other known bug pattern
     s
     
     • But does not consume resource A
     All w
     ithout w
     riting
     a
     single
     line
     of code
     

     View Slide

157. Gopinath, Nemati, Zeller. Input Algebras. ICSE 2021.
     Evocative Expressions
     https://rahul.gopinath.org/posts/
     111
     

     View Slide

158. 112
     Evocative Expressions
     What Does This Mean For You?
     • Mix and match specialized fuzzers
     
     
     • Use historical bugs
     
     
     • An ecosystem of targeted fuzzers
     
     
     • Open Source
     https://github.com/vrthra/Ewoks
     

     View Slide

159. 113
     Current Research
     Composable

     Fuzzers
     

     View Slide

160. 114
     Current Research
     Mining Input

     Formats
     Patterns of Failure
     Composable

     Fuzzers
     Specifying
     Constraints
     Learning from Inputs
     Input Coverage
     Fuzzing Digital
     Certificates
     

     View Slide

161. 115
     Current Research
     Mining Input

     Formats
     Patterns of Failure
     Composable

     Fuzzers
     Specifying
     Constraints
     Learning from Inputs
     Input Coverage
     Fuzzing Digital
     Certificates
     

     View Slide

162. View Slide