Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Inducing Subtle Mutations with Program Repair

Inducing Subtle Mutations with Program Repair

Mutation workshop 2021 Virtual

Rahul Gopinath

April 12, 2021
Tweet

More Decks by Rahul Gopinath

Other Decks in Research

Transcript

  1. Inducing Subtle Mutations with
    Program Repair
    Florian Schwander
    Rahul Gopinath
    Andreas Zeller
    CISPA Helmholtz Center for Information Security
    Best Paper
    Award

    View full-size slide

  2. Inducing Subtle Mutations with
    Program Repair
    Florian Schwander
    Rahul Gopinath
    Andreas Zeller
    CISPA Helmholtz Center for Information Security
    Best Paper
    Award

    View full-size slide

  3. https://www.json.org

    View full-size slide

  4. object
    { }
    { members }
    members
    pair
    pair , members
    pair
    string : value
    array
    [ ]
    [ elements ]
    elements
    value
    value , elements
    value
    string
    number
    object
    array
    true
    false
    null
    string
    " "
    " chars "
    chars
    char
    char chars
    char
    UNICODE \ [",\,CTRL]
    \" \\ \/ \b \f \n \r \t
    \u hex hex hex hex
    number
    int
    int frac
    int exp
    int frac exp
    int
    digit
    onenine digits
    - digit
    - onenine digits
    frac
    . digits
    exp
    e digits
    hex
    digit
    A - F
    a - f
    digits
    digit
    digit digits
    e
    e e+ e-
    E E+ E-
    https://www.json.org

    View full-size slide

  5. https://nvd.nist.gov/vuln/data-feeds
    JSON Vulnerability Feeds

    View full-size slide

  6. https://nvd.nist.gov/vuln/data-feeds
    JSON Vulnerability Feeds

    View full-size slide

  7. 5
    Parsing JSON is a Minefield
    http://seriot.ch/

    View full-size slide

  8. 5
    Parsing JSON is a Minefield
    http://seriot.ch/

    View full-size slide

  9. 5
    Parsing JSON is a Minefield
    http://seriot.ch/

    View full-size slide

  10. 5
    Parsing JSON is a Minefield
    http://seriot.ch/

    View full-size slide

  11. 5
    Parsing JSON is a Minefield
    http://seriot.ch/
    Expected
    Parse Fail (Expect Success)
    Parse Success (Expect Fail)
    Parse Success (Undefined)
    Parse Fail (Undefined)
    Parser Crash
    Timeout

    View full-size slide

  12. 5
    Parsing JSON is a Minefield
    http://seriot.ch/
    Expected
    Parse Fail (Expect Success)
    Parse Success (Expect Fail)
    Parse Success (Undefined)
    Parse Fail (Undefined)
    Parser Crash
    Timeout

    View full-size slide

  13. ::=
    ::= '"'
    | '['
    | '{'
    |
    | 'true'
    | 'false'
    | 'null'
    ::= +
    | + 'e' +
    ::= '+' | '-' | '.' | [0-9] | 'E' | 'e'
    ::= * '"'
    ::= ']'
    | (',')* ']'
    | ( ',' )+ (',' )* ']'
    ::= '}'
    | ( '"' ':' ',' )*
    '"' ':' '}'
    ::= ' ' | '!' | '#' | '$' | '%' | '&' | '''
    | '*' | '+' | '-' | ',' | '.' | '/' | ':' | ';'
    | '<' | '=' | '>' | '?' | '@' | '[' | ']' | '^'
    | '_', ''', | '{' | '|' | '}' | '~'
    | [A-Za-z0-9]
    | '\'
    ::= '"' | '/' | 'b' | 'f' | 'n' | 'r' | 't'

    View full-size slide

  14. ::=
    ::= '"'
    | '['
    | '{'
    |
    | 'true'
    | 'false'
    | 'null'
    ::= +
    | + 'e' +
    ::= '+' | '-' | '.' | [0-9] | 'E' | 'e'
    ::= * '"'
    ::= ']'
    | (',')* ']'
    | ( ',' )+ (',' )* ']'
    ::= '}'
    | ( '"' ':' ',' )*
    '"' ':' '}'
    ::= ' ' | '!' | '#' | '$' | '%' | '&' | '''
    | '*' | '+' | '-' | ',' | '.' | '/' | ':' | ';'
    | '<' | '=' | '>' | '?' | '@' | '[' | ']' | '^'
    | '_', ''', | '{' | '|' | '}' | '~'
    | [A-Za-z0-9]
    | '\'
    ::= '"' | '/' | 'b' | 'f' | 'n' | 'r' | 't'

    [ "1", "xx" ]
    { "" : [] }
    { "??_": null}
    738421343
    "A??3q43xre"
    { }





    View full-size slide

  15. ``All happy families are alike,
    each unhappy family path is unhappy in its own way.''
    (Leo Tolstoy)
    Anna Karenina

    View full-size slide

  16. ``All happy families paths are alike,
    each unhappy family path is unhappy in its own way.''
    (apologies to Leo Tolstoy)
    The Anna Karenina Principle

    View full-size slide

  17. Mutation testing to the rescue!

    View full-size slide

  18. Mutation testing to the rescue!

    View full-size slide

  19. = b2 4ac
    Mutation Testing

    View full-size slide

  20. d = b^2 - 4 * a * c
    Original
    = b2 4ac
    Mutation Testing

    View full-size slide

  21. d = b^3 - 4 * a * c
    d = b^2 + 4 * a * c
    d = b^2 - 4 + a * c
    Mutants
    d = b^2 - 4 * a * c
    Original
    = b2 4ac
    Mutation Testing

    View full-size slide

  22. d = b^3 - 4 * a * c
    d = b^2 + 4 * a * c
    d = b^2 - 4 + a * c
    Mutants
    d = b^2 - 4 * a * c
    Original
    (a = 0, b = 0, c = 0) => (d = 0)
    (a = 1, b = 1, c = 1) => (d = -3)
    (a = 0, b = 2, c = 0) => (d = 4)
    Mutants killed by test cases
    Test cases
    = b2 4ac
    Mutation Testing

    View full-size slide

  23. d = b^2 - 4 * a * c
    = b2 4ac
    Equivalent Mutants

    View full-size slide

  24. d = b^2 - 4 * a * c
    = b2 4ac
    d = (-b)^2 - 4 * a * c
    d = b^2 - 4 * (- a) * (-c)
    d = b^2 - 3 * a * c
    d = b^2 - 4 * a + c
    Equivalent Mutants

    View full-size slide

  25. d = b^2 - 4 * a * c
    = b2 4ac
    d = (-b)^2 - 4 * a * c
    d = b^2 - 4 * (- a) * (-c)
    d = b^2 - 3 * a * c
    d = b^2 - 4 * a + c
    Equivalent mutants
    Equivalent Mutants

    View full-size slide


  26. = b2 4ac
    Some Possible Mutants

    View full-size slide

  27. d = b^0 - 4 * a * c;

    d = b^1 - 4 * a * c;
    d = b^-1 - 4 * a * c;
    d = b^MAX - 4 * a * c;
    d = b^MIN - 4 * a * c;
    d = b - 4 * a * c;

    d = b ^ 4 * a * c;
    d = b^2 - 0 * a * c;

    d = b^2 - 1 * a * c;

    d = b^2 – (-1) * a * c;

    d = b^2 - MAX * a * c;

    d = b^2 - MIN * a * c;

    d = b^2 - 4 * a * c;

    d = b^2 - 4 * a * c;
    d = b^2 + 4 * a * c;

    d = b^2 * 4 * a * c;

    d = b^2 / 4 * a * c;

    d = b^2 ^ 4 * a * c;

    d = b^2 % 4 * a * c;
    d = b^2 << 4 * a * c;
    d = b^2 >> 4 * a * c;
    d = b^2 * 4 + a * c;

    d = b^2 * 4 - a * c;

    d = b^2 * 4 / a * c;

    d = b^2 * 4 ^ a * c;

    d = b^2 * 4 % a * c;
    d = b^2 * 4 << a * c;
    d = b^2 * 4 >> a * c;
    d = b^2 * 4 * a + c;

    d = b^2 * 4 * a - c;

    d = b^2 * 4 * a / c;

    d = b^2 * 4 * a ^ c;

    d = b^2 * 4 * a % c;
    d = b^2 * 4 * a << c;
    d = b^2 * 4 * a >> c;
    d = b + 2 - 4 * a * c;

    d = b - 2 - 4 * a * c;

    d = b * 2 - 4 * a * c;

    d = b / 2 - 4 * a * c;

    d = b % 2 - 4 * a * c;

    d = b << 2 - 4 * a * c;

    d = b >> 2 - 4 * a * c;


    = b2 4ac
    Some Possible Mutants

    View full-size slide

  28. d = b^0 - 4 * a * c;

    d = b^1 - 4 * a * c;
    d = b^-1 - 4 * a * c;
    d = b^MAX - 4 * a * c;
    d = b^MIN - 4 * a * c;
    d = b - 4 * a * c;

    d = b ^ 4 * a * c;
    d = b^2 - 0 * a * c;

    d = b^2 - 1 * a * c;

    d = b^2 – (-1) * a * c;

    d = b^2 - MAX * a * c;

    d = b^2 - MIN * a * c;

    d = b^2 - 4 * a * c;

    d = b^2 - 4 * a * c;
    d = b^2 + 4 * a * c;

    d = b^2 * 4 * a * c;

    d = b^2 / 4 * a * c;

    d = b^2 ^ 4 * a * c;

    d = b^2 % 4 * a * c;
    d = b^2 << 4 * a * c;
    d = b^2 >> 4 * a * c;
    d = b^2 * 4 + a * c;

    d = b^2 * 4 - a * c;

    d = b^2 * 4 / a * c;

    d = b^2 * 4 ^ a * c;

    d = b^2 * 4 % a * c;
    d = b^2 * 4 << a * c;
    d = b^2 * 4 >> a * c;
    d = b^2 * 4 * a + c;

    d = b^2 * 4 * a - c;

    d = b^2 * 4 * a / c;

    d = b^2 * 4 * a ^ c;

    d = b^2 * 4 * a % c;
    d = b^2 * 4 * a << c;
    d = b^2 * 4 * a >> c;
    d = b + 2 - 4 * a * c;

    d = b - 2 - 4 * a * c;

    d = b * 2 - 4 * a * c;

    d = b / 2 - 4 * a * c;

    d = b % 2 - 4 * a * c;

    d = b << 2 - 4 * a * c;

    d = b >> 2 - 4 * a * c;


    = b2 4ac
    Some Possible Mutants
    Traditional mutants are unusable as recommendations

    View full-size slide

  29. Requirement: Valid & Invalid Domain
    Program

    View full-size slide

  30. Requirement: Valid & Invalid Domain
    Program
    ✔ Accept

    View full-size slide

  31. Requirement: Valid & Invalid Domain
    Program

    ✔ Accept
    Reject

    View full-size slide

  32. Requirement: Valid & Invalid Domain
    JSON
    {}
    "}

    View full-size slide

  33. Requirement: Valid & Invalid Domain
    JSON
    {}
    "}
    ✔ Accept

    View full-size slide

  34. Requirement: Valid & Invalid Domain
    JSON
    {}
    "}
    ✔ Accept
    ✘ Reject

    View full-size slide

  35. JSON
    Repair to Accept Invalid Inputs
    {}
    "}
    ✔ Accept

    View full-size slide

  36. JSON
    Repair to Accept Invalid Inputs
    {}
    "}
    ✔ Accept
    ✔ Accept

    View full-size slide

  37. JSON
    Repair to Accept Invalid Inputs
    {}
    "}
    ✔ Accept
    ✔ Accept
    • Identify tests for unhappy paths

    View full-size slide

  38. JSON
    Repair to Accept Invalid Inputs
    {}
    "}
    ✔ Accept
    ✔ Accept
    • Identify tests for unhappy paths
    • No equivalent mutants

    View full-size slide

  39. Generating Valid Inputs
    Program ✔
    (Valid Input)

    View full-size slide

  40. Generating Valid Inputs
    • Existing tests
    Program ✔
    (Valid Input)

    View full-size slide

  41. Generating Valid Inputs
    • Existing tests
    • Specifications
    Program ✔
    (Valid Input)

    View full-size slide

  42. Generating Valid Inputs
    • Existing tests
    • Specifications
    • Monotonic prefix generation
    (2019 PLDI)
    Program ✔
    (Valid Input)

    View full-size slide

  43. Generating Invalid Inputs
    Program ✔
    (Valid Input)

    View full-size slide

  44. Generating Invalid Inputs
    Program ✔
    (Valid Input)
    Program ✘
    (Invalid Input)
    Input Mutations
    • trim
    • swap
    • delete
    • bit/byte flip

    View full-size slide

  45. Program Repair
    Make the invalid input follow the valid input's execution path
    (Valid Input)
    (Invalid Input)
    (accept)
    (reject)

    View full-size slide

  46. Program Repair
    Make the invalid input follow the valid input's execution path
    (Valid Input)
    (Invalid Input)
    (accept)
    (reject)

    View full-size slide

  47. Program Repair
    Make the invalid input follow the valid input's execution path
    (Valid Input)
    (Invalid Input)
    (accept)
    (reject)

    View full-size slide

  48. Program Repair
    Make the invalid input follow the valid input's execution path
    (Valid Input)
    (Invalid Input)
    (accept)
    (reject)

    View full-size slide

  49. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View full-size slide

  50. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,1,1)
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View full-size slide

  51. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,1,1)
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View full-size slide

  52. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,1,1)
    Example: Triangle Program

    1,1,1
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View full-size slide

  53. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View full-size slide

  54. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,2,1)
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View full-size slide

  55. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,2,1)
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View full-size slide

  56. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,2,1)
    Example: Triangle Program


    1,1,1
    1,2,1
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View full-size slide

  57. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,2,1)
    Example: Triangle Program


    1,1,1
    1,2,1
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View full-size slide

  58. def triangle(a, b, c):
    if a <= b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    <=
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View full-size slide

  59. def triangle(a, b, c):
    if a <= b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,2,1)
    <=
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View full-size slide

  60. def triangle(a, b, c):
    if a <= b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,2,1)
    <=
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View full-size slide

  61. def triangle(a, b, c):
    if a <= b:
    if b >= c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    <=
    >=
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View full-size slide

  62. def triangle(a, b, c):
    if a <= b:
    if b >= c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,2,1)
    <=
    >=
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View full-size slide

  63. def triangle(a, b, c):
    if a <= b:
    if b >= c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,2,1)
    <=
    >=
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    1,1,1
    1,2,1 ✔

    View full-size slide

  64. Test Suite Augmentation
    Possible Inputs
    Valid Inputs

    View full-size slide

  65. Test Suite Augmentation
    Possible Inputs
    Valid Inputs
    Existing test cases

    View full-size slide

  66. Test Suite Augmentation
    Possible Inputs
    Valid Inputs
    Existing test cases
    Augmented test cases

    View full-size slide

  67. Subjects
    Subject SLOC Stmt Cov%
    mathexpr 169 88
    urljava 230 89
    cgi 72 98
    xsum 13 100
    simplejson 1486 94
    ijson 309 99
    nayajson 546 88
    microjson 311 95

    View full-size slide

  68. Live Mutants
    Subject Live Relevant %Relevant
    mathexpr 91 79 86.8 %
    urljava 27 24 88.9 %
    cgi 80 41 51.3 %
    xsum 5 5 100 %
    simplejson 2 2 100 %
    ijson 13 9 69.2 %
    nayajson 14 11 78.6 %
    microjson 472 423 89.6 %

    View full-size slide

  69. Minimal Mutants (Uniqueness of Faults)
    log(minimal mutants)
    Mauris augmented Cosmic-ray
    Unaugmented Cosmic-ray

    View full-size slide

  70. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    <=
    >=
    return triangle(a, b, c)
    Future: Non-overlapping Domains

    View full-size slide

  71. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle(1,1,1) == Equivalent
    <=
    >=
    return triangle(a, b, c)
    Future: Non-overlapping Domains
    triangle(1,2,1) == Isosceles
    triangle(1,2,3) == Scalene

    View full-size slide

  72. Future: Better Repair
    JSON
    ""
    1
    {}
    []
    "]

    View full-size slide