Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Inducing Subtle Mutations with Program Repair

Inducing Subtle Mutations with Program Repair

Mutation workshop 2021 Virtual

Rahul Gopinath

April 12, 2021
Tweet

More Decks by Rahul Gopinath

Other Decks in Research

Transcript

  1. Inducing Subtle Mutations with
    Program Repair
    Florian Schwander
    Rahul Gopinath
    Andreas Zeller
    CISPA Helmholtz Center for Information Security
    Best Paper
    Award

    View Slide

  2. Inducing Subtle Mutations with
    Program Repair
    Florian Schwander
    Rahul Gopinath
    Andreas Zeller
    CISPA Helmholtz Center for Information Security
    Best Paper
    Award

    View Slide

  3. https://www.json.org

    View Slide

  4. object
    { }
    { members }
    members
    pair
    pair , members
    pair
    string : value
    array
    [ ]
    [ elements ]
    elements
    value
    value , elements
    value
    string
    number
    object
    array
    true
    false
    null
    string
    " "
    " chars "
    chars
    char
    char chars
    char
    UNICODE \ [",\,CTRL]
    \" \\ \/ \b \f \n \r \t
    \u hex hex hex hex
    number
    int
    int frac
    int exp
    int frac exp
    int
    digit
    onenine digits
    - digit
    - onenine digits
    frac
    . digits
    exp
    e digits
    hex
    digit
    A - F
    a - f
    digits
    digit
    digit digits
    e
    e e+ e-
    E E+ E-
    https://www.json.org

    View Slide

  5. View Slide

  6. https://nvd.nist.gov/vuln/data-feeds
    JSON Vulnerability Feeds

    View Slide

  7. https://nvd.nist.gov/vuln/data-feeds
    JSON Vulnerability Feeds

    View Slide

  8. 5
    Parsing JSON is a Minefield
    http://seriot.ch/

    View Slide

  9. 5
    Parsing JSON is a Minefield
    http://seriot.ch/

    View Slide

  10. 5
    Parsing JSON is a Minefield
    http://seriot.ch/

    View Slide

  11. 5
    Parsing JSON is a Minefield
    http://seriot.ch/

    View Slide

  12. 5
    Parsing JSON is a Minefield
    http://seriot.ch/
    Expected
    Parse Fail (Expect Success)
    Parse Success (Expect Fail)
    Parse Success (Undefined)
    Parse Fail (Undefined)
    Parser Crash
    Timeout

    View Slide

  13. 5
    Parsing JSON is a Minefield
    http://seriot.ch/
    Expected
    Parse Fail (Expect Success)
    Parse Success (Expect Fail)
    Parse Success (Undefined)
    Parse Fail (Undefined)
    Parser Crash
    Timeout

    View Slide

  14. View Slide

  15. ::=
    ::= '"'
    | '['
    | '{'
    |
    | 'true'
    | 'false'
    | 'null'
    ::= +
    | + 'e' +
    ::= '+' | '-' | '.' | [0-9] | 'E' | 'e'
    ::= * '"'
    ::= ']'
    | (',')* ']'
    | ( ',' )+ (',' )* ']'
    ::= '}'
    | ( '"' ':' ',' )*
    '"' ':' '}'
    ::= ' ' | '!' | '#' | '$' | '%' | '&' | '''
    | '*' | '+' | '-' | ',' | '.' | '/' | ':' | ';'
    | '' | '?' | '@' | '[' | ']' | '^'
    | '_', ''', | '{' | '|' | '}' | '~'
    | [A-Za-z0-9]
    | '\'
    ::= '"' | '/' | 'b' | 'f' | 'n' | 'r' | 't'

    View Slide

  16. ::=
    ::= '"'
    | '['
    | '{'
    |
    | 'true'
    | 'false'
    | 'null'
    ::= +
    | + 'e' +
    ::= '+' | '-' | '.' | [0-9] | 'E' | 'e'
    ::= * '"'
    ::= ']'
    | (',')* ']'
    | ( ',' )+ (',' )* ']'
    ::= '}'
    | ( '"' ':' ',' )*
    '"' ':' '}'
    ::= ' ' | '!' | '#' | '$' | '%' | '&' | '''
    | '*' | '+' | '-' | ',' | '.' | '/' | ':' | ';'
    | '' | '?' | '@' | '[' | ']' | '^'
    | '_', ''', | '{' | '|' | '}' | '~'
    | [A-Za-z0-9]
    | '\'
    ::= '"' | '/' | 'b' | 'f' | 'n' | 'r' | 't'

    [ "1", "xx" ]
    { "" : [] }
    { "??_": null}
    738421343
    "A??3q43xre"
    { }





    View Slide

  17. ``All happy families are alike,
    each unhappy family path is unhappy in its own way.''
    (Leo Tolstoy)
    Anna Karenina

    View Slide

  18. ``All happy families paths are alike,
    each unhappy family path is unhappy in its own way.''
    (apologies to Leo Tolstoy)
    The Anna Karenina Principle

    View Slide

  19. Mutation testing to the rescue!

    View Slide

  20. Mutation testing to the rescue!

    View Slide

  21. = b2 4ac
    Mutation Testing

    View Slide

  22. d = b^2 - 4 * a * c
    Original
    = b2 4ac
    Mutation Testing

    View Slide

  23. d = b^3 - 4 * a * c
    d = b^2 + 4 * a * c
    d = b^2 - 4 + a * c
    Mutants
    d = b^2 - 4 * a * c
    Original
    = b2 4ac
    Mutation Testing

    View Slide

  24. d = b^3 - 4 * a * c
    d = b^2 + 4 * a * c
    d = b^2 - 4 + a * c
    Mutants
    d = b^2 - 4 * a * c
    Original
    (a = 0, b = 0, c = 0) => (d = 0)
    (a = 1, b = 1, c = 1) => (d = -3)
    (a = 0, b = 2, c = 0) => (d = 4)
    Mutants killed by test cases
    Test cases
    = b2 4ac
    Mutation Testing

    View Slide

  25. d = b^2 - 4 * a * c
    = b2 4ac
    Equivalent Mutants

    View Slide

  26. d = b^2 - 4 * a * c
    = b2 4ac
    d = (-b)^2 - 4 * a * c
    d = b^2 - 4 * (- a) * (-c)
    d = b^2 - 3 * a * c
    d = b^2 - 4 * a + c
    Equivalent Mutants

    View Slide

  27. d = b^2 - 4 * a * c
    = b2 4ac
    d = (-b)^2 - 4 * a * c
    d = b^2 - 4 * (- a) * (-c)
    d = b^2 - 3 * a * c
    d = b^2 - 4 * a + c
    Equivalent mutants
    Equivalent Mutants

    View Slide


  28. = b2 4ac
    Some Possible Mutants

    View Slide

  29. d = b^0 - 4 * a * c;

    d = b^1 - 4 * a * c;
    d = b^-1 - 4 * a * c;
    d = b^MAX - 4 * a * c;
    d = b^MIN - 4 * a * c;
    d = b - 4 * a * c;

    d = b ^ 4 * a * c;
    d = b^2 - 0 * a * c;

    d = b^2 - 1 * a * c;

    d = b^2 – (-1) * a * c;

    d = b^2 - MAX * a * c;

    d = b^2 - MIN * a * c;

    d = b^2 - 4 * a * c;

    d = b^2 - 4 * a * c;
    d = b^2 + 4 * a * c;

    d = b^2 * 4 * a * c;

    d = b^2 / 4 * a * c;

    d = b^2 ^ 4 * a * c;

    d = b^2 % 4 * a * c;
    d = b^2 << 4 * a * c;
    d = b^2 >> 4 * a * c;
    d = b^2 * 4 + a * c;

    d = b^2 * 4 - a * c;

    d = b^2 * 4 / a * c;

    d = b^2 * 4 ^ a * c;

    d = b^2 * 4 % a * c;
    d = b^2 * 4 << a * c;
    d = b^2 * 4 >> a * c;
    d = b^2 * 4 * a + c;

    d = b^2 * 4 * a - c;

    d = b^2 * 4 * a / c;

    d = b^2 * 4 * a ^ c;

    d = b^2 * 4 * a % c;
    d = b^2 * 4 * a << c;
    d = b^2 * 4 * a >> c;
    d = b + 2 - 4 * a * c;

    d = b - 2 - 4 * a * c;

    d = b * 2 - 4 * a * c;

    d = b / 2 - 4 * a * c;

    d = b % 2 - 4 * a * c;

    d = b << 2 - 4 * a * c;

    d = b >> 2 - 4 * a * c;


    = b2 4ac
    Some Possible Mutants

    View Slide

  30. d = b^0 - 4 * a * c;

    d = b^1 - 4 * a * c;
    d = b^-1 - 4 * a * c;
    d = b^MAX - 4 * a * c;
    d = b^MIN - 4 * a * c;
    d = b - 4 * a * c;

    d = b ^ 4 * a * c;
    d = b^2 - 0 * a * c;

    d = b^2 - 1 * a * c;

    d = b^2 – (-1) * a * c;

    d = b^2 - MAX * a * c;

    d = b^2 - MIN * a * c;

    d = b^2 - 4 * a * c;

    d = b^2 - 4 * a * c;
    d = b^2 + 4 * a * c;

    d = b^2 * 4 * a * c;

    d = b^2 / 4 * a * c;

    d = b^2 ^ 4 * a * c;

    d = b^2 % 4 * a * c;
    d = b^2 << 4 * a * c;
    d = b^2 >> 4 * a * c;
    d = b^2 * 4 + a * c;

    d = b^2 * 4 - a * c;

    d = b^2 * 4 / a * c;

    d = b^2 * 4 ^ a * c;

    d = b^2 * 4 % a * c;
    d = b^2 * 4 << a * c;
    d = b^2 * 4 >> a * c;
    d = b^2 * 4 * a + c;

    d = b^2 * 4 * a - c;

    d = b^2 * 4 * a / c;

    d = b^2 * 4 * a ^ c;

    d = b^2 * 4 * a % c;
    d = b^2 * 4 * a << c;
    d = b^2 * 4 * a >> c;
    d = b + 2 - 4 * a * c;

    d = b - 2 - 4 * a * c;

    d = b * 2 - 4 * a * c;

    d = b / 2 - 4 * a * c;

    d = b % 2 - 4 * a * c;

    d = b << 2 - 4 * a * c;

    d = b >> 2 - 4 * a * c;


    = b2 4ac
    Some Possible Mutants
    Traditional mutants are unusable as recommendations

    View Slide

  31. Requirement: Valid & Invalid Domain
    Program

    View Slide

  32. Requirement: Valid & Invalid Domain
    Program
    ✔ Accept

    View Slide

  33. Requirement: Valid & Invalid Domain
    Program

    ✔ Accept
    Reject

    View Slide

  34. Requirement: Valid & Invalid Domain
    JSON
    {}
    "}

    View Slide

  35. Requirement: Valid & Invalid Domain
    JSON
    {}
    "}
    ✔ Accept

    View Slide

  36. Requirement: Valid & Invalid Domain
    JSON
    {}
    "}
    ✔ Accept
    ✘ Reject

    View Slide

  37. JSON
    Repair to Accept Invalid Inputs
    {}
    "}
    ✔ Accept

    View Slide

  38. JSON
    Repair to Accept Invalid Inputs
    {}
    "}
    ✔ Accept
    ✔ Accept

    View Slide

  39. JSON
    Repair to Accept Invalid Inputs
    {}
    "}
    ✔ Accept
    ✔ Accept
    • Identify tests for unhappy paths

    View Slide

  40. JSON
    Repair to Accept Invalid Inputs
    {}
    "}
    ✔ Accept
    ✔ Accept
    • Identify tests for unhappy paths
    • No equivalent mutants

    View Slide

  41. Generating Valid Inputs
    Program ✔
    (Valid Input)

    View Slide

  42. Generating Valid Inputs
    • Existing tests
    Program ✔
    (Valid Input)

    View Slide

  43. Generating Valid Inputs
    • Existing tests
    • Specifications
    Program ✔
    (Valid Input)

    View Slide

  44. Generating Valid Inputs
    • Existing tests
    • Specifications
    • Monotonic prefix generation
    (2019 PLDI)
    Program ✔
    (Valid Input)

    View Slide

  45. Generating Invalid Inputs
    Program ✔
    (Valid Input)

    View Slide

  46. Generating Invalid Inputs
    Program ✔
    (Valid Input)
    Program ✘
    (Invalid Input)
    Input Mutations
    • trim
    • swap
    • delete
    • bit/byte flip

    View Slide

  47. Program Repair
    Make the invalid input follow the valid input's execution path
    (Valid Input)
    (Invalid Input)
    (accept)
    (reject)

    View Slide

  48. Program Repair
    Make the invalid input follow the valid input's execution path
    (Valid Input)
    (Invalid Input)
    (accept)
    (reject)

    View Slide

  49. Program Repair
    Make the invalid input follow the valid input's execution path
    (Valid Input)
    (Invalid Input)
    (accept)
    (reject)

    View Slide

  50. Program Repair
    Make the invalid input follow the valid input's execution path
    (Valid Input)
    (Invalid Input)
    (accept)
    (reject)

    View Slide

  51. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View Slide

  52. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,1,1)
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View Slide

  53. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,1,1)
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View Slide

  54. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,1,1)
    Example: Triangle Program

    1,1,1
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View Slide

  55. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View Slide

  56. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,2,1)
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View Slide

  57. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,2,1)
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View Slide

  58. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,2,1)
    Example: Triangle Program


    1,1,1
    1,2,1
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View Slide

  59. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,2,1)
    Example: Triangle Program


    1,1,1
    1,2,1
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View Slide

  60. def triangle(a, b, c):
    if a <= b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    <=
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View Slide

  61. def triangle(a, b, c):
    if a <= b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,2,1)
    <=
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View Slide

  62. def triangle(a, b, c):
    if a <= b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,2,1)
    <=
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View Slide

  63. def triangle(a, b, c):
    if a <= b:
    if b >= c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    <=
    >=
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View Slide

  64. def triangle(a, b, c):
    if a <= b:
    if b >= c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,2,1)
    <=
    >=
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    View Slide

  65. def triangle(a, b, c):
    if a <= b:
    if b >= c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,2,1)
    <=
    >=
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject

    1,1,1
    1,2,1 ✔

    View Slide

  66. Test Suite Augmentation
    Possible Inputs
    Valid Inputs

    View Slide

  67. Test Suite Augmentation
    Possible Inputs
    Valid Inputs
    Existing test cases

    View Slide

  68. Test Suite Augmentation
    Possible Inputs
    Valid Inputs
    Existing test cases
    Augmented test cases

    View Slide

  69. Evaluation

    View Slide

  70. Subjects
    Subject SLOC Stmt Cov%
    mathexpr 169 88
    urljava 230 89
    cgi 72 98
    xsum 13 100
    simplejson 1486 94
    ijson 309 99
    nayajson 546 88
    microjson 311 95

    View Slide

  71. Live Mutants
    Subject Live Relevant %Relevant
    mathexpr 91 79 86.8 %
    urljava 27 24 88.9 %
    cgi 80 41 51.3 %
    xsum 5 5 100 %
    simplejson 2 2 100 %
    ijson 13 9 69.2 %
    nayajson 14 11 78.6 %
    microjson 472 423 89.6 %

    View Slide

  72. Minimal Mutants (Uniqueness of Faults)
    log(minimal mutants)
    Mauris augmented Cosmic-ray
    Unaugmented Cosmic-ray

    View Slide

  73. Future

    View Slide

  74. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    <=
    >=
    return triangle(a, b, c)
    Future: Non-overlapping Domains

    View Slide

  75. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle(1,1,1) == Equivalent
    <=
    >=
    return triangle(a, b, c)
    Future: Non-overlapping Domains
    triangle(1,2,1) == Isosceles
    triangle(1,2,3) == Scalene

    View Slide

  76. Future: Better Repair
    JSON
    ""
    1
    {}
    []
    "]

    View Slide

  77. Summary

    View Slide

  78. View Slide

  79. View Slide

  80. View Slide

  81. View Slide

  82. View Slide

  83. View Slide

  84. View Slide