Inducing Subtle Mutations with Program Repair

Mutation workshop 2021 Virtual

Transcript

1. Inducing Subtle Mutations with
   Program Repair
   Florian Schwander
   Rahul Gopinath
   Andreas Zeller
   CISPA Helmholtz Center for Information Security
   Best Paper
   Award
   

   View Slide

2. Inducing Subtle Mutations with
   Program Repair
   Florian Schwander
   Rahul Gopinath
   Andreas Zeller
   CISPA Helmholtz Center for Information Security
   Best Paper
   Award
   

   View Slide

3. https://www.json.org
   

   View Slide

4. object
   { }
   { members }
   members
   pair
   pair , members
   pair
   string : value
   array
   [ ]
   [ elements ]
   elements
   value
   value , elements
   value
   string
   number
   object
   array
   true
   false
   null
   string
   " "
   " chars "
   chars
   char
   char chars
   char
   UNICODE \ [",\,CTRL]
   \" \\ \/ \b \f \n \r \t
   \u hex hex hex hex
   number
   int
   int frac
   int exp
   int frac exp
   int
   digit
   onenine digits
   - digit
   - onenine digits
   frac
   . digits
   exp
   e digits
   hex
   digit
   A - F
   a - f
   digits
   digit
   digit digits
   e
   e e+ e-
   E E+ E-
   https://www.json.org
   

   View Slide

5. View Slide

6. https://nvd.nist.gov/vuln/data-feeds
   JSON Vulnerability Feeds
   

   View Slide

7. https://nvd.nist.gov/vuln/data-feeds
   JSON Vulnerability Feeds
   

   View Slide

8. 5
   Parsing JSON is a Minefield
   http://seriot.ch/
   

   View Slide

9. 5
   Parsing JSON is a Minefield
   http://seriot.ch/
   

   View Slide

10. 5
    Parsing JSON is a Minefield
    http://seriot.ch/
    

    View Slide

11. 5
    Parsing JSON is a Minefield
    http://seriot.ch/
    

    View Slide

12. 5
    Parsing JSON is a Minefield
    http://seriot.ch/
    Expected
    Parse Fail (Expect Success)
    Parse Success (Expect Fail)
    Parse Success (Undefined)
    Parse Fail (Undefined)
    Parser Crash
    Timeout
    

    View Slide

13. 5
    Parsing JSON is a Minefield
    http://seriot.ch/
    Expected
    Parse Fail (Expect Success)
    Parse Success (Expect Fail)
    Parse Success (Undefined)
    Parse Fail (Undefined)
    Parser Crash
    Timeout
    

    View Slide

14. View Slide

15. ::=
    ::= '"'
    | '['
    | '{'
    |
    | 'true'
    | 'false'
    | 'null'
    ::= +
    | + 'e' +
    ::= '+' | '-' | '.' | [0-9] | 'E' | 'e'
    ::= * '"'
    ::= ']'
    | (',')* ']'
    | ( ',' )+ (',' )* ']'
    ::= '}'
    | ( '"' ':' ',' )*
    '"' ':' '}'
    ::= ' ' | '!' | '#' | '$' | '%' | '&' | '''
    | '*' | '+' | '-' | ',' | '.' | '/' | ':' | ';'
    | '' | '?' | '@' | '[' | ']' | '^'
    | '_', ''', | '{' | '|' | '}' | '~'
    | [A-Za-z0-9]
    | '\'
    ::= '"' | '/' | 'b' | 'f' | 'n' | 'r' | 't'
    

    View Slide

16. ::=
    ::= '"'
    | '['
    | '{'
    |
    | 'true'
    | 'false'
    | 'null'
    ::= +
    | + 'e' +
    ::= '+' | '-' | '.' | [0-9] | 'E' | 'e'
    ::= * '"'
    ::= ']'
    | (',')* ']'
    | ( ',' )+ (',' )* ']'
    ::= '}'
    | ( '"' ':' ',' )*
    '"' ':' '}'
    ::= ' ' | '!' | '#' | '$' | '%' | '&' | '''
    | '*' | '+' | '-' | ',' | '.' | '/' | ':' | ';'
    | '' | '?' | '@' | '[' | ']' | '^'
    | '_', ''', | '{' | '|' | '}' | '~'
    | [A-Za-z0-9]
    | '\'
    ::= '"' | '/' | 'b' | 'f' | 'n' | 'r' | 't'
    ✓
    [ "1", "xx" ]
    { "" : [] }
    { "??_": null}
    738421343
    "A??3q43xre"
    { }
    ✓
    ✓
    ✓
    ✓
    ✓
    

    View Slide

17. ``All happy families are alike,
    each unhappy family path is unhappy in its own way.''
    (Leo Tolstoy)
    Anna Karenina
    

    View Slide

18. ``All happy families paths are alike,
    each unhappy family path is unhappy in its own way.''
    (apologies to Leo Tolstoy)
    The Anna Karenina Principle
    

    View Slide

19. Mutation testing to the rescue!
    

    View Slide

20. Mutation testing to the rescue!
    

    View Slide

21. = b2 4ac
    Mutation Testing
    

    View Slide

22. d = b^2 - 4 * a * c
    Original
    = b2 4ac
    Mutation Testing
    

    View Slide

23. d = b^3 - 4 * a * c
    d = b^2 + 4 * a * c
    d = b^2 - 4 + a * c
    Mutants
    d = b^2 - 4 * a * c
    Original
    = b2 4ac
    Mutation Testing
    

    View Slide

24. d = b^3 - 4 * a * c
    d = b^2 + 4 * a * c
    d = b^2 - 4 + a * c
    Mutants
    d = b^2 - 4 * a * c
    Original
    (a = 0, b = 0, c = 0) => (d = 0)
    (a = 1, b = 1, c = 1) => (d = -3)
    (a = 0, b = 2, c = 0) => (d = 4)
    Mutants killed by test cases
    Test cases
    = b2 4ac
    Mutation Testing
    

    View Slide

25. d = b^2 - 4 * a * c
    = b2 4ac
    Equivalent Mutants
    

    View Slide

26. d = b^2 - 4 * a * c
    = b2 4ac
    d = (-b)^2 - 4 * a * c
    d = b^2 - 4 * (- a) * (-c)
    d = b^2 - 3 * a * c
    d = b^2 - 4 * a + c
    Equivalent Mutants
    

    View Slide

27. d = b^2 - 4 * a * c
    = b2 4ac
    d = (-b)^2 - 4 * a * c
    d = b^2 - 4 * (- a) * (-c)
    d = b^2 - 3 * a * c
    d = b^2 - 4 * a + c
    Equivalent mutants
    Equivalent Mutants
    

    View Slide

28. …
    = b2 4ac
    Some Possible Mutants
    

    View Slide

29. d = b^0 - 4 * a * c;

    d = b^1 - 4 * a * c;
    d = b^-1 - 4 * a * c;
    d = b^MAX - 4 * a * c;
    d = b^MIN - 4 * a * c;
    d = b - 4 * a * c;

    d = b ^ 4 * a * c;
    d = b^2 - 0 * a * c;

    d = b^2 - 1 * a * c;

    d = b^2 – (-1) * a * c;

    d = b^2 - MAX * a * c;

    d = b^2 - MIN * a * c;

    d = b^2 - 4 * a * c;

    d = b^2 - 4 * a * c;
    d = b^2 + 4 * a * c;

    d = b^2 * 4 * a * c;

    d = b^2 / 4 * a * c;

    d = b^2 ^ 4 * a * c;

    d = b^2 % 4 * a * c;
    d = b^2 << 4 * a * c;
    d = b^2 >> 4 * a * c;
    d = b^2 * 4 + a * c;

    d = b^2 * 4 - a * c;

    d = b^2 * 4 / a * c;

    d = b^2 * 4 ^ a * c;

    d = b^2 * 4 % a * c;
    d = b^2 * 4 << a * c;
    d = b^2 * 4 >> a * c;
    d = b^2 * 4 * a + c;

    d = b^2 * 4 * a - c;

    d = b^2 * 4 * a / c;

    d = b^2 * 4 * a ^ c;

    d = b^2 * 4 * a % c;
    d = b^2 * 4 * a << c;
    d = b^2 * 4 * a >> c;
    d = b + 2 - 4 * a * c;

    d = b - 2 - 4 * a * c;

    d = b * 2 - 4 * a * c;

    d = b / 2 - 4 * a * c;

    d = b % 2 - 4 * a * c;

    d = b << 2 - 4 * a * c;

    d = b >> 2 - 4 * a * c;

    …
    = b2 4ac
    Some Possible Mutants
    

    View Slide

30. d = b^0 - 4 * a * c;

    d = b^1 - 4 * a * c;
    d = b^-1 - 4 * a * c;
    d = b^MAX - 4 * a * c;
    d = b^MIN - 4 * a * c;
    d = b - 4 * a * c;

    d = b ^ 4 * a * c;
    d = b^2 - 0 * a * c;

    d = b^2 - 1 * a * c;

    d = b^2 – (-1) * a * c;

    d = b^2 - MAX * a * c;

    d = b^2 - MIN * a * c;

    d = b^2 - 4 * a * c;

    d = b^2 - 4 * a * c;
    d = b^2 + 4 * a * c;

    d = b^2 * 4 * a * c;

    d = b^2 / 4 * a * c;

    d = b^2 ^ 4 * a * c;

    d = b^2 % 4 * a * c;
    d = b^2 << 4 * a * c;
    d = b^2 >> 4 * a * c;
    d = b^2 * 4 + a * c;

    d = b^2 * 4 - a * c;

    d = b^2 * 4 / a * c;

    d = b^2 * 4 ^ a * c;

    d = b^2 * 4 % a * c;
    d = b^2 * 4 << a * c;
    d = b^2 * 4 >> a * c;
    d = b^2 * 4 * a + c;

    d = b^2 * 4 * a - c;

    d = b^2 * 4 * a / c;

    d = b^2 * 4 * a ^ c;

    d = b^2 * 4 * a % c;
    d = b^2 * 4 * a << c;
    d = b^2 * 4 * a >> c;
    d = b + 2 - 4 * a * c;

    d = b - 2 - 4 * a * c;

    d = b * 2 - 4 * a * c;

    d = b / 2 - 4 * a * c;

    d = b % 2 - 4 * a * c;

    d = b << 2 - 4 * a * c;

    d = b >> 2 - 4 * a * c;

    …
    = b2 4ac
    Some Possible Mutants
    Traditional mutants are unusable as recommendations
    

    View Slide

31. Requirement: Valid & Invalid Domain
    Program
    

    View Slide

32. Requirement: Valid & Invalid Domain
    Program
    ✔ Accept
    

    View Slide

33. Requirement: Valid & Invalid Domain
    Program
    ✘
    ✔ Accept
    Reject
    

    View Slide

34. Requirement: Valid & Invalid Domain
    JSON
    {}
    "}
    

    View Slide

35. Requirement: Valid & Invalid Domain
    JSON
    {}
    "}
    ✔ Accept
    

    View Slide

36. Requirement: Valid & Invalid Domain
    JSON
    {}
    "}
    ✔ Accept
    ✘ Reject
    

    View Slide

37. JSON
    Repair to Accept Invalid Inputs
    {}
    "}
    ✔ Accept
    

    View Slide

38. JSON
    Repair to Accept Invalid Inputs
    {}
    "}
    ✔ Accept
    ✔ Accept
    

    View Slide

39. JSON
    Repair to Accept Invalid Inputs
    {}
    "}
    ✔ Accept
    ✔ Accept
    • Identify tests for unhappy paths
    

    View Slide

40. JSON
    Repair to Accept Invalid Inputs
    {}
    "}
    ✔ Accept
    ✔ Accept
    • Identify tests for unhappy paths
    • No equivalent mutants
    

    View Slide

41. Generating Valid Inputs
    Program ✔
    (Valid Input)
    

    View Slide

42. Generating Valid Inputs
    • Existing tests
    Program ✔
    (Valid Input)
    

    View Slide

43. Generating Valid Inputs
    • Existing tests
    • Specifications
    Program ✔
    (Valid Input)
    

    View Slide

44. Generating Valid Inputs
    • Existing tests
    • Specifications
    • Monotonic prefix generation
    (2019 PLDI)
    Program ✔
    (Valid Input)
    

    View Slide

45. Generating Invalid Inputs
    Program ✔
    (Valid Input)
    

    View Slide

46. Generating Invalid Inputs
    Program ✔
    (Valid Input)
    Program ✘
    (Invalid Input)
    Input Mutations
    • trim
    • swap
    • delete
    • bit/byte flip
    

    View Slide

47. Program Repair
    Make the invalid input follow the valid input's execution path
    (Valid Input)
    (Invalid Input)
    (accept)
    (reject)
    

    View Slide

48. Program Repair
    Make the invalid input follow the valid input's execution path
    (Valid Input)
    (Invalid Input)
    (accept)
    (reject)
    

    View Slide

49. Program Repair
    Make the invalid input follow the valid input's execution path
    (Valid Input)
    (Invalid Input)
    (accept)
    (reject)
    

    View Slide

50. Program Repair
    Make the invalid input follow the valid input's execution path
    (Valid Input)
    (Invalid Input)
    (accept)
    (reject)
    

    View Slide

51. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject
    

    View Slide

52. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,1,1)
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject
    

    View Slide

53. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,1,1)
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject
    

    View Slide

54. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,1,1)
    Example: Triangle Program
    ✔
    1,1,1
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject
    

    View Slide

55. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject
    

    View Slide

56. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,2,1)
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject
    

    View Slide

57. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,2,1)
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject
    

    View Slide

58. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,2,1)
    Example: Triangle Program
    ✔
    ✘
    1,1,1
    1,2,1
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject
    

    View Slide

59. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,2,1)
    Example: Triangle Program
    ✔
    ✘
    1,1,1
    1,2,1
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject
    

    View Slide

60. def triangle(a, b, c):
    if a <= b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    <=
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject
    

    View Slide

61. def triangle(a, b, c):
    if a <= b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,2,1)
    <=
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject
    

    View Slide

62. def triangle(a, b, c):
    if a <= b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,2,1)
    <=
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject
    

    View Slide

63. def triangle(a, b, c):
    if a <= b:
    if b >= c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    <=
    >=
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject
    

    View Slide

64. def triangle(a, b, c):
    if a <= b:
    if b >= c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,2,1)
    <=
    >=
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject
    

    View Slide

65. def triangle(a, b, c):
    if a <= b:
    if b >= c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle (1,2,1)
    <=
    >=
    Example: Triangle Program
    if triangle(a, b, c) == Equilateral:
    return Accept
    else:
    return Reject
    ✔
    1,1,1
    1,2,1 ✔
    

    View Slide

66. Test Suite Augmentation
    Possible Inputs
    Valid Inputs
    

    View Slide

67. Test Suite Augmentation
    Possible Inputs
    Valid Inputs
    Existing test cases
    

    View Slide

68. Test Suite Augmentation
    Possible Inputs
    Valid Inputs
    Existing test cases
    Augmented test cases
    

    View Slide

69. Evaluation
    

    View Slide

70. Subjects
    Subject SLOC Stmt Cov%
    mathexpr 169 88
    urljava 230 89
    cgi 72 98
    xsum 13 100
    simplejson 1486 94
    ijson 309 99
    nayajson 546 88
    microjson 311 95
    

    View Slide

71. Live Mutants
    Subject Live Relevant %Relevant
    mathexpr 91 79 86.8 %
    urljava 27 24 88.9 %
    cgi 80 41 51.3 %
    xsum 5 5 100 %
    simplejson 2 2 100 %
    ijson 13 9 69.2 %
    nayajson 14 11 78.6 %
    microjson 472 423 89.6 %
    

    View Slide

72. Minimal Mutants (Uniqueness of Faults)
    log(minimal mutants)
    Mauris augmented Cosmic-ray
    Unaugmented Cosmic-ray
    

    View Slide

73. Future
    

    View Slide

74. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    <=
    >=
    return triangle(a, b, c)
    Future: Non-overlapping Domains
    

    View Slide

75. def triangle(a, b, c):
    if a == b:
    if b == c:
    return Equilateral
    else:
    return Isosceles
    else:
    if b == c:
    return Isosceles
    else:
    if a == c:
    return Isosceles
    else:
    return Scalene
    triangle(1,1,1) == Equivalent
    <=
    >=
    return triangle(a, b, c)
    Future: Non-overlapping Domains
    triangle(1,2,1) == Isosceles
    triangle(1,2,3) == Scalene
    

    View Slide

76. Future: Better Repair
    JSON
    ""
    1
    {}
    []
    "]
    ✔
    

    View Slide

77. Summary
    

    View Slide

78. View Slide

79. View Slide

80. View Slide

81. View Slide

82. View Slide

83. View Slide

84. View Slide