Inducing Subtle Mutations with Program Repair

Mutation workshop 2021 Virtual

Transcript

1. Inducing Subtle Mutations with Program Repair Florian Schwander Rahul Gopinath Andreas Zeller CISPA Helmholtz Center for Information Security Best Paper Award
2. Inducing Subtle Mutations with Program Repair Florian Schwander Rahul Gopinath Andreas Zeller CISPA Helmholtz Center for Information Security Best Paper Award
3. https://www.json.org
4. object { } { members } members pair pair , members pair string : value array [ ] [ elements ] elements value value , elements value string number object array true false null string " " " chars " chars char char chars char UNICODE \ [",\,CTRL] \" \\ \/ \b \f \n \r \t \u hex hex hex hex number int int frac int exp int frac exp int digit onenine digits - digit - onenine digits frac . digits exp e digits hex digit A - F a - f digits digit digit digits e e e+ e- E E+ E- https://www.json.org
6. https://nvd.nist.gov/vuln/data-feeds JSON Vulnerability Feeds
7. https://nvd.nist.gov/vuln/data-feeds JSON Vulnerability Feeds
8. 5 Parsing JSON is a Minefield http://seriot.ch/
9. 5 Parsing JSON is a Minefield http://seriot.ch/
10. 5 Parsing JSON is a Minefield http://seriot.ch/
11. 5 Parsing JSON is a Minefield http://seriot.ch/
12. 5 Parsing JSON is a Minefield http://seriot.ch/ Expected Parse Fail (Expect Success) Parse Success (Expect Fail) Parse Success (Undefined) Parse Fail (Undefined) Parser Crash Timeout
13. 5 Parsing JSON is a Minefield http://seriot.ch/ Expected Parse Fail (Expect Success) Parse Success (Expect Fail) Parse Success (Undefined) Parse Fail (Undefined) Parser Crash Timeout
15. <START> ::= <json_raw> <json_raw> ::= '"' <json_string'> | '[' <json_list'> | '{' <json_dict'> | <json_number'> | 'true' | 'false' | 'null' <json_number'> ::= <json_number>+ | <json_number>+ 'e' <json_number>+ <json_number> ::= '+' | '-' | '.' | [0-9] | 'E' | 'e' <json_string'> ::= <json_string>* '"' <json_list'> ::= ']' | <json_raw> (','<json_raw>)* ']' | ( ',' <json_raw>)+ (',' <json_raw>)* ']' <json_dict'> ::= '}' | ( '"' <json_string'> ':' <json_raw> ',' )* '"'<json_string'> ':' <json_raw> '}' <json_string> ::= ' ' | '!' | '#' | '$' | '%' | '&' | ''' | '*' | '+' | '-' | ',' | '.' | '/' | ':' | ';' | '<' | '=' | '>' | '?' | '@' | '[' | ']' | '^' | '_', ''', | '{' | '|' | '}' | '~' | [A-Za-z0-9] | '\' <decode_escape> <decode_escape> ::= '"' | '/' | 'b' | 'f' | 'n' | 'r' | 't'
16. <START> ::= <json_raw> <json_raw> ::= '"' <json_string'> | '[' <json_list'> | '{' <json_dict'> | <json_number'> | 'true' | 'false' | 'null' <json_number'> ::= <json_number>+ | <json_number>+ 'e' <json_number>+ <json_number> ::= '+' | '-' | '.' | [0-9] | 'E' | 'e' <json_string'> ::= <json_string>* '"' <json_list'> ::= ']' | <json_raw> (','<json_raw>)* ']' | ( ',' <json_raw>)+ (',' <json_raw>)* ']' <json_dict'> ::= '}' | ( '"' <json_string'> ':' <json_raw> ',' )* '"'<json_string'> ':' <json_raw> '}' <json_string> ::= ' ' | '!' | '#' | '$' | '%' | '&' | ''' | '*' | '+' | '-' | ',' | '.' | '/' | ':' | ';' | '<' | '=' | '>' | '?' | '@' | '[' | ']' | '^' | '_', ''', | '{' | '|' | '}' | '~' | [A-Za-z0-9] | '\' <decode_escape> <decode_escape> ::= '"' | '/' | 'b' | 'f' | 'n' | 'r' | 't' ✓ [ "1", "xx" ] { "" : [] } { "??_": null} 738421343 "A??3q43xre" { } ✓ ✓ ✓ ✓ ✓
17. ``All happy families are alike, each unhappy family path is unhappy in its own way.'' (Leo Tolstoy) Anna Karenina
18. ``All happy families paths are alike, each unhappy family path is unhappy in its own way.'' (apologies to Leo Tolstoy) The Anna Karenina Principle
19. Mutation testing to the rescue!
20. Mutation testing to the rescue!
21. = b2 4ac Mutation Testing
22. d = b^2 - 4 * a * c Original = b2 4ac Mutation Testing
23. d = b^3 - 4 * a * c d = b^2 + 4 * a * c d = b^2 - 4 + a * c Mutants d = b^2 - 4 * a * c Original = b2 4ac Mutation Testing
24. d = b^3 - 4 * a * c d = b^2 + 4 * a * c d = b^2 - 4 + a * c Mutants d = b^2 - 4 * a * c Original (a = 0, b = 0, c = 0) => (d = 0) (a = 1, b = 1, c = 1) => (d = -3) (a = 0, b = 2, c = 0) => (d = 4) Mutants killed by test cases Test cases = b2 4ac Mutation Testing
25. d = b^2 - 4 * a * c = b2 4ac Equivalent Mutants
26. d = b^2 - 4 * a * c = b2 4ac d = (-b)^2 - 4 * a * c d = b^2 - 4 * (- a) * (-c) d = b^2 - 3 * a * c d = b^2 - 4 * a + c Equivalent Mutants
27. d = b^2 - 4 * a * c = b2 4ac d = (-b)^2 - 4 * a * c d = b^2 - 4 * (- a) * (-c) d = b^2 - 3 * a * c d = b^2 - 4 * a + c Equivalent mutants Equivalent Mutants
28. … = b2 4ac Some Possible Mutants
29. d = b^0 - 4 * a * c;
 d = b^1 - 4 * a * c; d = b^-1 - 4 * a * c; d = b^MAX - 4 * a * c; d = b^MIN - 4 * a * c; d = b - 4 * a * c;
 d = b ^ 4 * a * c; d = b^2 - 0 * a * c;
 d = b^2 - 1 * a * c;
 d = b^2 – (-1) * a * c;
 d = b^2 - MAX * a * c;
 d = b^2 - MIN * a * c;
 d = b^2 - 4 * a * c;
 d = b^2 - 4 * a * c; d = b^2 + 4 * a * c;
 d = b^2 * 4 * a * c;
 d = b^2 / 4 * a * c;
 d = b^2 ^ 4 * a * c;
 d = b^2 % 4 * a * c; d = b^2 << 4 * a * c; d = b^2 >> 4 * a * c; d = b^2 * 4 + a * c;
 d = b^2 * 4 - a * c;
 d = b^2 * 4 / a * c;
 d = b^2 * 4 ^ a * c;
 d = b^2 * 4 % a * c; d = b^2 * 4 << a * c; d = b^2 * 4 >> a * c; d = b^2 * 4 * a + c;
 d = b^2 * 4 * a - c;
 d = b^2 * 4 * a / c;
 d = b^2 * 4 * a ^ c;
 d = b^2 * 4 * a % c; d = b^2 * 4 * a << c; d = b^2 * 4 * a >> c; d = b + 2 - 4 * a * c;
 d = b - 2 - 4 * a * c;
 d = b * 2 - 4 * a * c;
 d = b / 2 - 4 * a * c;
 d = b % 2 - 4 * a * c;
 d = b << 2 - 4 * a * c;
 d = b >> 2 - 4 * a * c;
 … = b2 4ac Some Possible Mutants
30. d = b^0 - 4 * a * c;
 d = b^1 - 4 * a * c; d = b^-1 - 4 * a * c; d = b^MAX - 4 * a * c; d = b^MIN - 4 * a * c; d = b - 4 * a * c;
 d = b ^ 4 * a * c; d = b^2 - 0 * a * c;
 d = b^2 - 1 * a * c;
 d = b^2 – (-1) * a * c;
 d = b^2 - MAX * a * c;
 d = b^2 - MIN * a * c;
 d = b^2 - 4 * a * c;
 d = b^2 - 4 * a * c; d = b^2 + 4 * a * c;
 d = b^2 * 4 * a * c;
 d = b^2 / 4 * a * c;
 d = b^2 ^ 4 * a * c;
 d = b^2 % 4 * a * c; d = b^2 << 4 * a * c; d = b^2 >> 4 * a * c; d = b^2 * 4 + a * c;
 d = b^2 * 4 - a * c;
 d = b^2 * 4 / a * c;
 d = b^2 * 4 ^ a * c;
 d = b^2 * 4 % a * c; d = b^2 * 4 << a * c; d = b^2 * 4 >> a * c; d = b^2 * 4 * a + c;
 d = b^2 * 4 * a - c;
 d = b^2 * 4 * a / c;
 d = b^2 * 4 * a ^ c;
 d = b^2 * 4 * a % c; d = b^2 * 4 * a << c; d = b^2 * 4 * a >> c; d = b + 2 - 4 * a * c;
 d = b - 2 - 4 * a * c;
 d = b * 2 - 4 * a * c;
 d = b / 2 - 4 * a * c;
 d = b % 2 - 4 * a * c;
 d = b << 2 - 4 * a * c;
 d = b >> 2 - 4 * a * c;
 … = b2 4ac Some Possible Mutants Traditional mutants are unusable as recommendations
31. Requirement: Valid & Invalid Domain Program
32. Requirement: Valid & Invalid Domain Program ✔ Accept
33. Requirement: Valid & Invalid Domain Program ✘ ✔ Accept Reject
34. Requirement: Valid & Invalid Domain JSON {} "}
35. Requirement: Valid & Invalid Domain JSON {} "} ✔ Accept
36. Requirement: Valid & Invalid Domain JSON {} "} ✔ Accept ✘ Reject
37. JSON Repair to Accept Invalid Inputs {} "} ✔ Accept
38. JSON Repair to Accept Invalid Inputs {} "} ✔ Accept ✔ Accept
39. JSON Repair to Accept Invalid Inputs {} "} ✔ Accept ✔ Accept • Identify tests for unhappy paths
40. JSON Repair to Accept Invalid Inputs {} "} ✔ Accept ✔ Accept • Identify tests for unhappy paths • No equivalent mutants
41. Generating Valid Inputs Program ✔ (Valid Input)
42. Generating Valid Inputs • Existing tests Program ✔ (Valid Input)
43. Generating Valid Inputs • Existing tests • Specifications Program ✔ (Valid Input)
44. Generating Valid Inputs • Existing tests • Specifications • Monotonic prefix generation (2019 PLDI) Program ✔ (Valid Input)
45. Generating Invalid Inputs Program ✔ (Valid Input)
46. Generating Invalid Inputs Program ✔ (Valid Input) Program ✘ (Invalid Input) Input Mutations • trim • swap • delete • bit/byte flip
47. Program Repair Make the invalid input follow the valid input's execution path (Valid Input) (Invalid Input) (accept) (reject)
48. Program Repair Make the invalid input follow the valid input's execution path (Valid Input) (Invalid Input) (accept) (reject)
49. Program Repair Make the invalid input follow the valid input's execution path (Valid Input) (Invalid Input) (accept) (reject)
50. Program Repair Make the invalid input follow the valid input's execution path (Valid Input) (Invalid Input) (accept) (reject)
51. def triangle(a, b, c): if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
52. def triangle(a, b, c): if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,1,1) Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
53. def triangle(a, b, c): if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,1,1) Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
54. def triangle(a, b, c): if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,1,1) Example: Triangle Program ✔ 1,1,1 if triangle(a, b, c) == Equilateral: return Accept else: return Reject
55. def triangle(a, b, c): if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
56. def triangle(a, b, c): if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,2,1) Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
57. def triangle(a, b, c): if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,2,1) Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
58. def triangle(a, b, c): if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,2,1) Example: Triangle Program ✔ ✘ 1,1,1 1,2,1 if triangle(a, b, c) == Equilateral: return Accept else: return Reject
59. def triangle(a, b, c): if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,2,1) Example: Triangle Program ✔ ✘ 1,1,1 1,2,1 if triangle(a, b, c) == Equilateral: return Accept else: return Reject
60. def triangle(a, b, c): if a <= b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene <= Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
61. def triangle(a, b, c): if a <= b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,2,1) <= Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
62. def triangle(a, b, c): if a <= b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,2,1) <= Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
63. def triangle(a, b, c): if a <= b: if b >= c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene <= >= Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
64. def triangle(a, b, c): if a <= b: if b >= c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,2,1) <= >= Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
65. def triangle(a, b, c): if a <= b: if b >= c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,2,1) <= >= Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject ✔ 1,1,1 1,2,1 ✔
66. Test Suite Augmentation Possible Inputs Valid Inputs
67. Test Suite Augmentation Possible Inputs Valid Inputs Existing test cases
68. Test Suite Augmentation Possible Inputs Valid Inputs Existing test cases Augmented test cases
69. Evaluation
70. Subjects Subject SLOC Stmt Cov% mathexpr 169 88 urljava 230 89 cgi 72 98 xsum 13 100 simplejson 1486 94 ijson 309 99 nayajson 546 88 microjson 311 95
71. Live Mutants Subject Live Relevant %Relevant mathexpr 91 79 86.8 % urljava 27 24 88.9 % cgi 80 41 51.3 % xsum 5 5 100 % simplejson 2 2 100 % ijson 13 9 69.2 % nayajson 14 11 78.6 % microjson 472 423 89.6 %
72. Minimal Mutants (Uniqueness of Faults) log(minimal mutants) Mauris augmented Cosmic-ray Unaugmented Cosmic-ray
73. Future
74. def triangle(a, b, c): if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene <= >= return triangle(a, b, c) Future: Non-overlapping Domains
75. def triangle(a, b, c): if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle(1,1,1) == Equivalent <= >= return triangle(a, b, c) Future: Non-overlapping Domains triangle(1,2,1) == Isosceles triangle(1,2,3) == Scalene
76. Future: Better Repair JSON "" 1 {} [] "] ✔
77. Summary