Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Inducing Subtle Mutations with Program Repair

Inducing Subtle Mutations with Program Repair

Mutation workshop 2021 Virtual

Rahul Gopinath

April 12, 2021
Tweet

More Decks by Rahul Gopinath

Other Decks in Research

Transcript

  1. Inducing Subtle Mutations with Program Repair Florian Schwander Rahul Gopinath

    Andreas Zeller CISPA Helmholtz Center for Information Security Best Paper Award
  2. Inducing Subtle Mutations with Program Repair Florian Schwander Rahul Gopinath

    Andreas Zeller CISPA Helmholtz Center for Information Security Best Paper Award
  3. object { } { members } members pair pair ,

    members pair string : value array [ ] [ elements ] elements value value , elements value string number object array true false null string " " " chars " chars char char chars char UNICODE \ [",\,CTRL] \" \\ \/ \b \f \n \r \t \u hex hex hex hex number int int frac int exp int frac exp int digit onenine digits - digit - onenine digits frac . digits exp e digits hex digit A - F a - f digits digit digit digits e e e+ e- E E+ E- https://www.json.org
  4. 5 Parsing JSON is a Minefield http://seriot.ch/ Expected Parse Fail

    (Expect Success) Parse Success (Expect Fail) Parse Success (Undefined) Parse Fail (Undefined) Parser Crash Timeout
  5. 5 Parsing JSON is a Minefield http://seriot.ch/ Expected Parse Fail

    (Expect Success) Parse Success (Expect Fail) Parse Success (Undefined) Parse Fail (Undefined) Parser Crash Timeout
  6. <START> ::= <json_raw> <json_raw> ::= '"' <json_string'> | '[' <json_list'>

    | '{' <json_dict'> | <json_number'> | 'true' | 'false' | 'null' <json_number'> ::= <json_number>+ | <json_number>+ 'e' <json_number>+ <json_number> ::= '+' | '-' | '.' | [0-9] | 'E' | 'e' <json_string'> ::= <json_string>* '"' <json_list'> ::= ']' | <json_raw> (','<json_raw>)* ']' | ( ',' <json_raw>)+ (',' <json_raw>)* ']' <json_dict'> ::= '}' | ( '"' <json_string'> ':' <json_raw> ',' )* '"'<json_string'> ':' <json_raw> '}' <json_string> ::= ' ' | '!' | '#' | '$' | '%' | '&' | ''' | '*' | '+' | '-' | ',' | '.' | '/' | ':' | ';' | '<' | '=' | '>' | '?' | '@' | '[' | ']' | '^' | '_', ''', | '{' | '|' | '}' | '~' | [A-Za-z0-9] | '\' <decode_escape> <decode_escape> ::= '"' | '/' | 'b' | 'f' | 'n' | 'r' | 't'
  7. <START> ::= <json_raw> <json_raw> ::= '"' <json_string'> | '[' <json_list'>

    | '{' <json_dict'> | <json_number'> | 'true' | 'false' | 'null' <json_number'> ::= <json_number>+ | <json_number>+ 'e' <json_number>+ <json_number> ::= '+' | '-' | '.' | [0-9] | 'E' | 'e' <json_string'> ::= <json_string>* '"' <json_list'> ::= ']' | <json_raw> (','<json_raw>)* ']' | ( ',' <json_raw>)+ (',' <json_raw>)* ']' <json_dict'> ::= '}' | ( '"' <json_string'> ':' <json_raw> ',' )* '"'<json_string'> ':' <json_raw> '}' <json_string> ::= ' ' | '!' | '#' | '$' | '%' | '&' | ''' | '*' | '+' | '-' | ',' | '.' | '/' | ':' | ';' | '<' | '=' | '>' | '?' | '@' | '[' | ']' | '^' | '_', ''', | '{' | '|' | '}' | '~' | [A-Za-z0-9] | '\' <decode_escape> <decode_escape> ::= '"' | '/' | 'b' | 'f' | 'n' | 'r' | 't' ✓ [ "1", "xx" ] { "" : [] } { "??_": null} 738421343 "A??3q43xre" { } ✓ ✓ ✓ ✓ ✓
  8. ``All happy families are alike, each unhappy family path is

    unhappy in its own way.'' (Leo Tolstoy) Anna Karenina
  9. ``All happy families paths are alike, each unhappy family path

    is unhappy in its own way.'' (apologies to Leo Tolstoy) The Anna Karenina Principle
  10. d = b^2 - 4 * a * c Original

    = b2 4ac Mutation Testing
  11. d = b^3 - 4 * a * c d

    = b^2 + 4 * a * c d = b^2 - 4 + a * c Mutants d = b^2 - 4 * a * c Original = b2 4ac Mutation Testing
  12. d = b^3 - 4 * a * c d

    = b^2 + 4 * a * c d = b^2 - 4 + a * c Mutants d = b^2 - 4 * a * c Original (a = 0, b = 0, c = 0) => (d = 0) (a = 1, b = 1, c = 1) => (d = -3) (a = 0, b = 2, c = 0) => (d = 4) Mutants killed by test cases Test cases = b2 4ac Mutation Testing
  13. d = b^2 - 4 * a * c =

    b2 4ac Equivalent Mutants
  14. d = b^2 - 4 * a * c =

    b2 4ac d = (-b)^2 - 4 * a * c d = b^2 - 4 * (- a) * (-c) d = b^2 - 3 * a * c d = b^2 - 4 * a + c Equivalent Mutants
  15. d = b^2 - 4 * a * c =

    b2 4ac d = (-b)^2 - 4 * a * c d = b^2 - 4 * (- a) * (-c) d = b^2 - 3 * a * c d = b^2 - 4 * a + c Equivalent mutants Equivalent Mutants
  16. d = b^0 - 4 * a * c;
 d

    = b^1 - 4 * a * c; d = b^-1 - 4 * a * c; d = b^MAX - 4 * a * c; d = b^MIN - 4 * a * c; d = b - 4 * a * c;
 d = b ^ 4 * a * c; d = b^2 - 0 * a * c;
 d = b^2 - 1 * a * c;
 d = b^2 – (-1) * a * c;
 d = b^2 - MAX * a * c;
 d = b^2 - MIN * a * c;
 d = b^2 - 4 * a * c;
 d = b^2 - 4 * a * c; d = b^2 + 4 * a * c;
 d = b^2 * 4 * a * c;
 d = b^2 / 4 * a * c;
 d = b^2 ^ 4 * a * c;
 d = b^2 % 4 * a * c; d = b^2 << 4 * a * c; d = b^2 >> 4 * a * c; d = b^2 * 4 + a * c;
 d = b^2 * 4 - a * c;
 d = b^2 * 4 / a * c;
 d = b^2 * 4 ^ a * c;
 d = b^2 * 4 % a * c; d = b^2 * 4 << a * c; d = b^2 * 4 >> a * c; d = b^2 * 4 * a + c;
 d = b^2 * 4 * a - c;
 d = b^2 * 4 * a / c;
 d = b^2 * 4 * a ^ c;
 d = b^2 * 4 * a % c; d = b^2 * 4 * a << c; d = b^2 * 4 * a >> c; d = b + 2 - 4 * a * c;
 d = b - 2 - 4 * a * c;
 d = b * 2 - 4 * a * c;
 d = b / 2 - 4 * a * c;
 d = b % 2 - 4 * a * c;
 d = b << 2 - 4 * a * c;
 d = b >> 2 - 4 * a * c;
 … = b2 4ac Some Possible Mutants
  17. d = b^0 - 4 * a * c;
 d

    = b^1 - 4 * a * c; d = b^-1 - 4 * a * c; d = b^MAX - 4 * a * c; d = b^MIN - 4 * a * c; d = b - 4 * a * c;
 d = b ^ 4 * a * c; d = b^2 - 0 * a * c;
 d = b^2 - 1 * a * c;
 d = b^2 – (-1) * a * c;
 d = b^2 - MAX * a * c;
 d = b^2 - MIN * a * c;
 d = b^2 - 4 * a * c;
 d = b^2 - 4 * a * c; d = b^2 + 4 * a * c;
 d = b^2 * 4 * a * c;
 d = b^2 / 4 * a * c;
 d = b^2 ^ 4 * a * c;
 d = b^2 % 4 * a * c; d = b^2 << 4 * a * c; d = b^2 >> 4 * a * c; d = b^2 * 4 + a * c;
 d = b^2 * 4 - a * c;
 d = b^2 * 4 / a * c;
 d = b^2 * 4 ^ a * c;
 d = b^2 * 4 % a * c; d = b^2 * 4 << a * c; d = b^2 * 4 >> a * c; d = b^2 * 4 * a + c;
 d = b^2 * 4 * a - c;
 d = b^2 * 4 * a / c;
 d = b^2 * 4 * a ^ c;
 d = b^2 * 4 * a % c; d = b^2 * 4 * a << c; d = b^2 * 4 * a >> c; d = b + 2 - 4 * a * c;
 d = b - 2 - 4 * a * c;
 d = b * 2 - 4 * a * c;
 d = b / 2 - 4 * a * c;
 d = b % 2 - 4 * a * c;
 d = b << 2 - 4 * a * c;
 d = b >> 2 - 4 * a * c;
 … = b2 4ac Some Possible Mutants Traditional mutants are unusable as recommendations
  18. JSON Repair to Accept Invalid Inputs {} "} ✔ Accept

    ✔ Accept • Identify tests for unhappy paths
  19. JSON Repair to Accept Invalid Inputs {} "} ✔ Accept

    ✔ Accept • Identify tests for unhappy paths • No equivalent mutants
  20. Generating Valid Inputs • Existing tests • Specifications • Monotonic

    prefix generation (2019 PLDI) Program ✔ (Valid Input)
  21. Generating Invalid Inputs Program ✔ (Valid Input) Program ✘ (Invalid

    Input) Input Mutations • trim • swap • delete • bit/byte flip
  22. Program Repair Make the invalid input follow the valid input's

    execution path (Valid Input) (Invalid Input) (accept) (reject)
  23. Program Repair Make the invalid input follow the valid input's

    execution path (Valid Input) (Invalid Input) (accept) (reject)
  24. Program Repair Make the invalid input follow the valid input's

    execution path (Valid Input) (Invalid Input) (accept) (reject)
  25. Program Repair Make the invalid input follow the valid input's

    execution path (Valid Input) (Invalid Input) (accept) (reject)
  26. def triangle(a, b, c): if a == b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  27. def triangle(a, b, c): if a == b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,1,1) Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  28. def triangle(a, b, c): if a == b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,1,1) Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  29. def triangle(a, b, c): if a == b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,1,1) Example: Triangle Program ✔ 1,1,1 if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  30. def triangle(a, b, c): if a == b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  31. def triangle(a, b, c): if a == b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,2,1) Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  32. def triangle(a, b, c): if a == b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,2,1) Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  33. def triangle(a, b, c): if a == b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,2,1) Example: Triangle Program ✔ ✘ 1,1,1 1,2,1 if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  34. def triangle(a, b, c): if a == b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,2,1) Example: Triangle Program ✔ ✘ 1,1,1 1,2,1 if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  35. def triangle(a, b, c): if a <= b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene <= Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  36. def triangle(a, b, c): if a <= b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,2,1) <= Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  37. def triangle(a, b, c): if a <= b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,2,1) <= Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  38. def triangle(a, b, c): if a <= b: if b

    >= c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene <= >= Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  39. def triangle(a, b, c): if a <= b: if b

    >= c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,2,1) <= >= Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  40. def triangle(a, b, c): if a <= b: if b

    >= c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,2,1) <= >= Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject ✔ 1,1,1 1,2,1 ✔
  41. Subjects Subject SLOC Stmt Cov% mathexpr 169 88 urljava 230

    89 cgi 72 98 xsum 13 100 simplejson 1486 94 ijson 309 99 nayajson 546 88 microjson 311 95
  42. Live Mutants Subject Live Relevant %Relevant mathexpr 91 79 86.8

    % urljava 27 24 88.9 % cgi 80 41 51.3 % xsum 5 5 100 % simplejson 2 2 100 % ijson 13 9 69.2 % nayajson 14 11 78.6 % microjson 472 423 89.6 %
  43. def triangle(a, b, c): if a == b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene <= >= return triangle(a, b, c) Future: Non-overlapping Domains
  44. def triangle(a, b, c): if a == b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle(1,1,1) == Equivalent <= >= return triangle(a, b, c) Future: Non-overlapping Domains triangle(1,2,1) == Isosceles triangle(1,2,3) == Scalene