Save 37% off PRO during our Black Friday Sale! »

Inducing Subtle Mutations with Program Repair

Inducing Subtle Mutations with Program Repair

Mutation workshop 2021 Virtual

D27cb84e0d30e2778e9b66d6a5f42106?s=128

Rahul Gopinath

April 12, 2021
Tweet

Transcript

  1. Inducing Subtle Mutations with Program Repair Florian Schwander Rahul Gopinath

    Andreas Zeller CISPA Helmholtz Center for Information Security Best Paper Award
  2. Inducing Subtle Mutations with Program Repair Florian Schwander Rahul Gopinath

    Andreas Zeller CISPA Helmholtz Center for Information Security Best Paper Award
  3. https://www.json.org

  4. object { } { members } members pair pair ,

    members pair string : value array [ ] [ elements ] elements value value , elements value string number object array true false null string " " " chars " chars char char chars char UNICODE \ [",\,CTRL] \" \\ \/ \b \f \n \r \t \u hex hex hex hex number int int frac int exp int frac exp int digit onenine digits - digit - onenine digits frac . digits exp e digits hex digit A - F a - f digits digit digit digits e e e+ e- E E+ E- https://www.json.org
  5. None
  6. https://nvd.nist.gov/vuln/data-feeds JSON Vulnerability Feeds

  7. https://nvd.nist.gov/vuln/data-feeds JSON Vulnerability Feeds

  8. 5 Parsing JSON is a Minefield http://seriot.ch/

  9. 5 Parsing JSON is a Minefield http://seriot.ch/

  10. 5 Parsing JSON is a Minefield http://seriot.ch/

  11. 5 Parsing JSON is a Minefield http://seriot.ch/

  12. 5 Parsing JSON is a Minefield http://seriot.ch/ Expected Parse Fail

    (Expect Success) Parse Success (Expect Fail) Parse Success (Undefined) Parse Fail (Undefined) Parser Crash Timeout
  13. 5 Parsing JSON is a Minefield http://seriot.ch/ Expected Parse Fail

    (Expect Success) Parse Success (Expect Fail) Parse Success (Undefined) Parse Fail (Undefined) Parser Crash Timeout
  14. None
  15. <START> ::= <json_raw> <json_raw> ::= '"' <json_string'> | '[' <json_list'>

    | '{' <json_dict'> | <json_number'> | 'true' | 'false' | 'null' <json_number'> ::= <json_number>+ | <json_number>+ 'e' <json_number>+ <json_number> ::= '+' | '-' | '.' | [0-9] | 'E' | 'e' <json_string'> ::= <json_string>* '"' <json_list'> ::= ']' | <json_raw> (','<json_raw>)* ']' | ( ',' <json_raw>)+ (',' <json_raw>)* ']' <json_dict'> ::= '}' | ( '"' <json_string'> ':' <json_raw> ',' )* '"'<json_string'> ':' <json_raw> '}' <json_string> ::= ' ' | '!' | '#' | '$' | '%' | '&' | ''' | '*' | '+' | '-' | ',' | '.' | '/' | ':' | ';' | '<' | '=' | '>' | '?' | '@' | '[' | ']' | '^' | '_', ''', | '{' | '|' | '}' | '~' | [A-Za-z0-9] | '\' <decode_escape> <decode_escape> ::= '"' | '/' | 'b' | 'f' | 'n' | 'r' | 't'
  16. <START> ::= <json_raw> <json_raw> ::= '"' <json_string'> | '[' <json_list'>

    | '{' <json_dict'> | <json_number'> | 'true' | 'false' | 'null' <json_number'> ::= <json_number>+ | <json_number>+ 'e' <json_number>+ <json_number> ::= '+' | '-' | '.' | [0-9] | 'E' | 'e' <json_string'> ::= <json_string>* '"' <json_list'> ::= ']' | <json_raw> (','<json_raw>)* ']' | ( ',' <json_raw>)+ (',' <json_raw>)* ']' <json_dict'> ::= '}' | ( '"' <json_string'> ':' <json_raw> ',' )* '"'<json_string'> ':' <json_raw> '}' <json_string> ::= ' ' | '!' | '#' | '$' | '%' | '&' | ''' | '*' | '+' | '-' | ',' | '.' | '/' | ':' | ';' | '<' | '=' | '>' | '?' | '@' | '[' | ']' | '^' | '_', ''', | '{' | '|' | '}' | '~' | [A-Za-z0-9] | '\' <decode_escape> <decode_escape> ::= '"' | '/' | 'b' | 'f' | 'n' | 'r' | 't' ✓ [ "1", "xx" ] { "" : [] } { "??_": null} 738421343 "A??3q43xre" { } ✓ ✓ ✓ ✓ ✓
  17. ``All happy families are alike, each unhappy family path is

    unhappy in its own way.'' (Leo Tolstoy) Anna Karenina
  18. ``All happy families paths are alike, each unhappy family path

    is unhappy in its own way.'' (apologies to Leo Tolstoy) The Anna Karenina Principle
  19. Mutation testing to the rescue!

  20. Mutation testing to the rescue!

  21. = b2 4ac Mutation Testing

  22. d = b^2 - 4 * a * c Original

    = b2 4ac Mutation Testing
  23. d = b^3 - 4 * a * c d

    = b^2 + 4 * a * c d = b^2 - 4 + a * c Mutants d = b^2 - 4 * a * c Original = b2 4ac Mutation Testing
  24. d = b^3 - 4 * a * c d

    = b^2 + 4 * a * c d = b^2 - 4 + a * c Mutants d = b^2 - 4 * a * c Original (a = 0, b = 0, c = 0) => (d = 0) (a = 1, b = 1, c = 1) => (d = -3) (a = 0, b = 2, c = 0) => (d = 4) Mutants killed by test cases Test cases = b2 4ac Mutation Testing
  25. d = b^2 - 4 * a * c =

    b2 4ac Equivalent Mutants
  26. d = b^2 - 4 * a * c =

    b2 4ac d = (-b)^2 - 4 * a * c d = b^2 - 4 * (- a) * (-c) d = b^2 - 3 * a * c d = b^2 - 4 * a + c Equivalent Mutants
  27. d = b^2 - 4 * a * c =

    b2 4ac d = (-b)^2 - 4 * a * c d = b^2 - 4 * (- a) * (-c) d = b^2 - 3 * a * c d = b^2 - 4 * a + c Equivalent mutants Equivalent Mutants
  28. … = b2 4ac Some Possible Mutants

  29. d = b^0 - 4 * a * c;
 d

    = b^1 - 4 * a * c; d = b^-1 - 4 * a * c; d = b^MAX - 4 * a * c; d = b^MIN - 4 * a * c; d = b - 4 * a * c;
 d = b ^ 4 * a * c; d = b^2 - 0 * a * c;
 d = b^2 - 1 * a * c;
 d = b^2 – (-1) * a * c;
 d = b^2 - MAX * a * c;
 d = b^2 - MIN * a * c;
 d = b^2 - 4 * a * c;
 d = b^2 - 4 * a * c; d = b^2 + 4 * a * c;
 d = b^2 * 4 * a * c;
 d = b^2 / 4 * a * c;
 d = b^2 ^ 4 * a * c;
 d = b^2 % 4 * a * c; d = b^2 << 4 * a * c; d = b^2 >> 4 * a * c; d = b^2 * 4 + a * c;
 d = b^2 * 4 - a * c;
 d = b^2 * 4 / a * c;
 d = b^2 * 4 ^ a * c;
 d = b^2 * 4 % a * c; d = b^2 * 4 << a * c; d = b^2 * 4 >> a * c; d = b^2 * 4 * a + c;
 d = b^2 * 4 * a - c;
 d = b^2 * 4 * a / c;
 d = b^2 * 4 * a ^ c;
 d = b^2 * 4 * a % c; d = b^2 * 4 * a << c; d = b^2 * 4 * a >> c; d = b + 2 - 4 * a * c;
 d = b - 2 - 4 * a * c;
 d = b * 2 - 4 * a * c;
 d = b / 2 - 4 * a * c;
 d = b % 2 - 4 * a * c;
 d = b << 2 - 4 * a * c;
 d = b >> 2 - 4 * a * c;
 … = b2 4ac Some Possible Mutants
  30. d = b^0 - 4 * a * c;
 d

    = b^1 - 4 * a * c; d = b^-1 - 4 * a * c; d = b^MAX - 4 * a * c; d = b^MIN - 4 * a * c; d = b - 4 * a * c;
 d = b ^ 4 * a * c; d = b^2 - 0 * a * c;
 d = b^2 - 1 * a * c;
 d = b^2 – (-1) * a * c;
 d = b^2 - MAX * a * c;
 d = b^2 - MIN * a * c;
 d = b^2 - 4 * a * c;
 d = b^2 - 4 * a * c; d = b^2 + 4 * a * c;
 d = b^2 * 4 * a * c;
 d = b^2 / 4 * a * c;
 d = b^2 ^ 4 * a * c;
 d = b^2 % 4 * a * c; d = b^2 << 4 * a * c; d = b^2 >> 4 * a * c; d = b^2 * 4 + a * c;
 d = b^2 * 4 - a * c;
 d = b^2 * 4 / a * c;
 d = b^2 * 4 ^ a * c;
 d = b^2 * 4 % a * c; d = b^2 * 4 << a * c; d = b^2 * 4 >> a * c; d = b^2 * 4 * a + c;
 d = b^2 * 4 * a - c;
 d = b^2 * 4 * a / c;
 d = b^2 * 4 * a ^ c;
 d = b^2 * 4 * a % c; d = b^2 * 4 * a << c; d = b^2 * 4 * a >> c; d = b + 2 - 4 * a * c;
 d = b - 2 - 4 * a * c;
 d = b * 2 - 4 * a * c;
 d = b / 2 - 4 * a * c;
 d = b % 2 - 4 * a * c;
 d = b << 2 - 4 * a * c;
 d = b >> 2 - 4 * a * c;
 … = b2 4ac Some Possible Mutants Traditional mutants are unusable as recommendations
  31. Requirement: Valid & Invalid Domain Program

  32. Requirement: Valid & Invalid Domain Program ✔ Accept

  33. Requirement: Valid & Invalid Domain Program ✘ ✔ Accept Reject

  34. Requirement: Valid & Invalid Domain JSON {} "}

  35. Requirement: Valid & Invalid Domain JSON {} "} ✔ Accept

  36. Requirement: Valid & Invalid Domain JSON {} "} ✔ Accept

    ✘ Reject
  37. JSON Repair to Accept Invalid Inputs {} "} ✔ Accept

  38. JSON Repair to Accept Invalid Inputs {} "} ✔ Accept

    ✔ Accept
  39. JSON Repair to Accept Invalid Inputs {} "} ✔ Accept

    ✔ Accept • Identify tests for unhappy paths
  40. JSON Repair to Accept Invalid Inputs {} "} ✔ Accept

    ✔ Accept • Identify tests for unhappy paths • No equivalent mutants
  41. Generating Valid Inputs Program ✔ (Valid Input)

  42. Generating Valid Inputs • Existing tests Program ✔ (Valid Input)

  43. Generating Valid Inputs • Existing tests • Specifications Program ✔

    (Valid Input)
  44. Generating Valid Inputs • Existing tests • Specifications • Monotonic

    prefix generation (2019 PLDI) Program ✔ (Valid Input)
  45. Generating Invalid Inputs Program ✔ (Valid Input)

  46. Generating Invalid Inputs Program ✔ (Valid Input) Program ✘ (Invalid

    Input) Input Mutations • trim • swap • delete • bit/byte flip
  47. Program Repair Make the invalid input follow the valid input's

    execution path (Valid Input) (Invalid Input) (accept) (reject)
  48. Program Repair Make the invalid input follow the valid input's

    execution path (Valid Input) (Invalid Input) (accept) (reject)
  49. Program Repair Make the invalid input follow the valid input's

    execution path (Valid Input) (Invalid Input) (accept) (reject)
  50. Program Repair Make the invalid input follow the valid input's

    execution path (Valid Input) (Invalid Input) (accept) (reject)
  51. def triangle(a, b, c): if a == b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  52. def triangle(a, b, c): if a == b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,1,1) Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  53. def triangle(a, b, c): if a == b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,1,1) Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  54. def triangle(a, b, c): if a == b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,1,1) Example: Triangle Program ✔ 1,1,1 if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  55. def triangle(a, b, c): if a == b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  56. def triangle(a, b, c): if a == b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,2,1) Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  57. def triangle(a, b, c): if a == b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,2,1) Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  58. def triangle(a, b, c): if a == b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,2,1) Example: Triangle Program ✔ ✘ 1,1,1 1,2,1 if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  59. def triangle(a, b, c): if a == b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,2,1) Example: Triangle Program ✔ ✘ 1,1,1 1,2,1 if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  60. def triangle(a, b, c): if a <= b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene <= Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  61. def triangle(a, b, c): if a <= b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,2,1) <= Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  62. def triangle(a, b, c): if a <= b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,2,1) <= Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  63. def triangle(a, b, c): if a <= b: if b

    >= c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene <= >= Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  64. def triangle(a, b, c): if a <= b: if b

    >= c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,2,1) <= >= Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject
  65. def triangle(a, b, c): if a <= b: if b

    >= c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,2,1) <= >= Example: Triangle Program if triangle(a, b, c) == Equilateral: return Accept else: return Reject ✔ 1,1,1 1,2,1 ✔
  66. Test Suite Augmentation Possible Inputs Valid Inputs

  67. Test Suite Augmentation Possible Inputs Valid Inputs Existing test cases

  68. Test Suite Augmentation Possible Inputs Valid Inputs Existing test cases

    Augmented test cases
  69. Evaluation

  70. Subjects Subject SLOC Stmt Cov% mathexpr 169 88 urljava 230

    89 cgi 72 98 xsum 13 100 simplejson 1486 94 ijson 309 99 nayajson 546 88 microjson 311 95
  71. Live Mutants Subject Live Relevant %Relevant mathexpr 91 79 86.8

    % urljava 27 24 88.9 % cgi 80 41 51.3 % xsum 5 5 100 % simplejson 2 2 100 % ijson 13 9 69.2 % nayajson 14 11 78.6 % microjson 472 423 89.6 %
  72. Minimal Mutants (Uniqueness of Faults) log(minimal mutants) Mauris augmented Cosmic-ray

    Unaugmented Cosmic-ray
  73. Future

  74. def triangle(a, b, c): if a == b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene <= >= return triangle(a, b, c) Future: Non-overlapping Domains
  75. def triangle(a, b, c): if a == b: if b

    == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle(1,1,1) == Equivalent <= >= return triangle(a, b, c) Future: Non-overlapping Domains triangle(1,2,1) == Isosceles triangle(1,2,3) == Scalene
  76. Future: Better Repair JSON "" 1 {} [] "] ✔

  77. Summary

  78. None
  79. None
  80. None
  81. None
  82. None
  83. None
  84. None