$30 off During Our Annual Pro Sale. View Details »

Mutation Analysis: Answering the Fuzzing Challenge

Rahul Gopinath
November 03, 2024
11

Mutation Analysis: Answering the Fuzzing Challenge

University of Melbourne 2022

Rahul Gopinath

November 03, 2024
Tweet

Transcript

  1. 6

  2. 8 Input ✓ ✘ Testing @app.route('/admin') def admin(): username =

    request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500
  3. HTTP/1.1 401 Not Authorized Content-Type: application/json { "Error": "Assignment of

    internal role 'superadmin' is forbidden" } HTTP/1.1 200 OK Content-type: application/json { "result": "OK: Updated user 'exampleUser' with role 'superadmin'" } 9 Input ✓ ✘ Testing POST /user/update HTTP/1.1 { "user": "exampleUser", "roles": [ "superadmin" ] } @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 HTTP/1.1 500 Internal Server Error ✘
  4. 10 Can We Trust Our Tests? Tests are still predominantly

    written manually (Patrick Lam 2014) Developer Tester
  5. 11 • Modern tests -- • Are complex and have

    non-deterministic control flow • Interact: network, file system ... • No rigorous quality control • 50% of test code is cut and paste (Lam 2014) • 65% of test assertions are inadequate or wrong (Zhi 2013) • Once a test is written, it is rarely looked at unless it fails (Coplian 2014) Can We Trust Our Tests?
  6. @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return

    {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 13 Thou Shalt Cover Thy Code But is coverage su ffi cient? @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500
  7. 14 Code Coverage is Useful • Statement coverage vs effectiveness

    • Developer written or organic test suites Results from 250 programs from Github Effectiveness = 0.9 x S.Coverage
 R2 = 0.939 Effectiveness Statement Coverage Effectiveness computed with faults produced by PIT 2014 ICSE
  8. 15 But Maybe Misleading • Statement coverage against effectiveness •

    Machine generated test suites (using Randoop) Results from 250 programs from Github Effectiveness = 0.6 x S.Coverage R2 = 0.72 Effectiveness Statement Coverage Effectiveness computed with faults produced by PIT 2014 ICSE
  9. 16 Coverage Maybe Misleading More complex coverage measures are easier

    to attain for automated tools using signi fi cantly weaker assertions. Effectiveness Branch Coverage Effectiveness Effectiveness Effectiveness Branch Coverage Path Coverage Path Coverage Gopinath, Jensen, and Groce “Code Coverage for Suite Evaluation by Developers” 2014 ICSE 2014 ICSE
  10. 17 Can We Trust Our Tests? HTTP/1.1 401 Not Authorized

    Content-Type: application/json { "Error": "Assignment of internal role 'superadmin' is forbidden" } HTTP/1.1 200 OK Content-type: application/json { "result": "OK: Updated user 'exampleUser' with role 'superadmin'" } Input ✓ ✘ POST /user/update HTTP/1.1 { "user": "exampleUser", "roles": [ "superadmin" ] } @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 HTTP/1.1 500 Internal Server Error ✘ 17 Unless backed by test assertions, code coverage is pointless
  11. 19 A Simple Idea: Fault Seeding • Effectiveness = M/N

    (here 5/10 = 0.5) • Seed the program with N random faults (here 10) • Run the test suite against the faults • Say M faults were caught (here 5) Typically used in benchmarks
  12. 20 Problems with Fault Seeding • Relies on manual generation/curation

    of faults • Hand produced faults are different from real-world faults (Andrews 2005)
  13. 22 What Is Mutation Testing? d = b^3 - 4

    * a * c d = b^2 + 4 * a * c d = b^2 - 4 + a * c Mutants d = b^2 - 4 * a * c Original (a = 0, b = 0, c = 0) => (d = 0) (a = 1, b = 1, c = 1) => (d = -3) (a = 0, b = 2, c = 0) => (d = 4) Mutants killed by test cases Test cases = b2 4ac
  14. 23 Some Possible Mutants d = b^0 - 4 *

    a * c;
 d = b^1 - 4 * a * c; d = b^-1 - 4 * a * c; d = b^MAX - 4 * a * c; d = b^MIN - 4 * a * c; d = b - 4 * a * c;
 d = b ^ 4 * a * c; d = b^2 - 0 * a * c;
 d = b^2 - 1 * a * c;
 d = b^2 – (-1) * a * c;
 d = b^2 - MAX * a * c;
 d = b^2 - MIN * a * c;
 d = b^2 - 4 * a * c;
 d = b^2 - 4 * a * c; d = b^2 + 4 * a * c;
 d = b^2 * 4 * a * c;
 d = b^2 / 4 * a * c;
 d = b^2 ^ 4 * a * c;
 d = b^2 % 4 * a * c; d = b^2 << 4 * a * c; d = b^2 >> 4 * a * c; d = b^2 * 4 + a * c;
 d = b^2 * 4 - a * c;
 d = b^2 * 4 / a * c;
 d = b^2 * 4 ^ a * c;
 d = b^2 * 4 % a * c; d = b^2 * 4 << a * c; d = b^2 * 4 >> a * c; d = b^2 * 4 * a + c;
 d = b^2 * 4 * a - c;
 d = b^2 * 4 * a / c;
 d = b^2 * 4 * a ^ c;
 d = b^2 * 4 * a % c; d = b^2 * 4 * a << c; d = b^2 * 4 * a >> c; d = b + 2 - 4 * a * c;
 d = b - 2 - 4 * a * c;
 d = b * 2 - 4 * a * c;
 d = b / 2 - 4 * a * c;
 d = b % 2 - 4 * a * c;
 d = b << 2 - 4 * a * c;
 d = b >> 2 - 4 * a * c;
 … = b2 4ac
  15. 24 Why Does It Work? • The simple mutations are

    not the entire set of faults • n mutations can produce 2^n complex faults • So why does mutation testing work?
  16. 25 Assumptions of Mutation Testing The finite neighborhood assumption Programmers

    make simple mistakes. d = b^2 + 4 * a * c d = b^2 - 4 * a * c = b2 4ac
  17. 26 The coupling effect • Faults rarely interact • If

    they interact, they become easier to detect (kill) d = b^2 + 4 * a + c a=1,b=1,c=1 => d = 0 a=0,b=0,c=0 => d = 0 a=1,b=1,c=1 => d = 0 a=0,b=0,c=0 => d = 0 d = b^2 - 4 * a + c d = b^2 + 4 * a * c = b2 4ac Assumptions of Mutation Testing
  18. 27 Finite Neighborhood Bug fi x patches analyzed from: 1850

    C
 1128 Java
 1000 Python 1393 Haskell open source projects from Github 2014 ISSRE
  19. 29 Can Mutation Score Predict Future Bugs? Statements with detected

    mutants twice less likely to contain future bugs (1-Mutation score) is a good proxy for residual defects Boxplot of bug fi xes on covered program elements with killed mutants vs elements with no killed mutants FSE 2016
  20. 30

  21. 32 Debian 5 ~ 70 million lines Smart cars ~

    100 million lines Google is ~ 2 Billion lines A million lines zoomed (informationisbeautiful.net) ‘70s ‘80s ‘90s 2000 onwards Software Complexity Size of software systems has a nasty habit of doubling every few years
  22. HTTP/1.1 401 Not Authorized Content-Type: application/json { "Error": "Assignment of

    internal role 'superadmin' is forbidden" } HTTP/1.1 200 OK Content-type: application/json { "result": "OK: Created user 'exampleUser' with role 'superadmin\ud888'" } 34 Input ✓ ✘ E ff ective Testing? POST /user/create HTTP/1.1 { "user": "exampleUser", "roles": [ "superadmin\ud888" ] } @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 HTTP/1.1 500 Internal Server Error ✘
  23. 35 E ff ective Testing? 24 = 16 2 Testing

    cannot keep up when components (layers, libraries, services) multiply.
  24. 36 Verify Behavior Input ✓ ✘ Automatic Testing @app.route('/admin') def

    admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500
  25. 37 (Oracle) Input ✓ ✘ Automatic Testing Oracles require domain

    expertise @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 Verify Behavior
  26. @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return

    {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 38 Fuzzing Trash deck technique: 1950s - Gerald Weinberg Crash?
  27. @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return

    {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 39 Fuzzing Crash? • Memory Bounds Violation • Privilege Escalation • Safety Violations • Metamorphic Relations • Differential Execution ASAN,MSAN,TSAN,NSan,FuZZan,UBSan
  28. @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return

    {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 [ ; x 1 - G P Z + w c c k c ] ; , N 9 J + ? # 6 ^ 6 \ e ? ] 9 l u 2 _ % ' 4 G X " 0 V U B [ E / r ~ f A p u 6 b 8 < { % s i q 8 Z h . 6 { V , h r ? ; {Ti.r3PIxMMMv6{xS^+'Hq!AxB"YXRS@! Kd6;wtAMefFWM(`|J_<1~o}z3K(CCzRH J I I v H z > _ * . \ > J r l U 3 2 ~ e G P ? lR=bF3+;y$3lodQ<B89!5"W2fK*vE7v{')KC- i,c{<[~m!]o;{.'}Gj\(X}EtYetrpbY@aGZ1{P! A Z U 7 x # 4 ( R t n ! q 4 n C w q o l ^ y 6 } 0 | Ko=*JK~;zMKV=9Nai:wxu{J&UV#HaU)*Bi C < ) , ` + t * g k a < W = Z . % T 5 W G H Z p I 3 0 D < P q > & ] B S 6 R & j ? # t P 7 i a V } - } ` \ ? [ _ [ Z ^ L B M P G - FKj'\xwuZ1=Q`^`5,$N$Q@[!CuRzJ2D|vBy! ^ z k h d f 3 C 5 P A k R ? V ( ( - % > < h n | 3='i2Qx]D$qs4O`1@fevnG'2\11Vf3piU37@ 5 : d f d 4 5 * ( 7 ^ % 5 a p \ z I y l " ' f , $ee,J4Gw:cgNKLie3nx9(`efSlg6#[K"@Wjh Z}r[Scun&sBCS,T[/3]KAeEnQ7lU)3Pn,0)G/ 6N-wyzj/MTd#A;r Program 41 https://www.fuzzingbook.org/html/Fuzzer.html Traditional Fuzzing
  29. • Insert Instrumentation • Generate inputs • Collect execution feedback

    • Branches covered during execution • Slightly Mutate Input and try again Collect inputs obtaining new coverage 43 Feedback Driven Fuzzing 43 https://www.fuzzingbook.org/html/MutationFuzzer.html
  30. 44 def triangle(a, b, c): if a == b: if

    b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene def triangle(a, b, c): __probe_enter() if a == b: __probe_1() if b == c: __probe_2() return Equilateral else: __probe_3() return Isosceles else: __probe_4() if b == c: __probe_5() return Isosceles else: __probe_6() if a == c: __probe_7() return Isosceles else: __probe_8() return Scalene def triangle(a, b, c): if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene • Insert Instrumentation • Generate inputs • Collect execution feedback • Branches covered during execution • Slightly Mutate Input and try again Collect inputs obtaining new coverage 44 https://www.fuzzingbook.org/html/MutationFuzzer.html Feedback Driven Fuzzing
  31. 45 Feedback Driven Fuzzing triangle (1,1,1) def triangle(a, b, c):

    if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene • Insert Instrumentation • Generate inputs • Collect execution feedback • Branches covered during execution • Slightly Mutate Input and try again Collect inputs obtaining new coverage 45 https://www.fuzzingbook.org/html/MutationFuzzer.html
  32. 46 Feedback Driven Fuzzing triangle (1,1,1) def triangle(a, b, c):

    if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene • Insert Instrumentation • Generate inputs • Collect execution feedback • Branches covered during execution • Slightly Mutate Input and try again Collect inputs obtaining new coverage 46 https://www.fuzzingbook.org/html/MutationFuzzer.html
  33. triangle (1,1,1) 47 Feedback Driven Fuzzing triangle (1,1,2) def triangle(a,

    b, c): if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene • Insert Instrumentation • Generate inputs • Collect execution feedback • Branches covered during execution • Slightly Mutate Input and try again Collect inputs obtaining new coverage Mutated 47 https://www.fuzzingbook.org/html/MutationFuzzer.html
  34. 48 Feedback Driven Fuzzing triangle (1,1,3) def triangle(a, b, c):

    if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene • Insert Instrumentation • Generate inputs • Collect execution feedback • Branches covered during execution • Slightly Mutate Input and try again Collect inputs obtaining new coverage Mutated 48 https://www.fuzzingbook.org/html/MutationFuzzer.html
  35. 49 Feedback Driven Fuzzing triangle (1,1,2) def triangle(a, b, c):

    if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene • Insert Instrumentation • Generate inputs • Collect execution feedback • Branches covered during execution • Slightly Mutate Input and try again Collect inputs obtaining new coverage 49 https://www.fuzzingbook.org/html/MutationFuzzer.html triangle (1,1,1) triangle (1,1,3)
  36. • Insert Instrumentation • Generate inputs • Collect execution feedback

    • Branches covered during execution • Slightly Mutate Input and try again Collect inputs obtaining new coverage 50 Feedback Driven Fuzzing triangle (1,1,2) def triangle(a, b, c): if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,1,1) AFL 50
  37. 51 Feedback Driven Fuzzing Weakness: static int is_reserved_word_token(const char *s,

    int len) { const char *reserved[] = { "break", "case", "catch", "continue", "debugger", "default", "delete", "do", "else", "false", "finally", "for", "function", "if", "in", "instanceof", "new", "null", "return", "switch", "this", "throw", "true", "try", "typeof", "var", "void", "while", "with", "let", "undefined", ((void *)0)}; int i; if (!mjs_is_alpha(s[0])) return 0; for (i = 0; reserved[i] != ((void *)0); i++) { if (len == (int)strlen(reserved[i]) && strncmp(s, reserved[i], len) == 0) return i + 1; } return 0; } Tokens if (x > 100) { } coverage: 20% if (x > 100) { } e coverage: 5% if (x > 100) { } el coverage: 5% if (x > 100) { } els coverage: 5% if (x > 100) { } else coverage: 25% No smooth coverage gradient in parsers 51
  38. 52 Feedback Driven Fuzzing def json_raw(stm): while True: stm.skipspaces() c

    = stm.peek() if c == 't': return json_fixed(stm, 'true') elif c == 'f': return json_fixed(stm, 'false') elif c == 'n': return json_fixed(stm, 'null') elif c == '"': return json_string(stm) elif c == '{': return json_dict(stm) elif c == '[': return json_list(stm) elif c in NUMSTART: return json_number(stm) raise JSONError(E_MALF, stm, stm.pos) Weak points: • Need for smooth coverage gradient • Coverage only provides first level guidance 1. {"abc":[]} 2. [{"a":[]}, {"b":[]}, {"c":["ab","c"]}] 52
  39. 53 Feedback Driven Fuzzing def json_raw(stm): while True: stm.skipspaces() c

    = stm.peek() if c == 't': return json_fixed(stm, 'true') elif c == 'f': return json_fixed(stm, 'false') elif c == 'n': return json_fixed(stm, 'null') elif c == '"': return json_string(stm) elif c == '{': return json_dict(stm) elif c == '[': return json_list(stm) elif c in NUMSTART: return json_number(stm) raise JSONError(E_MALF, stm, stm.pos) Weak points: • Need for smooth coverage gradient • Coverage only provides first level guidance 1. {"abc":[]} 2. [{"a":[]}, {"b":[]}, {"c":["ab","c"]}] 53
  40. 54 Feedback Driven Fuzzing def json_raw(stm): while True: stm.skipspaces() c

    = stm.peek() if c == 't': return json_fixed(stm, 'true') elif c == 'f': return json_fixed(stm, 'false') elif c == 'n': return json_fixed(stm, 'null') elif c == '"': return json_string(stm) elif c == '{': return json_dict(stm) elif c == '[': return json_list(stm) elif c in NUMSTART: return json_number(stm) raise JSONError(E_MALF, stm, stm.pos) Weak points: • Need for smooth coverage gradient • Coverage only provides first level guidance 1. {"abc":[]} 2. [{"a":[]}, {"b":[]}, {"c":["ab","c"]}] 54
  41. 55 def json_raw(stm): while True: stm.skipspaces() c = stm.peek() if

    c == 't': return json_fixed(stm, 'true') elif c == 'f': return json_fixed(stm, 'false') elif c == 'n': return json_fixed(stm, 'null') elif c == '"': return json_string(stm) elif c == '{': return json_dict(stm) elif c == '[': return json_list(stm) elif c in NUMSTART: return json_number(stm) raise JSONError(E_MALF, stm, stm.pos) 55 {
 '<json>' : [['<elt>']], '<elt>' : [['<object>'], ['<array>'], ['<string>'], ['<number>'], ['true'], ['false'], ['null']], '<object>' : [['{', '<items>','}'], ['{}']], '<items>' : [['<item>,',',<items>'], ['<item>']], '<item>' : [['<string>',':', '<elt>']], '<array>' : [['[', '<elts>', ']'], ['[]']], '<elts>' : [['<elt>,',',<elts>'], ['<elt>']], '<string>' : [['"', '<chars>', '"'], ['""']], '<chars>' : [['<char>','<chars>'], ['<char>']], '<number>' : [['<digits>']], '<digits>' : [['<digit>','<digits>'], ['<digit>']], '<char>' : [[c] for c in string.characters] '<digit>' : [[c] for c in string.digits]
 } Solution: Structure Aware Fuzzing JSON Grammar
  42. 56 {
 '<json>' : [['<elt>']], '<elt>' : [['<object>'], ['<array>'], ['<string>'],

    ['<number>'], ['true'], ['false'], ['null']], '<object>' : [['{', '<items>','}'], ['{}']], '<items>' : [['<item>,',',<items>'], ['<item>']], '<item>' : [['<string>',':', '<elt>']], '<array>' : [['[', '<elts>', ']'], ['[]']], '<elts>' : [['<elt>,',',<elts>'], ['<elt>']], '<string>' : [['"', '<chars>', '"'], ['""']], '<chars>' : [['<char>','<chars>'], ['<char>']], '<number>' : [['<digits>']], '<digits>' : [['<digit>','<digits>'], ['<digit>']], '<char>' : [[c] for c in string.characters] '<digit>' : [[c] for c in string.digits]
 } Structure Aware Feedback Driven Fuzzer
  43. {
 '<json>' : [['<elt>']], '<elt>' : [['<object>'], ['<array>'], ['<string>'], ['<number>'],

    ['true'], ['false'], ['null']], '<object>' : [['{', '<items>','}'], ['{}']], '<items>' : [['<item>,',',<items>'], ['<item>']], '<item>' : [['<string>',':', '<elt>']], '<array>' : [['[', '<elts>', ']'], ['[]']], '<elts>' : [['<elt>,',',<elts>'], ['<elt>']], '<string>' : [['"', '<chars>', '"'], ['""']], '<chars>' : [['<char>','<chars>'], ['<char>']], '<number>' : [['<digits>']], '<digits>' : [['<digit>','<digits>'], ['<digit>']], '<char>' : [[c] for c in string.characters] '<digit>' : [[c] for c in string.digits]
 } 57 Grammar Fuzzer
  44. General Parser {
 '<json>' : [['<elt>']], '<elt>' : [['<object>'], ['<array>'],

    ['<string>'], ['<number>'], ['true'], ['false'], ['null']], '<object>' : [['{', '<items>','}'], ['{}']], '<items>' : [['<item>,',',<items>'], ['<item>']], '<item>' : [['<string>',':', '<elt>']], '<array>' : [['[', '<elts>', ']'], ['[]']], '<elts>' : [['<elt>,',',<elts>'], ['<elt>']], '<string>' : [['"', '<chars>', '"'], ['""']], '<chars>' : [['<char>','<chars>'], ['<char>']], '<number>' : [['<digits>']], '<digits>' : [['<digit>','<digits>'], ['<digit>']], '<char>' : [[c] for c in string.characters] '<digit>' : [[c] for c in string.digits]
 } 58
  45. Lang Fuzzer {
 '<json>' : [['<elt>']], '<elt>' : [['<object>'], ['<array>'],

    ['<string>'], ['<number>'], ['true'], ['false'], ['null']], '<object>' : [['{', '<items>','}'], ['{}']], '<items>' : [['<item>,',',<items>'], ['<item>']], '<item>' : [['<string>',':', '<elt>']], '<array>' : [['[', '<elts>', ']'], ['[]']], '<elts>' : [['<elt>,',',<elts>'], ['<elt>']], '<string>' : [['"', '<chars>', '"'], ['""']], '<chars>' : [['<char>','<chars>'], ['<char>']], '<number>' : [['<digits>']], '<digits>' : [['<digit>','<digits>'], ['<digit>']], '<char>' : [[c] for c in string.characters] '<digit>' : [[c] for c in string.digits]
 } https://www.fuzzingbook.org/html/LangFuzzer.html 59
  46. def process_input(input): try: ✔val = parse(input) res = process(val) return

    res except SyntaxError: return Error {
 '<json>' : [['<elt>']], '<elt>' : [['<object>'], ['<array>'], ['<string>'], ['<number>'], ['true'], ['false'], ['null']], '<object>' : [['{', '<items>','}'], ['{}']], '<items>' : [['<item>,',',<items>'], ['<item>']], '<item>' : [['<string>',':', '<elt>']], '<array>' : [['[', '<elts>', ']'], ['[]']], '<elts>' : [['<elt>,',',<elts>'], ['<elt>']], '<string>' : [['"', '<chars>', '"'], ['""']], '<chars>' : [['<char>','<chars>'], ['<char>']], '<number>' : [['<digits>']], '<digits>' : [['<digit>','<digits>'], ['<digit>']], '<char>' : [[c] for c in string.characters] '<digit>' : [[c] for c in string.digits]
 } 60
  47. QUIRK_ALLOW_ASCII_CONTROL_CODES QUIRK_ALLOW_BACKSLASH_A QUIRK_ALLOW_BACKSLASH_CAPITAL_U QUIRK_ALLOW_BACKSLASH_E QUIRK_ALLOW_BACKSLASH_NEW_LINE QUIRK_ALLOW_BACKSLASH_QUESTION_MARK QUIRK_ALLOW_BACKSLASH_SINGLE_QUOTE QUIRK_ALLOW_BACKSLASH_V QUIRK_ALLOW_BACKSLASH_X_AS_BYTES QUIRK_ALLOW_BACKSLASH_X_AS_CODE_POINTS

    QUIRK_ALLOW_BACKSLASH_ZERO QUIRK_ALLOW_COMMENT_BLOCK QUIRK_ALLOW_COMMENT_LINE QUIRK_ALLOW_EXTRA_COMMA QUIRK_ALLOW_INF_NAN_NUMBERS QUIRK_ALLOW_LEADING_ASCII_RECORD_SEPARATOR QUIRK_ALLOW_LEADING_UNICODE_BYTE_ORDER_MARK QUIRK_ALLOW_TRAILING_FILLER QUIRK_EXPECT_TRAILING_NEW_LINE_OR_EOF QUIRK_JSON_POINTER_ALLOW_TILDE_N_TILDE_R_TILDE_T QUIRK_REPLACE_INVALID_UNICODE JSON common quirks from https://github.com/google/wuffs 65
  48. "Be liberal in what you accept, and conservative in what

    you send"
 Postel's Law The Specification The Implementation Extra "Features" Where to Get the Grammar From? 66
  49. def json_raw(stm): while True: stm.skipspaces() c = stm.peek() if c

    == 't': return json_fixed(stm, 'true') elif c == 'f': return json_fixed(stm, 'false') elif c == 'n': return json_fixed(stm, 'null') elif c == '"': return json_string(stm) elif c == '{': return json_dict(stm) elif c == '[': return json_list(stm) elif c in NUMSTART: return json_number(stm) raise JSONError(E_MALF, stm, stm.pos) <json_raw>::= <json_object>
 | <json_list>
 | <json_string>
 | <json_number>
 | <json_fixed(true)> | <json_fixed(false)> | <json_fixed(null)> <json_string> ::= `"` <chars> `"` | `""` <chars> ::= <char><chars> | <char> <json_object> ::= `{`<items>`}` | `{}` <items> ::= <item>`,`<items> | <item> <item> ::= <string>`:`<elt> <json_list> ::= `[`<elts>`]` | `[]` <elts> ::= <elt>`,`<elts> | <elt> <json_number> ::= <digits> <digits> ::= <digit><digits> | <digit> https://github.com/phensley/microjson MicroJSON 68 68
  50. def json_raw(stm): while True: stm.skipspaces() c = stm.peek() if c

    == 't': return json_fixed(stm, 'true') elif c == 'f': return json_fixed(stm, 'false') elif c == 'n': return json_fixed(stm, 'null') elif c == '"': return json_string(stm) elif c == '{': return json_dict(stm) elif c == '[': return json_list(stm) elif c in NUMSTART: return json_number(stm) raise JSONError(E_MALF, stm, stm.pos) <json_raw>::= <json_object>
 | <json_list>
 | <json_string>
 | <json_number>
 | <json_fixed(true)> | <json_fixed(false)> | <json_fixed(null)> <json_string> ::= `"` <chars> `"` | `""` <chars> ::= <char><chars> | <char> <json_object> ::= `{`<items>`}` | `{}` <items> ::= <item>`,`<items> | <item> <item> ::= <string>`:`<elt> <json_list> ::= `[`<elts>`]` | `[]` <elts> ::= <elt>`,`<elts> | <elt> <json_number> ::= <digits> <digits> ::= <digit><digits> | <digit> https://github.com/phensley/microjson MicroJSON 69
  51. <F> := <A> | <B> <F> := <A> <B> <C>

    <Fs> := <B> <Fs> | <empty> Structured Control Flow to Grammar Sequence A B C [F] Selection cond A B [F] F T Iteration cond B [F] 70
  52. def json_raw(stm): while True: stm.skipspaces() c = stm.peek() if c

    == 't': return json_fixed(stm, 'true') elif c == 'f': return json_fixed(stm, 'false') elif c == 'n': return json_fixed(stm, 'null') elif c == '"': return json_string(stm) elif c == '{': return json_object(stm) elif c == '[': return json_array(stm) elif c in NUMSTART: return json_number(stm) raise JSONError(E_MALF, stm, stm.pos) <json_raw> ::= <loop1> <loop1> ::= <iter1> <loop1> | <empty> <iter1>::= <json_fixed(true)> | <json_fixed(false)> | <json_fixed(null)> 
 | `"` <json_string> | `{` <json_object> 
 | `[` <json_array> 
 | [[1-9] <json_number>
 71
  53. def json_raw(stm): while True: stm.skipspaces() c = stm.peek() if c

    == 't': return json_fixed(stm, 'true') elif c == 'f': return json_fixed(stm, 'false') elif c == 'n': return json_fixed(stm, 'null') elif c == '"': return json_string(stm) elif c == '{': return json_object(stm) elif c == '[': return json_array(stm) elif c in NUMSTART: return json_number(stm) raise JSONError(E_MALF, stm, stm.pos) <json_raw> ::= <loop1> <loop1> ::= <iter1> <loop1> | <empty> <iter1>::= <json_fixed(true)> | <json_fixed(false)> | <json_fixed(null)> 
 | `"` <json_string> | `{` <json_object> 
 | `[` <json_array> 
 | [[1-9] <json_number>
 72
  54. def json_string(stm): # skip over '"' stm.next() r = []

    while True: c = stm.next() if c == '': raise JSONError(E_TRUNC) elif c == '\\': c = stm.next() r.append(decode_escape(c, stm)) elif c == '"': return ''.join(r) else: r.append(c) <json_raw> ::= <loop1> ... <iter1>::= `"` <json_string> ... <json_string> ::= <loop2> <loop2> ::= <iter2> <loop2> | <empty> <iter2>::= `\\` <decode_escape> | `"` | <other_char> "ab" <json_string> ::= <loop2> <loop2_end> <loop2> ::= <iter2> <loop2> | <empty> <iter2>::= `\\` <decode_escape> | <other_char> <loop2_end> ::= <iter2_end> <iter2_end>::=`"` 73
  55. 74 <START> ::= <json_raw> <json_raw> ::= '"' <json_string'> | '['

    <json_list'> | '{' <json_dict'> | <json_number'> | 'true' | 'false' | 'null' <json_number'> ::= <json_number>+ | <json_number>+ 'e' <json_number>+ <json_number> ::= '+' | '-' | '.' | [0-9] | 'E' | 'e' <json_string'> ::= <json_string>* '"' <json_list'> ::= ']' | <json_raw> (','<json_raw>)* ']' | ( ',' <json_raw>)+ (',' <json_raw>)* ']' <json_dict'> ::= '}' | ( '"' <json_string'> ':' <json_raw> ',' )* '"'<json_string'> ':' <json_raw> '}' <json_string> ::= ' ' | '!' | '#' | '$' | '%' | '&' | ''' | '*' | '+' | '-' | ',' | '.' | '/' | ':' | ';' | '<' | '=' | '>' | '?' | '@' | '[' | ']' | '^' | '_', ''',| '{' | '|' | '}' | '~' | '[A-Za-z0-9]' | '\' <decode_escape> <decode_escape> ::= '"' | '/' | 'b' | 'f' | 'n' | 'r' | 't' stm.next() if expect_key: raise JSONError(E_DKEY, stm, stm.pos) if c == '}': return result expect_key = 1 continue # parse out a key/value pair elif c == '"': key = _from_json_string(stm) stm.skipspaces() c = stm.next() if c != ':': raise JSONError(E_COLON, stm, stm.pos) stm.skipspaces() val = _from_json_raw(stm) result[key] = val expect_key = 0 continue raise JSONError(E_MALF, stm, stm.pos) def _from_json_raw(stm): while True: stm.skipspaces() c = stm.peek() if c == '"': return _from_json_string(stm) elif c == '{': return _from_json_dict(stm) elif c == '[': return _from_json_list(stm) elif c == 't': return _from_json_fixed(stm, 'true', True, E_BOOL) elif c == 'f': return _from_json_fixed(stm, 'false', False, E_BOOL) elif c == 'n': return _from_json_fixed(stm, 'null', None, E_NULL) elif c in NUMSTART: return _from_json_number(stm) raise JSONError(E_MALF, stm, stm.pos) def from_json(data): stm = JSONStream(data) return _from_json_raw(stm) microjson.py Recovered JSON grammar 74
  56. Recall Subjects Mimid calc.py 100.0% mathexpr.py 87.5% cgidecode.py 100.0% urlparse.py

    100.0% microjson.py 98.7% parseclisp.py 99.3% jsonparser.c 100.0% tiny.c 100.0% mjs.c 95.4% Inputs generated by inferred grammar that were accepted by the program Subjects Mimid calc.py 100.0% mathexpr.py 92.7% cgidecode.py 100.0% urlparse.py 96.4% microjson.py 93.0% parseclisp.py 80.6% jsonparser.c 83.8% tiny.c 92.8% mjs.c 95.9% Inputs generated by golden grammar that were accepted by the inferred grammar parser Precision 75 75 Evaluation: Accuracy
  57. 76 <START> ::= <json_raw> <json_raw> ::= '"' <json_string'> | '['

    <json_list'> | '{' <json_dict'> | <json_number'> | 'true' | 'false' | 'null' <json_number'> ::= <json_number>+ | <json_number>+ 'e' <json_number>+ <json_number> ::= '+' | '-' | '.' | [0-9] | 'E' | 'e' <json_string'> ::= <json_string>* '"' <json_list'> ::= ']' | <json_raw> (','<json_raw>)* ']' | ( ',' <json_raw>)+ (',' <json_raw>)* ']' <json_dict'> ::= '}' | ( '"' <json_string'> ':' <json_raw> ',' )* '"'<json_string'> ':' <json_raw> '}' <json_string> ::= ' ' | '!' | '#' | '$' | '%' | '&' | ''' | '*' | '+' | '-' | ',' | '.' | '/' | ':' | ';' | '<' | '=' | '>' | '?' | '@' | '[' | ']' | '^' | '_', ''',| '{' | '|' | '}' | '~' | '[A-Za-z0-9]' | '\' <decode_escape> <decode_escape> ::= '"' | '/' | 'b' | 'f' | 'n' | 'r' | 't' stm.next() if expect_key: raise JSONError(E_DKEY, stm, stm.pos) if c == '}': return result expect_key = 1 continue # parse out a key/value pair elif c == '"': key = _from_json_string(stm) stm.skipspaces() c = stm.next() if c != ':': raise JSONError(E_COLON, stm, stm.pos) stm.skipspaces() val = _from_json_raw(stm) result[key] = val expect_key = 0 continue raise JSONError(E_MALF, stm, stm.pos) def _from_json_raw(stm): while True: stm.skipspaces() c = stm.peek() if c == '"': return _from_json_string(stm) elif c == '{': return _from_json_dict(stm) elif c == '[': return _from_json_list(stm) elif c == 't': return _from_json_fixed(stm, 'true', True, E_BOOL) elif c == 'f': return _from_json_fixed(stm, 'false', False, E_BOOL) elif c == 'n': return _from_json_fixed(stm, 'null', None, E_NULL) elif c in NUMSTART: return _from_json_number(stm) raise JSONError(E_MALF, stm, stm.pos) def from_json(data): stm = JSONStream(data) return _from_json_raw(stm) microjson.py Recovered JSON grammar 76 ESEC/FSE 2020. Mimid
  58. 77 HTTP Parser XML Parser SOAP Parser RPC Parser C

    Parser Check Declarations Check Types Static Checks Challenges Compilers Servers Semantics Application
  59. 78

  60. 79 Which Fuzzer Should We Use? Exponential growth in fuzzing

    literature Cumulative publications Publications per year
  61. 84 Seeded Fault Benchmarks? • Easy to fine-tune a fuzzer

    to overfit • Faults are rarely similar to real faults • Based on bugs we know about! • Human bias in bug curation • Limited supply • Bug interactions requiring deduplication
  62. 85 Seeded Fault Benchmarks? • Easy to fine-tune a fuzzer

    to overfit • Faults are rarely similar to real faults • Based on bugs we know about! • Human bias in bug curation • Limited supply Mutation Analysis?
  63. 86 Mutation Analysis • Easy to fine-tune a fuzzer to

    overfit • Faults are rarely similar to real faults • Based on bugs we know about! • Human bias in bug curation • Limited supply • Bug interactions requiring deduplication
  64. 87 Mutation Analysis • Easy to fine-tune a fuzzer to

    overfit • Faults are rarely similar to real faults • Based on bugs we know about! • Human bias in bug curation • Limited supply • Bug interactions requiring deduplication
  65. 88 Mutation Analysis • Easy to fine-tune a fuzzer to

    overfit • Faults are rarely similar to real faults • Based on bugs we know about! • Human bias in bug curation • Limited supply • Bug interactions requiring deduplication
  66. 89 Mutation Analysis • Easy to fine-tune a fuzzer to

    overfit • Faults are rarely similar to real faults • Based on bugs we know about! • Human bias in bug curation • Limited supply • Bug interactions requiring deduplication
  67. 90 Mutation Analysis • Easy to fine-tune a fuzzer to

    overfit • Faults are rarely similar to real faults • Based on bugs we know about! • Human bias in bug curation • Limited supply • Bug interactions requiring deduplication
  68. 91 Mutation Analysis • Easy to fine-tune a fuzzer to

    overfit • Faults are rarely similar to real faults • Based on bugs we know about! • Human bias in bug curation • Limited supply • Bug interactions requiring deduplication
  69. 92 Mutation Analysis • Easy to fine-tune a fuzzer to

    overfit • Faults are rarely similar to real faults • Based on bugs we know about! • Human bias in bug curation • Limited supply • Bug interactions requiring deduplication
  70. 94 Mutation Analysis Is Costly Deterministically insert all simple faults,

    and try to fi nd each. = b2 4ac d = b^0 - 4 * a * c;
 d = b^1 - 4 * a * c; d = b^-1 - 4 * a * c; d = b^MAX - 4 * a * c; d = b^MIN - 4 * a * c; d = b - 4 * a * c;
 d = b ^ 4 * a * c; d = b^2 - 0 * a * c;
 d = b^2 - 1 * a * c;
 d = b^2 – (-1) * a * c;
 d = b^2 - MAX * a * c;
 d = b^2 - MIN * a * c;
 d = b^2 - 4 * a * c;
 d = b^2 - 4 * a * c; d = b^2 + 4 * a * c;
 d = b^2 * 4 * a * c;
 d = b^2 / 4 * a * c;
 d = b^2 ^ 4 * a * c;
 d = b^2 % 4 * a * c; d = b^2 << 4 * a * c; d = b^2 >> 4 * a * c; d = b^2 * 4 + a * c;
 d = b^2 * 4 - a * c;
 d = b^2 * 4 / a * c;
 d = b^2 * 4 ^ a * c;
 d = b^2 * 4 % a * c; d = b^2 * 4 << a * c; d = b^2 * 4 >> a * c; d = b^2 * 4 * a + c;
 d = b^2 * 4 * a - c;
 d = b^2 * 4 * a / c;
 d = b^2 * 4 * a ^ c;
 d = b^2 * 4 * a % c; d = b^2 * 4 * a << c; d = b^2 * 4 * a >> c; d = b + 2 - 4 * a * c;
 d = b - 2 - 4 * a * c;
 d = b * 2 - 4 * a * c;
 d = b / 2 - 4 * a * c;
 d = b % 2 - 4 * a * c;
 d = b << 2 - 4 * a * c;
 d = b >> 2 - 4 * a * c;
 … and more …
  71. 95 Mutation Analysis Is Costly M mutants Lines of code

    Mutation points Lines of code Number of tests T tests Program size Effort for mutation testing = MxT test runs Test cases are no longer static
  72. 96 Ongoing Research • Non interacting Faults with Higher Order

    Mutants • Separate Coverage Analysis • Parallel Executions