Building Blocks for Fuzzing

Talk at IIT-Gandhinagar

Transcript

1. 1 Building Blocks for Fuzzing Rahul Gopinath https://rahul.gopinath.org @_rahulgopinath
2. Rahul Gopinath Building Blocks for Fuzzing https://rahul.gopinath.org @_rahulgopinath 2
3. The story begins 2500 years ago. Vedic Era 500 BC 3
4. Classical Era 500 BC 4
5. 500 BC Aṣṭādhyāyī Dakṣiputra Pāṇini Ad hoc rules Formal specification Vedic Sanskrit Classical Sanskrit 5
6. 6 2500 years later.... 2022 CE Dawn of a New Era
7. 7 New Challenges
8. Bugs 8
9. 9
10. 10 Input ✓ ✘ Testing @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500
11. HTTP/1.1 401 Not Authorize d Content-Type: application/jso n { "Error" : "Assignment of internal rol e 'superadmin' is forbidden " } HTTP/1.1 200 O K Content-type: application/jso n { "result": "OK: Updated use r 'exampleUser' with rol e 'superadmin' " } 11 Input ✓ ✘ Testing POST /user/update HTTP/1. 1 { "user": "exampleUser", "roles": [ "superadmin " ] } @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 HTTP/1.1 500 Internal Server Error ✘
12. 12 Verify Behavior Input ✓ ✘ Automatic Testing @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500
13. 13 (Oracle) Input ✓ ✘ Automatic Testing Oracles require domain expertise @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 Verify Behavior
14. @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 Fuzzing Trash deck technique: 1950s - Gerald Weinberg Crash? 14
15. @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 Fuzzing Crash? • Memory Bounds Violation • Privilege Escalation • Safety Violations • Metamorphic Relations • Differential Execution ASAN,MSAN,TSAN,NSan,FuZZan,UBSan 15
16. 16 https://www.fuzzingbook.org
17. @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 [ ; x 1 - G P Z + w c c k c ] ; , N 9 J + ? # 6 ^ 6 \ e ? ] 9 l u 2 _ % ' 4 G X " 0 V U B [ E / r ~ f A p u 6 b 8 < { % s i q 8 Z h . 6 { V , h r ? ; {Ti.r3PIxMMMv6{xS^+'Hq!AxB"YXRS@! Kd6;wtAMefFWM(`|J_<1~o}z3K(CCzRH J I I v H z > _ * . \ > J r l U 3 2 ~ e G P ? lR=bF3+;y$3lodQ<B89!5"W2fK*vE7v{')KC- i,c{<[~m!]o;{.'}Gj\(X}EtYetrpbY@aGZ1{P! A Z U 7 x # 4 ( R t n ! q 4 n C w q o l ^ y 6 } 0 | Ko=*JK~;zMKV=9Nai:wxu{J&UV#HaU)*Bi C < ) , ` + t * g k a < W = Z . % T 5 W G H Z p I 3 0 D < P q > & ] B S 6 R & j ? # t P 7 i a V } - } ` \ ? [ _ [ Z ^ L B M P G - FKj'\xwuZ1=Q`^`5,$N$Q@[!CuRzJ2D|vBy! ^ z k h d f 3 C 5 P A k R ? V ( ( - % > < h n | 3='i2Qx]D$qs4O`1@fevnG'2\11Vf3piU37@ 5 : d f d 4 5 * ( 7 ^ % 5 a p \ z I y l " ' f , $ee,J4Gw:cgNKLie3nx9(`efSlg6#[K"@Wjh Z}r[Scun&sBCS,T[/3]KAeEnQ7lU)3Pn,0)G/ 6N-wyzj/MTd#A;r Program Traditional Fuzzing 17 https://www.fuzzingbook.org/html/Fuzzer.html
18. @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 [ ; x 1 - G P Z + w c c k c ] ; , N 9 J + ? # 6 ^ 6 \ e ? ] 9 l u 2 _ % ' 4 G X " 0 V U B [ E / r ~ f A p u 6 b 8 < { % s i q 8 Z h . 6 { V , h r ? ; {Ti.r3PIxMMMv6{xS^+'Hq!AxB"YXRS@! Kd6;wtAMefFWM(`|J_<1~o}z3K(CCzRH J I I v H z > _ * . \ > J r l U 3 2 ~ e G P ? lR=bF3+;y$3lodQ<B89!5"W2fK*vE7v{')KC- i,c{<[~m!]o;{.'}Gj\(X}EtYetrpbY@aGZ1{P! A Z U 7 x # 4 ( R t n ! q 4 n C w q o l ^ y 6 } 0 | Ko=*JK~;zMKV=9Nai:wxu{J&UV#HaU)*Bi C < ) , ` + t * g k a < W = Z . % T 5 W G H Z p I 3 0 D < P q > & ] B S 6 R & j ? # t P 7 i a V } - } ` \ ? [ _ [ Z ^ L B M P G - FKj'\xwuZ1=Q`^`5,$N$Q@[!CuRzJ2D|vBy! ^ z k h d f 3 C 5 P A k R ? V h n | 3='i2Qx]D$qs4O`1@fevnG'2\11Vf3piU37@ 5 5 a p \ z I y l " ' f , $ee,J4Gw:cgNKLie3nx9(`efSlg6#[K"@Wjh Z}r[Scun&sBCS,T[/3]KAeEnQ7lU)3Pn,0)G/ 6N-wyzj/MTd#A;r Structured Inputs SYNTAX ERROR ✘ 18
19. def process_input(input) : try : ✘val = parse(input ) res = process(val ) return re s except SyntaxError : return Erro r SYNTAX ERROR 19 Parser
20. SYNTAX ERROR def process_input(input) : try : ✘val = parse(input ) res = process(val ) return re s except SyntaxError : return Erro r 20 The Core
21. 21 def process_input(input) : try : val = parse(input ) res = process(val ) return re s except SyntaxError : return Erro r SYNTAX ERROR 21
22. • Insert Instrumentatio n • Generate input s • Collect execution feedbac k • Branches covered during executio n • Slightly Mutate Input and try agai n Collect inputs obtaining new coverage Feedback Driven Fuzzing 22 22 https://www.fuzzingbook.org/html/MutationFuzzer.html
23. 23 def triangle(a, b, c): if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene def triangle(a, b, c): __probe_enter() if a == b: __probe_1() if b == c: __probe_2() return Equilateral else: __probe_3() return Isosceles else: __probe_4() if b == c: __probe_5() return Isosceles else: __probe_6() if a == c: __probe_7() return Isosceles else: __probe_8() return Scalene def triangle(a, b, c): if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene Feedback Driven Fuzzing • Insert Instrumentatio n • Generate input s • Collect execution feedbac k • Branches covered during executio n • Slightly Mutate Input and try agai n Collect inputs obtaining new coverage 23 https://www.fuzzingbook.org/html/MutationFuzzer.html
24. 24 Feedback Driven Fuzzing triangle (1,1,1) def triangle(a, b, c): if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene • Insert Instrumentatio n • Generate input s • Collect execution feedbac k • Branches covered during executio n • Slightly Mutate Input and try agai n Collect inputs obtaining new coverage 24 https://www.fuzzingbook.org/html/MutationFuzzer.html
25. 25 Feedback Driven Fuzzing triangle (1,1,1) def triangle(a, b, c): if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene • Insert Instrumentatio n • Generate input s • Collect execution feedbac k • Branches covered during executio n • Slightly Mutate Input and try agai n Collect inputs obtaining new coverage 25 https://www.fuzzingbook.org/html/MutationFuzzer.html
26. triangle (1,1,1) 26 Feedback Driven Fuzzing triangle (1,1,2) def triangle(a, b, c): if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene • Insert Instrumentatio n • Generate input s • Collect execution feedbac k • Branches covered during executio n • Slightly Mutate Input and try agai n Collect inputs obtaining new coverage Mutated 26 https://www.fuzzingbook.org/html/MutationFuzzer.html
27. 27 Feedback Driven Fuzzing triangle (1,1,3) def triangle(a, b, c): if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene • Insert Instrumentatio n • Generate input s • Collect execution feedbac k • Branches covered during executio n • Slightly Mutate Input and try agai n Collect inputs obtaining new coverage Mutated 27 https://www.fuzzingbook.org/html/MutationFuzzer.html
28. 28 Feedback Driven Fuzzing triangle (1,1,2) def triangle(a, b, c): if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene • Insert Instrumentatio n • Generate input s • Collect execution feedbac k • Branches covered during executio n • Slightly Mutate Input and try agai n Collect inputs obtaining new coverage 28 https://www.fuzzingbook.org/html/MutationFuzzer.html triangle (1,1,1) triangle (1,1,3)
29. • Insert Instrumentatio
30. 30 Feedback Driven Fuzzing Weakness: static int is_reserved_word_token(const char *s, int len) { const char *reserved[] = { "break", "case", "catch", "continue", "debugger", "default", "delete", "do", "else", "false", "finally", "for", "function", "if", "in", "instanceof", "new", "null", "return", "switch", "this", "throw", "true", "try", "typeof", "var", "void", "while", "with", "let", "undefined", ((void *)0)}; int i; if (!mjs_is_alpha(s[0])) return 0; for (i = 0; reserved[i] != ((void *)0); i++) { if (len == (int)strlen(reserved[i]) && strncmp(s, reserved[i], len) == 0) return i + 1; } return 0; } Tokens if (x > 100) { } coverage: 20% if (x > 100) { } e coverage: 5% if (x > 100) { } el coverage: 5% if (x > 100) { } els coverage: 5% if (x > 100) { } else coverage: 25% No smooth coverage gradient in parsers 30
31. 31 Feedback Driven Fuzzing def json_raw(stm) : while True : stm.skipspaces( ) c = stm.peek( ) if c == 't' : return json_fixed(stm, 'true' ) elif c == 'f' : return json_fixed(stm, 'false' ) elif c == 'n': return json_fixed(stm, 'null' ) elif c == '"': return json_string(stm ) elif c == '{': return json_dict(stm ) elif c == '[': return json_list(stm ) elif c in NUMSTART : return json_number(stm ) raise JSONError(E_MALF, stm, stm.pos) Weak points : • Need for smooth coverage gradien t • Coverage only provides first level guidance 1. {"abc":[]} 2. [{"a":[]}, {"b":[]}, {"c":["ab","c"]}] 31
32. 32 Feedback Driven Fuzzing def json_raw(stm) : while True : stm.skipspaces( ) c = stm.peek( ) if c == 't' : return json_fixed(stm, 'true' ) elif c == 'f' : return json_fixed(stm, 'false' ) elif c == 'n': return json_fixed(stm, 'null' ) elif c == '"': return json_string(stm ) elif c == '{': return json_dict(stm ) elif c == '[': return json_list(stm ) elif c in NUMSTART : return json_number(stm ) raise JSONError(E_MALF, stm, stm.pos) Weak points : • Need for smooth coverage gradien t • Coverage only provides first level guidance 1. {"abc":[]} 2. [{"a":[]}, {"b":[]}, {"c":["ab","c"]}] 32
33. 33 Feedback Driven Fuzzing def json_raw(stm) : while True : stm.skipspaces( ) c = stm.peek( ) if c == 't' : return json_fixed(stm, 'true' ) elif c == 'f' : return json_fixed(stm, 'false' ) elif c == 'n': return json_fixed(stm, 'null' ) elif c == '"': return json_string(stm ) elif c == '{': return json_dict(stm ) elif c == '[': return json_list(stm ) elif c in NUMSTART : return json_number(stm ) raise JSONError(E_MALF, stm, stm.pos) Weak points : • Need for smooth coverage gradien t • Coverage only provides first level guidance 1. {"abc":[]} 2. [{"a":[]}, {"b":[]}, {"c":["ab","c"]}] 33
34. 34 Feedback Driven Fuzzing 34 • Insert Instrumentatio n • Generate input s • Collect execution feedbac k • Branches covered during executio n • Slightly Mutate Input and try agai n Collect inputs obtaining new coverage
35. 35 Solver Directed Fuzzing • Collect path constraint s • Solve negated constraints for new inputs (a == b) (b == c) (b != c) triangle(1,2,1) https://www.fuzzingbook.org/html/ConcolicFuzzer.html 35
36. 36 Solver Directed Fuzzing • Collect path constraint s • Solve constraints for new inputs void next_sym() { while(!eof) { switch (ch){ case '{': next_ch(); sym = LBRA; return; case '}': next_ch(); sym = RBRA; return; case '(': next_ch(); sym = LPAR; return; case ')': next_ch(); sym = RPAR; return; case '+': next_ch(); sym = PLUS; return; case '-': next_ch(); sym = MINUS; return; case '<': next_ch(); sym = LESS; return; case ';': next_ch(); sym = SEMI; return; case '=': next_ch(); sym = EQUAL; return; default: if (ch >= '0' && ch <= '9') { int_val = 0; /* missing overflow check */ while (ch >= '0' && ch <= '9') { int_val = int_val*10 + (ch - '0'); next_ch(); } sym = INT; } else if (ch >= 'a' && ch <= 'z') { int i = 0; /* missing overflow check */ while ((ch >= 'a' && ch <= 'z') || ch == '_'){ id_name[i++] = ch; next_ch(); } id_name[i] = '\0'; sym = 0; while (words[sym] != NULL && strcmp(words[sym], id_name) != 0) sym++; if (words[sym] == NULL) if (id_name[1] == '\0') sym = ID; else syntax_error(); } else syntax_error(); return; } Weak point: Path explosion https://www.fuzzingbook.org/html/ConcolicFuzzer.html 36
37. 37 Solver Directed Fuzzing • Collect path constraint s • Solve negated constraints for new inputs 37
38. 38 Specialized Generators • Specialize generation for a domain 38
39. 39 Overcoming Parsers
40. Needed: Input Language 40 40
41. 41 Formal Languages Language Descriptions: Grammars 3. Regular Context Free Recursively Enumerable (Chomsky,1956) Easy to produce and parse Argument Stack Return Stack 41
42. 42 Grammar JSON grammar De f inition for <elt> <elt> key <json> ::= <elt > 
 <elt> ::= <object>
 | <array>
 | <string>
 | <number>
 | `true ` | `false ` | `null ` <object> ::= `{`<items>`}`| `{}` <items> ::= <item>`,`<items> | <item> <item> ::= <string>`:`<elt> <array> ::= `[`<elts>`]` | `[]` <elts> ::= <elt>`,`<elts> | <elt> <string> ::= `"` <chars> `"` | `""` <chars> ::= <char><chars> | <char> <char> ::= [A-Za-z0-9] <number> ::= <digits> <digits> ::= <digit><digits> | <digit> 42
43. 43 Grammar JSON grammar <json> ::= <elt > 
 <elt> ::= <object>
 | <array>
 | <string>
 | <number>
 | `true ` | `false ` | `null ` <object> ::= `{`<items>`}`| `{}` <items> ::= <item>`,`<items> | <item> <item> ::= <string>`:`<elt> <array> ::= `[`<elts>`]` | `[]` <elts> ::= <elt>`,`<elts> | <elt> <string> ::= `"` <chars> `"` | `""` <chars> ::= <char><chars> | <char> <char> ::= [A-Za-z0-9] <number> ::= <digits> <digits> ::= <digit><digits> | <digit> Expansion Rule Terminal Symbol Nonterminal Symbol 43
44. <json> ::= <elt>
 <elt> ::= <object>
 | <array>
 | <string>
 | <number>
 | `true` | `false` | `null` <object> ::= `{`<items>`}`| `{}` <items> ::= <item>`,`<items> | <item> <item> ::= <string>`:`<elt> <array> ::= `[`<elts>`]` | `[]` <elts> ::= <elt>`,`<elts> | <elt> <string> ::= `"` <chars> `"` | `""` <chars> ::= <char><chars> | <char> <char> ::= [A-Za-z0-9] <number> ::= <digits> <digits> ::= <digit><digits> | <digit> <digit> ::= [0-9] Tree 44
45. <json> ::= <elt>
 <elt> ::= <object>
 | <array>
 | <string>
 | <number>
 | `true` | `false` | `null` <object> ::= `{`<items>`}`| `{}` <items> ::= <item>`,`<items> | <item> <item> ::= <string>`:`<elt> <array> ::= `[`<elts>`]` | `[]` <elts> ::= <elt>`,`<elts> | <elt> <string> ::= `"` <chars> `"` | `""` <chars> ::= <char><chars> | <char> <char> ::= [A-Za-z0-9] <number> ::= <digits> <digits> ::= <digit><digits> | <digit> <digit> ::= [0-9] <json> Tree <elt> 45
46. <json> ::= <elt>
 <elt> ::= <object>
 | <array>
 | <string>
 | <number>
 | `true` | `false` | `null` <object> ::= `{`<items>`}`| `{}` <items> ::= <item>`,`<items> | <item> <item> ::= <string>`:`<elt> <array> ::= `[`<elts>`]` | `[]` <elts> ::= <elt>`,`<elts> | <elt> <string> ::= `"` <chars> `"` | `""` <chars> ::= <char><chars> | <char> <char> ::= [A-Za-z0-9] <number> ::= <digits> <digits> ::= <digit><digits> | <digit> <digit> ::= [0-9] <elt> <object> Tree 46
47. <json> ::= <elt>
 <elt> ::= <object>
 | <array>
 | <string>
 | <number>
 | `true` | `false` | `null` <object> ::= `{`<items>`}`| `{}` <items> ::= <item>`,`<items> | <item> <item> ::= <string>`:`<elt> <array> ::= `[`<elts>`]` | `[]` <elts> ::= <elt>`,`<elts> | <elt> <string> ::= `"` <chars> `"` | `""` <chars> ::= <char><chars> | <char> <char> ::= [A-Za-z0-9] <number> ::= <digits> <digits> ::= <digit><digits> | <digit> <digit> ::= [0-9] <object> { <items> } Tree 47
48. <json> ::= <elt>
 <elt> ::= <object>
 | <array>
 | <string>
 | <number>
 | `true` | `false` | `null` <object> ::= `{`<items>`}`| `{}` <items> ::= <item>`,`<items> | <item> <item> ::= <string>`:`<elt> <array> ::= `[`<elts>`]` | `[]` <elts> ::= <elt>`,`<elts> | <elt> <string> ::= `"` <chars> `"` | `""` <chars> ::= <char><chars> | <char> <char> ::= [A-Za-z0-9] <number> ::= <digits> <digits> ::= <digit><digits> | <digit> <digit> ::= [0-9] { <items> } { <item> } Tree 48
49. <json> ::= <elt>
 <elt> ::= <object>
 | <array>
 | <string>
 | <number>
 | `true` | `false` | `null` <object> ::= `{`<items>`}`| `{}` <items> ::= <item>`,`<items> | <item> <item> ::= <string>`:`<elt> <array> ::= `[`<elts>`]` | `[]` <elts> ::= <elt>`,`<elts> | <elt> <string> ::= `"` <chars> `"` | `""` <chars> ::= <char><chars> | <char> <char> ::= [A-Za-z0-9] <number> ::= <digits> <digits> ::= <digit><digits> | <digit> <digit> ::= [0-9] { <item> } { <string> : <elt> } Tree 49
50. <json> ::= <elt>
 <elt> ::= <object>
 | <array>
 | <string>
 | <number>
 | `true` | `false` | `null` <object> ::= `{`<items>`}`| `{}` <items> ::= <item>`,`<items> | <item> <item> ::= <string>`:`<elt> <array> ::= `[`<elts>`]` | `[]` <elts> ::= <elt>`,`<elts> | <elt> <string> ::= `"` <chars> `"` | `""` <chars> ::= <char><chars> | <char> <char> ::= [A-Za-z0-9] <number> ::= <digits> <digits> ::= <digit><digits> | <digit> <digit> ::= [0-9] { <string> : <elt> } { "" : <elt> } Tree 50
51. <json> ::= <elt>
 <elt> ::= <object>
 | <array>
 | <string>
 | <number>
 | `true` | `false` | `null` <object> ::= `{`<items>`}`| `{}` <items> ::= <item>`,`<items> | <item> <item> ::= <string>`:`<elt> <array> ::= `[`<elts>`]` | `[]` <elts> ::= <elt>`,`<elts> | <elt> <string> ::= `"` <chars> `"` | `""` <chars> ::= <char><chars> | <char> <char> ::= [A-Za-z0-9] <number> ::= <digits> <digits> ::= <digit><digits> | <digit> <digit> ::= [0-9] { "" : <elt> } Tree 51
52. <json> ::= <elt>
 <elt> ::= <object>
 | <array>
 | <string>
 | <number>
 | `true` | `false` | `null` <object> ::= `{`<items>`}`| `{}` <items> ::= <item>`,`<items> | <item> <item> ::= <string>`:`<elt> <array> ::= `[`<elts>`]` | `[]` <elts> ::= <elt>`,`<elts> | <elt> <string> ::= `"` <chars> `"` | `""` <chars> ::= <char><chars> | <char> <char> ::= [A-Za-z0-9] <number> ::= <digits> <digits> ::= <digit><digits> | <digit> <digit> ::= [0-9] { "" : <elt> } { "" : true } Tree 52
53. <json> ::= <elt>
 <elt> ::= <object>
 | <array>
 | <string>
 | <number>
 | `true` | `false` | `null` <object> ::= `{`<items>`}`| `{}` <items> ::= <item>`,`<items> | <item> <item> ::= <string>`:`<elt> <array> ::= `[`<elts>`]` | `[]` <elts> ::= <elt>`,`<elts> | <elt> <string> ::= `"` <chars> `"` | `""` <chars> ::= <char><chars> | <char> <char> ::= [A-Za-z0-9] <number> ::= <digits> <digits> ::= <digit><digits> | <digit> <digit> ::= [0-9] {"":true} Derivation Tree 53
54. {
 '<json>' : [['<elt>']] , '<elt>' : [['<object>'] , ['<array>'] , ['<string>'] , ['<number>'] , ['true'], ['false'], ['null']] , '<object>' : [['{', '<items>','}'] , ['{}']] , '<items>' : [['<item>,',',<items>'] , ['<item>']] , '<item>' : [['<string>',':', '<elt>']] , '<array>' : [['[', '<elts>', ']'] , ['[]']] , '<elts>' : [['<elt>,',',<elts>'] , ['<elt>']] , '<string>' : [['"', '<chars>', '"'] , ['""']] , '<chars>' : [['<char>','<chars>'] , ['<char>']] , '<number>' : [['<digits>']] , '<digits>' : [['<digit>','<digits>'] , ['<digit>']] , '<char>' : [[c] for c in string.characters ] '<digit>' : [[c] for c in string.digits]
 } Grammar Fuzzer 54
55. General Parser {
 '<json>' : [['<elt>']] , '<elt>' : [['<object>'] , ['<array>'] , ['<string>'] , ['<number>'] , ['true'], ['false'], ['null']] , '<object>' : [['{', '<items>','}'] , ['{}']] , '<items>' : [['<item>,',',<items>'] , ['<item>']] , '<item>' : [['<string>',':', '<elt>']] , '<array>' : [['[', '<elts>', ']'] , ['[]']] , '<elts>' : [['<elt>,',',<elts>'] , ['<elt>']] , '<string>' : [['"', '<chars>', '"'] , ['""']] , '<chars>' : [['<char>','<chars>'] , ['<char>']] , '<number>' : [['<digits>']] , '<digits>' : [['<digit>','<digits>'] , ['<digit>']] , '<char>' : [[c] for c in string.characters ] '<digit>' : [[c] for c in string.digits]
 } 55
56. {
 '<json>' : [['<elt>']] , '<elt>' : [['<object>'] , ['<array>'] , ['<string>'] , ['<number>'] , ['true'], ['false'], ['null']] , '<object>' : [['{', '<items>','}'] , ['{}']] , '<items>' : [['<item>,',',<items>'] , ['<item>']] , '<item>' : [['<string>',':', '<elt>']] , '<array>' : [['[', '<elts>', ']'] , ['[]']] , '<elts>' : [['<elt>,',',<elts>'] , ['<elt>']] , '<string>' : [['"', '<chars>', '"'] , ['""']] , '<chars>' : [['<char>','<chars>'] , ['<char>']] , '<number>' : [['<digits>']] , '<digits>' : [['<digit>','<digits>'] , ['<digit>']] , '<char>' : [[c] for c in string.characters ] '<digit>' : [[c] for c in string.digits]
 } Lang Fuzzer https://www.fuzzingbook.org/html/LangFuzzer.html 56
57. def process_input(input) : try : ✔val = parse(input ) res = process(val ) return re s except SyntaxError : return Erro r {
 '<json>' : [['<elt>']] , '<elt>' : [['<object>'] , ['<array>'] , ['<string>'] , ['<number>'] , ['true'], ['false'], ['null']] , '<object>' : [['{', '<items>','}'] , ['{}']] , '<items>' : [['<item>,',',<items>'] , ['<item>']] , '<item>' : [['<string>',':', '<elt>']] , '<array>' : [['[', '<elts>', ']'] , ['[]']] , '<elts>' : [['<elt>,',',<elts>'] , ['<elt>']] , '<string>' : [['"', '<chars>', '"'] , ['""']] , '<chars>' : [['<char>','<chars>'] , ['<char>']] , '<number>' : [['<digits>']] , '<digits>' : [['<digit>','<digits>'] , ['<digit>']] , '<char>' : [[c] for c in string.characters ] '<digit>' : [[c] for c in string.digits]
 } 57
58. Where to Get the Grammar From? 58
59. Almost Everyone Uses Handwritten Parsers https://notes.eatonphil.com/parser-generators-vs-handwritten-parsers-survey-2021.html 59
60. Where to Get the Grammar From? 60
61. 61 "Be liberal in what you accept, and conservative in what you send" Postel's Law 61
62. QUIRK_ALLOW_ASCII_CONTROL_CODES QUIRK_ALLOW_BACKSLASH_A QUIRK_ALLOW_BACKSLASH_CAPITAL_U QUIRK_ALLOW_BACKSLASH_E QUIRK_ALLOW_BACKSLASH_NEW_LINE QUIRK_ALLOW_BACKSLASH_QUESTION_MARK QUIRK_ALLOW_BACKSLASH_SINGLE_QUOTE QUIRK_ALLOW_BACKSLASH_V QUIRK_ALLOW_BACKSLASH_X_AS_BYTE S QUIRK_ALLOW_BACKSLASH_X_AS_CODE_POINTS QUIRK_ALLOW_BACKSLASH_ZERO QUIRK_ALLOW_COMMENT_BLOCK QUIRK_ALLOW_COMMENT_LINE QUIRK_ALLOW_EXTRA_COMMA QUIRK_ALLOW_INF_NAN_NUMBERS QUIRK_ALLOW_LEADING_ASCII_RECORD_SEPARATOR QUIRK_ALLOW_LEADING_UNICODE_BYTE_ORDER_MARK QUIRK_ALLOW_TRAILING_FILLER QUIRK_EXPECT_TRAILING_NEW_LINE_OR_EOF QUIRK_JSON_POINTER_ALLOW_TILDE_N_TILDE_R_TILDE_T QUIRK_REPLACE_INVALID_UNICODE JSON common quirks from https://github.com/google/wuffs 62
63. "Be liberal in what you accept, and conservative in what you send"
 Postel's Law The Specification The Implementation Extra "Features" Where to Get the Grammar From? 63
64. 64 Where to Get the Grammar From? Hand-written parsers already encode the grammar 64
65. def json_raw(stm) : while True : stm.skipspaces( ) c = stm.peek( ) if c == 't' : return json_fixed(stm, 'true' ) elif c == 'f' : return json_fixed(stm, 'false' ) elif c == 'n': return json_fixed(stm, 'null' ) elif c == '"': return json_string(stm ) elif c == '{': return json_dict(stm ) elif c == '[': return json_list(stm ) elif c in NUMSTART : return json_number(stm ) raise JSONError(E_MALF, stm, stm.pos) <json_raw>::= <json_object>
 | <json_list>
 | <json_string>
 | <json_number>
 | <json_fixed(true)> | <json_fixed(false) > | <json_fixed(null) > <json_string> ::= `"` <chars> `" ` | `"" ` <chars> ::= <char><chars> | <char > <json_object> ::= `{`<items>`} ` | `{} ` <items> ::= <item>`,`<items > | <item > <item> ::= <string>`:`<elt > <json_list> ::= `[`<elts>`] ` | `[] ` <elts> ::= <elt>`,`<elts > | <elt > <json_number> ::= <digits > <digits> ::= <digit><digits> | <digit> https://github.com/phensley/microjson MicroJSON 65 65
66. def json_raw(stm) : while True : stm.skipspaces( ) c = stm.peek( ) if c == 't' : return json_fixed(stm, 'true' ) elif c == 'f' : return json_fixed(stm, 'false' ) elif c == 'n': return json_fixed(stm, 'null' ) elif c == '"': return json_string(stm ) elif c == '{': return json_dict(stm ) elif c == '[': return json_list(stm ) elif c in NUMSTART : return json_number(stm ) raise JSONError(E_MALF, stm, stm.pos) <json_raw>::= <json_object>
 | <json_list>
 | <json_string>
 | <json_number>
 | <json_fixed(true)> | <json_fixed(false) > | <json_fixed(null)> <json_string> ::= `"` <chars> `" ` | `"" ` <chars> ::= <char><chars> | <char> <json_object> ::= `{`<items>`} ` | `{}` <items> ::= <item>`,`<items > | <item> <item> ::= <string>`:`<elt> <json_list> ::= `[`<elts>`] ` | `[]` <elts> ::= <elt>`,`<elts > | <elt> <json_number> ::= <digits > <digits> ::= <digit><digits> | <digit> https://github.com/phensley/microjson MicroJSON 66
67. <F> := <A > | <B> <F> := <A> <B> <C> <Fs> := <B> <Fs > | <empty> Structured Control Flow to Grammar Sequence A B C [F] Selection cond A B [F] F T Iteration cond B [F] 67
68. def json_raw(stm) : while True : stm.skipspaces( ) c = stm.peek( ) if c == 't' : return json_fixed(stm, 'true' ) elif c == 'f' : return json_fixed(stm, 'false' ) elif c == 'n': return json_fixed(stm, 'null' ) elif c == '"': return json_string(stm ) elif c == '{': return json_object(stm ) elif c == '[': return json_array(stm ) elif c in NUMSTART : return json_number(stm ) raise JSONError(E_MALF, stm, stm.pos) <json_raw> ::= <loop1> <loop1> ::= <iter1> <loop1 > | <empty > <iter1>::= <json_fixed(true) > | <json_fixed(false) > | <json_fixed(null) > 
 | `"` <json_string > | `{` <json_object > 
 | `[` <json_array > 
 | [[1-9] <json_number>
 68
69. def json_raw(stm) : while True : stm.skipspaces( ) c = stm.peek( ) if c == 't' : return json_fixed(stm, 'true' ) elif c == 'f' : return json_fixed(stm, 'false' ) elif c == 'n': return json_fixed(stm, 'null' ) elif c == '"': return json_string(stm ) elif c == '{': return json_object(stm ) elif c == '[': return json_array(stm ) elif c in NUMSTART : return json_number(stm ) raise JSONError(E_MALF, stm, stm.pos) <json_raw> ::= <loop1> <loop1> ::= <iter1> <loop1 > | <empty > <iter1>::= <json_fixed(true) > | <json_fixed(false) > | <json_fixed(null) > 
 | `"` <json_string > | `{` <json_object > 
 | `[` <json_array > 
 | [[1-9] <json_number>
 69
70. def json_string(stm) : # skip over '" ' stm.next() r = [ ] while True : c = stm.next( ) if c == '' : raise JSONError(E_TRUNC ) elif c == '\\' : c = stm.next( ) r.append(decode_escape(c, stm) ) elif c == '"' : return ''.join(r ) else : r.append(c) <json_raw> ::= <loop1> ... <iter1>::= `"` <json_string > .. . <json_string> ::= <loop2 > <loop2> ::= <iter2> <loop2 > | <empty > <iter2>::= `\\` <decode_escape > | `" ` | <other_char> "ab" <json_string> ::= <loop2> <loop2_end > <loop2> ::= <iter2> <loop2 > | <empty > <iter2>::= `\\` <decode_escape > | <other_char > <loop2_end> ::= <iter2_end > <iter2_end>::=`"` 70
71. 71 <START> ::= <json_raw > <json_raw> ::= '"' <json_string' > | '[' <json_list' > | '{' <json_dict' > | <json_number' > | 'true ' | 'false ' | 'null ' <json_number'> ::= <json_number> + | <json_number>+ 'e' <json_number> + <json_number> ::= '+' | '-' | '.' | [0-9] | 'E' | 'e ' <json_string'> ::= <json_string>* '" ' <json_list'> ::= '] ' | <json_raw> (','<json_raw>)* '] ' | ( ',' <json_raw>)+ (',' <json_raw>)* '] ' <json_dict'> ::= '} ' | ( '"' <json_string'> ':' <json_raw> ',' )* '"'<json_string'> ':' <json_raw> '} ' <json_string> ::= ' ' | '!' | '#' | '$' | '%' | '&' | '' ' | '*' | '+' | '-' | ',' | '.' | '/' | ':' | '; ' | '<' | '=' | '>' | '?' | '@' | '[' | ']' | '^ ' | '_', ''',| '{' | '|' | '}' | '~ ' | '[A-Za-z0-9] ' | '\' <decode_escape > <decode_escape> ::= '"' | '/' | 'b' | 'f' | 'n' | 'r' | 't' stm.next()
72. Recall Subjects Mimid calc.py 100.0 % mathexpr.py 87.5 % cgidecode.py 100.0 % urlparse.py 100.0 % microjson.py 98.7 % parseclisp.py 99.3 % jsonparser.c 100.0 % tiny.c 100.0 % mjs.c 95.4 % Inputs generated by inferred grammar that were accepted by the program Subjects Mimid calc.py 100.0 % mathexpr.py 92.7 % cgidecode.py 100.0 % urlparse.py 96.4 % microjson.py 93.0 % parseclisp.py 80.6 % jsonparser.c 83.8 % tiny.c 92.8 % mjs.c 95.9 % Inputs generated by golden grammar that were accepted by the inferred grammar parser Precision Evaluation: Accuracy 72 72
73. 73 <START> ::= <json_raw
74. 74
75. 75 HTTP Parser XML Parser SOAP Parser RPC Parser C Parser Check Declarations Check Types Static Checks Challenges Compilers Servers Semantics Application
76. 76 76
77. What Can We Do With Grammars? 77 77
78. {
 '<json>' : [['<elt>']] , '<elt>' : [['<object>'] , ['<array>'] , ['<string>'] , ['<number>'] , ['true'], ['false'], ['null']] , '<object>' : [['{', '<items>','}'], ['{}']] , '<items>' : [['<item>,',',<items>'], ['<item>']] , '<item>' : [['<string>',':', '<elt>']] , '<array>' : [['[', '<elts>', ']'], ['[]']] , '<elts>' : [['<elt>,',',<elts>'], ['<elt>']] , '<string>' : [['"', '<chars>', '"'], ['""']] , '<chars>' : [['<char>','<chars>'], ['<char>']] , '<number>' : [['<digits>']] , '<digits>' : [['<digit>','<digits>'], ['<digit>']] , '<char>' : [[c] for c in string.characters ] '<digit>' : [[c] for c in string.digits]
 } def process_input(input) : try : ✔val = parse(input ) res = process(val ) return re s except SyntaxError : return Erro r Fuzz Programs? 78
79. Grammar Fuzzer Fast Fuzzers def json() : elt( ) def elt() : match (random() % 7) : case 0: object( ) case 1: array( ) case 2: string( ) case 3: number( ) case 4: print('true' ) case 5: print('false' ) case 5: print('null')
 def object() : match (random() % 2) : case 0: print('{'); items(); print('}' ) case 1: print('{');print('}' ) def array() : match (random() % 2) : case 0: print('['); elts(); print(']' ) case 1: print('[');print(']' ) def items() : match (random() % 2) : case 0: item(); print(','); items( ) case 1: item() Compiled Grammar (F1) Building Fast Fuzzers Gopinath and Zeller 2019 def json_0(rops) : elt_0(rops ) def elt_0(rops) : r = next(rops ) if 0 <= r < 36: elt_0( ) elif 36 <= r < 73: elt_1( ) elif 73 <= r < 109: elt_2( ) elif 109 <= r <146: elt_3( ) elif 146 <= r < 182: elt_4( ) elif 182 <= r < 219: elt_5( ) else: elt_6( ) def elt_0(rops) : r = next(rops ) if 0 <= r < 43: expr_0( ) elif 43 <= r < 85: expr_1( ) elif 85 <= r < 128: expr_2( ) elif 128 <= r < 171: expr_3( ) elif 171 <= r < 213: expr_4( ) else: expr_5( ) print('+' ) elif 171 <= r < 213: expr_4( ) else: expr_5() Grammar VM (F1) 79 {
 '<json>' : [['<elt>']] , '<elt>' : [['<object>'] , ['<array>'] , ['<string>'] , ['<number>'] , ['true'], ['false'], ['null']] , '<object>' : [['{', '<items>','}'] , ['{}']] , '<items>' : [['<item>,',',<items>'] , ['<item>']] , '<item>' : [['<string>',':', '<elt>']] , '<array>' : [['[', '<elts>', ']'] , ['[]']] , '<elts>' : [['<elt>,',',<elts>'] , ['<elt>']] , '<string>' : [['"', '<chars>', '"'] , ['""']] , '<chars>' : [['<char>','<chars>'] , ['<char>']] , '<number>' : [['<digits>']] , '<digits>' : [['<digit>','<digits>'] , ['<digit>']] , '<char>' : [[c] for c in string.characters ] '<digit>' : [[c] for c in string.digits]
 }
80. 80 Focused Fuzzing Generate inputs that focus on speci fi c system parts • Covering speci fi c portions of code • Speci fi c paths through the code • Performing speci fi c functions Without damaging the nondeterminism. 80
81. def process_input(input) : try : val = parse(input ) res = process(val ) return re s except SyntaxError : return Erro r {"a": ["key"]} ✓ 81
82. def process_input(input) : try : val = parse(input ) res = process(val ) return re s except SyntaxError : return Erro r {"a": ["key"]} {"": [1,2,"k"]} ✘ 82
83. def process_input(input) : try : val = parse(input ) res = process(val ) return re s except SyntaxError : return Erro r {"a": ["key"]} {"": [1,2,"k"]} ✓ ["A", "B", "C"] 83
84. def process_input(input) : try : val = parse(input ) res = process(val ) return re s except SyntaxError : return Erro r {"a": ["key"]} {"": [1,2,"k"]} ["A", "B", "C"] [{"": [1,2,3,4]}] ✘ 84
85. def process_input(input) : try : val = parse(input ) res = process(val ) return re s except SyntaxError : return Erro r {"a": ["key"]} {"": [1,2,"k"]} ["A", "B", "C"] [{"": [1,2,3,4]}] if json.has_key("") : raise Exception() 85
86. def process_input(input) : try : val = parse(input ) res = process(val ) return re s except SyntaxError : return Erro r {"type":"PathNode","matrix": {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948 628471,"dy":371.5686591257294},"children": [],"strokeStyle":"#000000"," fi llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5, "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r": [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415 6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r": [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971 7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r": [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]} ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r": [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}], "shadow":false},{"type":"PathNode","matrix": {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328 55618,"dy":133.20628077515605},"children": [],"strokeStyle":"#000000"," fi llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star tX":126.25,"startY":127.50445838342671,"closed":true,"segments": [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127 .50445838342671,"r": [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439 5]}, {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049 55,"r": [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]}, {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r": [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337 9]}, {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834 2671,"r": [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]} ],"shadow":false},{"type":"TextNode","matrix": {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children": []," fi llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20}, {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children": []," fi llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20}, {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children": [],"arrowSize":10,"path":{"type":"PathNode","matrix": {"m11":1,"m12":0,"m21":0,"m22":1,"dx":464,"dy":-3},"children": [],"strokeStyle":"#000000"," fi llStyle":"#ffffff","lineWidth":2,"smoothness":0.3," ✘ 86
87. {"type":"PathNode","matrix": {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948 628471,"dy":371.5686591257294},"children": [],"strokeStyle":"#000000"," fi llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5, "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r": [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415 6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r": [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971 7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r": [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]} ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r": [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}], "shadow":false},{"type":"PathNode","matrix": {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328 55618,"dy":133.20628077515605},"children": [],"strokeStyle":"#000000"," fi llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star tX":126.25,"startY":127.50445838342671,"closed":true,"segments": [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127 .50445838342671,"r": [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439 5]}, {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049 55,"r": [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]}, {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r": [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337 9]}, {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834 2671,"r": [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]} ],"shadow":false},{"type":"TextNode","matrix": {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children": []," fi llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20}, {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children": []," fi llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20}, {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children": [],"arrowSize":10,"path":{"type":"PathNode","matrix": What is the smallest failure inducing input? Delta Debugging 87
88. <json> ::= <elt>
 <elt> ::= <object>
 | <array>
 | <string>
 | <number>
 | `true` | `false ` | `null ` <string> ::= `"` <chars> `" ` | `"" ` <chars> ::= <char><chars> | <char> <object> ::= `{`<items>`}`| `{}` <items> ::= <item>`,`<items > | <item> <item> ::= <string>`:`<elt> <array> ::= `[`<elts>`]`| `[]` <elts> ::= <elt>`,`<elts > | <elt> <number> ::= <digits > <digits> ::= <digit><digits> | <digit> Hierarchical Delta Debugging {"type":"PathNode","matrix": {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948 628471,"dy":371.5686591257294},"children": [],"strokeStyle":"#000000"," fi llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5, "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r": [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415 6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r": [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971 7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r": [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]} ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r": [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}], "shadow":false},{"type":"PathNode","matrix": {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328 55618,"dy":133.20628077515605},"children": [],"strokeStyle":"#000000"," fi llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star tX":126.25,"startY":127.50445838342671,"closed":true,"segments": [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127 .50445838342671,"r": [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439 5]}, {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049 55,"r": [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]}, {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r": [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337 9]}, {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834 2671,"r": [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]} ],"shadow":false},{"type":"TextNode","matrix": {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children": []," fi llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20}, {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children": []," fi llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20}, {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children": [],"arrowSize":10,"path":{"type":"PathNode","matrix": 88
89. {"":[]} ✘ 89 Test Minimization {"type":"PathNode","matrix": {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948 628471,"dy":371.5686591257294},"children": [],"strokeStyle":"#000000"," fi llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5, "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r": [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415 6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r": [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971 7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r": [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]} ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r": [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}], "shadow":false},{"type":"PathNode","matrix": {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328 55618,"dy":133.20628077515605},"children": [],"strokeStyle":"#000000"," fi llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star tX":126.25,"startY":127.50445838342671,"closed":true,"segments": [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127 .50445838342671,"r": [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439 5]}, {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049 55,"r": [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]}, {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r": [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337 9]}, {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834 2671,"r": [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]} ],"shadow":false},{"type":"TextNode","matrix": {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children": []," fi llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20}, {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children": []," fi llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20}, {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children": [],"arrowSize":10,"path":{"type":"PathNode","matrix": Hierarchical Delta Debugging 89
90. 90 Why? [12345] {"":[]} {"":0} {"x":[]} {"type":"PathNode","matrix": {"m11":-0.6630394213564543,"m12":0,"m21":0,"m22":0.5236476835782672,"dx":565.5201948 628471,"dy":371.5686591257294},"children": [],"strokeStyle":"#000000"," fi llStyle":"#e1e1e1","lineWidth":4,"smoothness":0.3,"sloppiness":0.5, "startX":50,"startY":0,"closed":true,"segments":[{"type":3,"x":100,"y":50,"x1":100,"y1":0,"r": [-0.3779207859188318,0.07996635790914297,-0.47163885831832886,-0.0710031278431415 6]},{"type":3,"x":50,"y":100,"x1":100,"y1":100,"r": [0.24857700895518064,0.030472169630229473,0.49844827968627214,0.1326016811653971 7]},{"type":3,"x":0,"y":50,"x1":0,"y1":100,"r": [0.1751830680295825,-0.18606301862746477,-0.4092112798243761,-0.4790717279538512]} ,{"type":3,"x":50,"y":0,"x1":0,"y1":0,"r": [0.37117584701627493,0.3612578883767128,0.0462839687243104,-0.1564063960686326]}], "shadow":false},{"type":"PathNode","matrix": {"m11":-1.475090930376591,"m12":0,"m21":0,"m22":1.2306765694828008,"dx":700.13810328 55618,"dy":133.20628077515605},"children": [],"strokeStyle":"#000000"," fi llStyle":"#ffffff","lineWidth":2,"smoothness":0.3,"sloppiness":0.5,"star tX":126.25,"startY":127.50445838342671,"closed":true,"segments": [{"type":3,"x":146.01190476190476,"y":147.5936260519611,"x1":146.01190476190476,"y1":127 .50445838342671,"r": [-0.1750196823850274,-0.05804965365678072,-0.3536788672208786,0.05322327278554439 5]}, {"type":3,"x":126.25,"y":167.6827937204955,"x1":146.01190476190476,"y1":167.68279372049 55,"r": [-0.32906053867191076,-0.11536165233701468,0.35579121299088,0.38731588050723076]}, {"type":3,"x":108,"y":147,"x1":106.48809523809524,"y1":167.6827937204955,"r": [0.08825046103447676,0.011088204570114613,0.43411328736692667,-0.133069220930337 9]}, {"type":3,"x":126.25,"y":127.50445838342671,"x1":106.48809523809524,"y1":127.5044583834 2671,"r": [0.42778260353952646,0.24726040940731764,0.3631806019693613,0.05325550492852926]} ],"shadow":false},{"type":"TextNode","matrix": {"m11":1,"m12":0,"m21":0,"m22":1,"dx":543,"dy":225},"children": []," fi llStyle":"#000000","text":"Y","fontName":"FG Virgil","fontSize":20}, {"type":"TextNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":559,"dy":144},"children": []," fi llStyle":"#000000","text":"x","fontName":"FG Virgil","fontSize":20}, {"type":"ArrowNode","matrix":{"m11":1,"m12":0,"m21":0,"m22":1,"dx":0,"dy":0},"children": [],"arrowSize":10,"path":{"type":"PathNode","matrix": Hierarchical Delta Debugging 90
91. DDSET Gopinath, Kampmann, Havrikov, Soremekun, and Zeller. Abstracting Failure Inducing Inputs. ISSTA 2020. https://github.com/vrthra/ddset 91
92. {"": []} DDSET: <json> ::= <elt>
 <elt> ::= <object>
 | <array>
 | <string>
 | <number>
 | `true` | `false ` | `null ` <string> ::= `"` <chars> `" ` | `"" ` <chars> ::= <char><chars> | <char> <object> ::= `{`<items>`} ` | `{}` <items> ::= <item>`,`<items > | <item> <item> ::= <string>`:`<elt> <array> ::= `[`<elts>`] ` | `[]` <elts> ::= <elt>`,`<elts > | <elt> <number> ::= <digits > <digits> ::= <digit><digits> | <digit> 92
93. {"": []} 122489 {"A":{}, {"23": {"P":[]}}] [[], [[[]],[]],{"A":{}, {"23": {"P":[]}}] <json> ::= <elt>
 <elt> ::= <object>
 | <array>
 | <string>
 | <number>
 | `true` | `false ` | `null ` <string> ::= `"` <chars> `" ` | `"" ` <chars> ::= <char><chars> | <char> <object> ::= `{`<items>`} ` | `{}` <items> ::= <item>`,`<items > | <item> <item> ::= <string>`:`<elt> <array> ::= `[`<elts>`] ` | `[]` <elts> ::= <elt>`,`<elts > | <elt> <number> ::= <digits > <digits> ::= <digit><digits> | <digit> DDSET: 93
94. {"": []} "XYZR389" {"A":{}, {"23": {"P":[]}}} [[], [[[]],[]],{"A":{}, {"23": {"P":[]}}] <json> ::= <elt>
 <elt> ::= <object>
 | <array>
 | <string>
 | <number>
 | `true` | `false ` | `null ` <string> ::= `"` <chars> `" ` | `"" ` <chars> ::= <char><chars> | <char> <object> ::= `{`<items>`} ` | `{}` <items> ::= <item>`,`<items > | <item> <item> ::= <string>`:`<elt> <array> ::= `[`<elts>`] ` | `[]` <elts> ::= <elt>`,`<elts > | <elt> <number> ::= <digits > <digits> ::= <digit><digits> | <digit> DDSET: 94
95. {"": []} [true, null] {"A":{}, {"23": {"P":[]}}} [[], [[[]],[]],{"A":{}, {"23": {"P":[]}}] <json> ::= <elt>
 <elt> ::= <object>
 | <array>
 | <string>
 | <number>
 | `true` | `false ` | `null ` <string> ::= `"` <chars> `" ` | `"" ` <chars> ::= <char><chars> | <char> <object> ::= `{`<items>`} ` | `{}` <items> ::= <item>`,`<items > | <item> <item> ::= <string>`:`<elt> <array> ::= `[`<elts>`] ` | `[]` <elts> ::= <elt>`,`<elts > | <elt> <number> ::= <digits > <digits> ::= <digit><digits> | <digit> DDSET: 95
96. {"": []} {"__": [[]]} {"?P":[{}], {"|": {"":[]}}} {"X":[[],[]],{"A":{}, {"2": {"R":[]}}} <json> ::= <elt>
 <elt> ::= <object>
 | <array>
 | <string>
 | <number>
 | `true` | `false ` | `null ` <string> ::= `"` <chars> `" ` | `"" ` <chars> ::= <char><chars> | <char> <object> ::= `{`<items>`} ` | `{}` <items> ::= <item>`,`<items > | <item> <item> ::= <string>`:`<elt> <array> ::= `[`<elts>`] ` | `[]` <elts> ::= <elt>`,`<elts > | <elt> <number> ::= <digits > <digits> ::= <digit><digits> | <digit> DDSET: 96
97. {"": []} {"": [[]]} {"?P":[{}], {"|": {"P":[]}}} {"X":[[],[]],{"A":{}, {"2": {"R":[]}}} 97 <json> ::= <elt>
 <elt> ::= <object>
 | <array>
 | <string>
 | <number>
 | `true` | `false ` | `null ` <string> ::= `"` <chars> `" ` | `"" ` <chars> ::= <char><chars> | <char> <object> ::= `{`<items>`} ` | `{}` <items> ::= <item>`,`<items > | <item> <item> ::= <string>`:`<elt> <array> ::= `[`<elts>`] ` | `[]` <elts> ::= <elt>`,`<elts > | <elt> <number> ::= <digits > <digits> ::= <digit><digits> | <digit> DDSET:
98. {"": []} {"7897A": []} {"klnm,.qer;dfs?P":[]} {"123KOUIJ!qR30578950":[]} 98 <json> ::= <elt>
 <elt> ::= <object>
 | <array>
 | <string>
 | <number>
 | `true` | `false ` | `null ` <string> ::= `"` <chars> `" ` | `"" ` <chars> ::= <char><chars> | <char> <object> ::= `{`<items>`} ` | `{}` <items> ::= <item>`,`<items > | <item> <item> ::= <string>`:`<elt> <array> ::= `[`<elts>`] ` | `[]` <elts> ::= <elt>`,`<elts > | <elt> <number> ::= <digits > <digits> ::= <digit><digits> | <digit> DDSET:
99. {"": []} {"": true} {"":[1,2,445,"x"]} {"":{"PQ":[true, false, 223,"a"]}} 99 <json> ::= <elt>
 <elt> ::= <object>
 | <array>
 | <string>
 | <number>
 | `true` | `false ` | `null ` <string> ::= `"` <chars> `" ` | `"" ` <chars> ::= <char><chars> | <char> <object> ::= `{`<items>`} ` | `{}` <items> ::= <item>`,`<items > | <item> <item> ::= <string>`:`<elt> <array> ::= `[`<elts>`] ` | `[]` <elts> ::= <elt>`,`<elts > | <elt> <number> ::= <digits > <digits> ::= <digit><digits> | <digit> DDSET:
100. {"": []} Abstraction {"": <elt>} Abstract Input 100 <json> ::= <elt>
 <elt> ::= <object>
 | <array>
 | <string>
 | <number>
 | `true` | `false ` | `null ` <string> ::= `"` <chars> `" ` | `"" ` <chars> ::= <char><chars> | <char> <object> ::= `{`<items>`} ` | `{}` <items> ::= <item>`,`<items > | <item> <item> ::= <string>`:`<elt> <array> ::= `[`<elts>`] ` | `[]` <elts> ::= <elt>`,`<elts > | <elt> <number> ::= <digits > <digits> ::= <digit><digits> | <digit> DDSET:
101. {"": <elt>} Abstract Input {"": []} Minimized Input 101 <json> ::= <elt>
 <elt> ::= <object>
 | <array>
 | <string>
 | <number>
 | `true` | `false ` | `null ` <string> ::= `"` <chars> `" ` | `"" ` <chars> ::= <char><chars> | <char> <object> ::= `{`<items>`} ` | `{}` <items> ::= <item>`,`<items > | <item> <item> ::= <string>`:`<elt> <array> ::= `[`<elts>`] ` | `[]` <elts> ::= <elt>`,`<elts > | <elt> <number> ::= <digits > <digits> ::= <digit><digits> | <digit>
102. Issue 386 from Rhino var A = class extends (class {}){}; Issue 2937 from Closure const [y,y] = []; var {baz:{} = baz => {}} = baz => {}; Issue 385 from Rhino {while ((l_0)){ if ((l_0)) {break;;var l_0; continue }0}} Issue 2842 from Closure <varModifier> <Identifier> = class extends (class {}){} var {<$Id1>:{} = <$Id1> => {}} <variableDeclaration>; const [<$Id1>,<$Id1>] = [] {while ((<$Id1>)){ if ((<$Id1>)) {break;;var <$Id1>; continue }0}} 102
103. {"": <elt>} Abstract Input {"": []} Minimized Input 103 <json> ::= <elt>
 <elt> ::= <object>
 | <array>
 | <string>
 | <number>
 | `true` | `false ` | `null ` <string> ::= `"` <chars> `" ` | `"" ` <chars> ::= <char><chars> | <char> <object> ::= `{`<items>`} ` | `{}` <items> ::= <item>`,`<items > | <item> <item> ::= <string>`:`<elt> <array> ::= `[`<elts>`] ` | `[]` <elts> ::= <elt>`,`<elts > | <elt> <number> ::= <digits > <digits> ::= <digit><digits> | <digit> <json E> ::= <elt E>
 <elt E> ::= <object E>
 | <array E > | <string E>
 | <number E>
 <object E> ::= `{`<items E>`}`
 <items E> ::= <item E> | <item E>`,`<items > | <item>`,`<items E>
 <item E> ::= <string>`:`<elt E > | <string E1>`:`<elt>
 <array E> ::= `[`<elts E>`]`
 <elts E> ::= <elt E > | <elt E>`,`<elts > | <elt>`,`<elts E > <string E1> ::= `"" ` <json> ::= <elt>
 <elt> ::= <object>
 | <array > | <string>
 | <number>
 | `true` | `false` | `null`
 <object> ::= `{`<items>`}` | `{}`
 <items> ::= <item> | <item>`,`<items>
 <item> ::= <string>`:`<elt>
 <array> ::= `[`<elts>`]` | `[]`
 <elts> ::= <elt> | <elt>`,`<elts>
 <string> ::= `"` <chars> `"` | `""`
 <chars> ::= <char><chars>
 <char> ::= [A-Za-z0-9]
 <number> ::= <digits>
 <digits> ::= <digit><digits> | <digit>
 <digit> ::= [0-9] <json E> where <item E> is "":<elt> Evocative Pattern Evocative Grammar
104. 104 {"": 100} {"": [343,{},44998]} [{"": {"xxy":44998, {"b":[1,2,3]}}},[],[]] {"_": {"ket":[], {"":[],"y",[[],[1,2,3,455,6]]}}} {".":[{3243435656:"xy,zzzpqiu"},[{"":[112]},{"d":[[]]},{}]]} [{"": [1,2,3,4]}] {"pqr": {"": [1,2,3,4]}, "abc":[]} [[1132],{"xx":[{6:"dafjli;y,zzzdfaiu"},[{"__":[1{}{}]},{"":[[444456]]},{}]]} generate(<json E>) ✘ ✘ ✘ ✘ ✘ ✘ ✘ ✘ <json E> ::= <elt E>
 <elt E> ::= <object E>
 | <array E > | <string E>
 | <number E>
 <object E> ::= `{`<items E>`}`
 <items E> ::= <item E> | <item E>`,`<items > | <item>`,`<items E>
 <item E> ::= <string>`:`<elt E > | <string E1>`:`<elt>
 <array E> ::= `[`<elts E>`]`
 <elts E> ::= <elt E > | <elt E>`,`<elts > | <elt>`,`<elts E > <string E1> ::= `"" ` <json> ::= <elt>
 <elt> ::= <object>
 | <array > | <string>
 | <number>
 | `true` | `false` | `null`
 <object> ::= `{`<items>`}` | `{}`
 <items> ::= <item> | <item>`,`<items>
 <item> ::= <string>`:`<elt>
 <array> ::= `[`<elts>`]` | `[]`
 <elts> ::= <elt> | <elt>`,`<elts>
 <string> ::= `"` <chars> `"` | `""`
 <chars> ::= <char><chars>
 <char> ::= [A-Za-z0-9]
 <number> ::= <digits>
 <digits> ::= <digit><digits> | <digit>
 <digit> ::= [0-9] <json E> where <item E> is "":<elt> Evocative Grammar
105. 105 <json E> where <item E> is "":<elt> 1. We can produce any and all instances of the failure inducing pattern . 2. We can recognize any input that contains the failure inducing pattern . 3. The grammar will reject any input that doesn't contain the failure inducing pattern. <json E> ::= <elt E>
 <elt E> ::= <object E>
 | <array E > | <string E>
 | <number E>
 <object E> ::= `{`<items E>`}`
 <items E> ::= <item E> | <item E>`,`<items > | <item>`,`<items E>
 <item E> ::= <string>`:`<elt E > | <string E1>`:`<elt>
 <array E> ::= `[`<elts E>`]`
 <elts E> ::= <elt E > | <elt E>`,`<elts > | <elt>`,`<elts E > <string E1> ::= `"" ` <json> ::= <elt>
 <elt> ::= <object>
 | <array > | <string>
 | <number>
 | `true` | `false` | `null`
 <object> ::= `{`<items>`}` | `{}`
 <items> ::= <item> | <item>`,`<items>
 <item> ::= <string>`:`<elt>
 <array> ::= `[`<elts>`]` | `[]`
 <elts> ::= <elt> | <elt>`,`<elts>
 <string> ::= `"` <chars> `"` | `""`
 <chars> ::= <char><chars>
 <char> ::= [A-Za-z0-9]
 <number> ::= <digits>
 <digits> ::= <digit><digits> | <digit>
 <digit> ::= [0-9] Evocative Grammar
106. <json E> where <item E> is "":<elt> if json.has_key("") : raise Exception() if json.has_key_value(null) : raise Exception() <json N> where <item N> is <string>: null 106 Combining Evocative Patterns
107. if json.has_key("") and json.has_key_value(null) : raise Exception() <json E & N> where <item E> is "":<elt> 
 <item N> is <string>: null 107 Combining Evocative Patterns
108. if json.has_key("") and not json.has_key_value(null) : raise Exception() <json E & not(N)> where <item E> is "":<elt> <item N> is <string>: null 108 Combining Evocative Patterns
109. if json.has_key("") : raise Exception( ) if json.has_key_value(null) : raise Exception() <json not(E) & not(N)> where <item E> is "":<elt> <item N> is <string>: null 109 Combining Evocative Patterns
110. <json E&N> := <elt E&N > <elt E&N> := <object E&N > | <array E&N > <array E&N>:= '[' <elts E&N> '] ' <object E&N>:= '{' <items E&N> '} ' <elts E&N> := <elt E&N > | <elt E&N>','<elts N > | <elt N>','<elts E&N > <items E&N> := <item E&N > | <item E&N>','<items N > | <item N>','<items E&N > <item E&N> := <string E1>':'<elt N&N1 > | <string>':'<elt E&N&N1 > <elt E&N&N1> := <object E&N> <array E&N > <elt N> := 'false' | 'true ' | <number> | <string > | <object N> <array N > <array N> := '[]' | '[' <elts N> '] ' <object N> := '{}' | '{' <items N> '} ' <elts N> := <elt N > | <elt N>','<elts N > <items N> := <item N > | <item N>','<items N > <item N> := <string>':'<elt N&N1 > <elt N&N1> := 'false' | 'true ' | <number> | <string > | <object N> | <array N> <json E & N> where <item E> is "":<elt> <item N> is <string>: null {"": 100} {"": [343,{},44998]} [{"": {"xxy":44998, {"b":[1,2,3]}}},[],[]] {"_": {"ket":[], {"":[],"y",[[],[1,2,3,455,6]]}}} {".":[{3243435656:"xy,zzzpqiu"},[{"":[112]},{"d":[[]]},{}]]} [{"": [1,2,3,4]}] {"pqr": {"": [1,2,3,4]}, "abc":[]} [[1132],{"xx":[{6:"dafjli;y,zzzdfaiu"},[{"__":[1{}{}]},{"":[[444456]]},{}]]} generate(<json E&N>) ✘ ✘ ✘ ✘ ✘ ✘ ✘ ✘ 110
111. Issue 386 from Rhino var A = class extends (class {}){}; Issue 2937 from Closure const [y,y] = []; var {baz:{} = baz => {}} = baz => {}; Issue 385 from Rhino {while ((l_0)){ if ((l_0)) {break;;var l_0; continue }0}} Issue 2842 from Closure <varModifier> <Identifier> = class extends (class {}){} var {<$Id1>:{} = <$Id1> => {}} <variableDeclaration>; const [<$Id1>,<$Id1>] = [] {while ((<$Id1>)){ if ((<$Id1>)) {break;;var <$Id1>; continue }0}} 111
112. where <variableDeclarationList C2937> is <varModifier> <Identifier> = class extends (class {}){} <iterationStatement C2842> is {while ((<$Id1>)){ if ((<$Id1>)) {break;;var <$Id1>; continue }0}} <variableStatement R385> is var {<$Id2>:{} = <$Id2> => {}} <variableDeclaration>; <variableDeclarationList R386> is const [<$Id3>,<$Id3>] = [] Evocative Expressions <JavaScript C2937 and C2842> <JavaScript R385 and R386> <JavaScript (C2937 or C2842) and (R385 or R386)> <JavaScript not(C2937 or C2842 or R385 or R386)> 112
113. 113 Evocative Expressions: Validating Inputs where <variableDeclarationList C2937> is <varModifier> <Identifier> = class extends (class {}){} <iterationStatement C2842> is {while ((<$Id1>)){ if ((<$Id1>)) {break;;var <$Id1>; continue }0}} <variableStatement R385> is var {<$Id2>:{} = <$Id2> => {}} <variableDeclaration>; <variableDeclarationList R386> is const [<$Id3>,<$Id3>] = [] <JavaScript not(C2937 or C2842 or R385 or R386)>
114. Evocative Expressions for Focused Fuzzing def assign_admin_rights() : self.db_rights.add(MODIFY_DB ) self.fs_rights = R W self.timeout = 6 0 self.deploy = Tru e def assign_guest_rights() : self.db_rights = [QUERY_DB ] self.fs_rights = Non e self.timeout = Non e self.deploy = Fals e def modify_db(stmt) : if ADMIN in self.db_rights : process(stmt ) else : raise Error( ) def query_db(stmt) : process(stmt ) 114
115. 115 <json ADM> where <item ADM> is "role": "admin" def assign_admin_rights() : self.db_rights.add(MODIFY_DB ) self.fs_rights = R W self.timeout = 6 0 self.deploy = Tru e def assign_guest_rights() : self.db_rights = [QUERY_DB ] self.fs_rights = Non e self.timeout = Non e self.deploy = Fals e def modify_db(stmt) : if ADMIN in self.db_rights : process(stmt ) else : raise Error( ) def query_db(stmt) : process(stmt ) {"role" : "admin"} Evocative Expressions for Focused Fuzzing
116. 116 def assign_admin_rights() : self.db_rights.add(MODIFY_DB ) self.fs_rights = R W self.timeout = 6 0 self.deploy = Tru e def assign_guest_rights() : self.db_rights = [QUERY_DB ] self.fs_rights = Non e self.timeout = Non e self.deploy = Fals e def modify_db(stmt) : if ADMIN in self.db_rights : process(stmt ) else : raise Error( ) def query_db(stmt) : process(stmt ) {"method":"remove_table","args":["orders", "inventory"]} <json DBDT> where <items DBDT> is "method":"remove_table","args":<elt> Evocative Expressions for Focused Fuzzing
117. 117 def assign_admin_rights() : self.db_rights.add(MODIFY_DB ) self.fs_rights = R W self.timeout = 6 0 self.deploy = Tru e def assign_guest_rights() : self.db_rights = [QUERY_DB ] self.fs_rights = Non e self.timeout = Non e self.deploy = Fals e def modify_db(stmt) : if ADMIN in self.db_rights : process(stmt ) else : raise Error( ) def query_db(stmt) : process(stmt ) <json ADM & DBDT> where <item ADM> is "role": "admin" <object DBDT> is {"method":"remove_table","args":<elt>} {"method":"remove_table","args":["orders", "inventory"], "role":"admin"} {"role":"admin", "method":"remove_table","args":["orders", "inventory"]} {"method":"remove_table","role":"admin","args":["orders", "inventory"]} {"method":"remove_table","args":["orders","inventory",{"role":"admin"}]} Evocative Expressions for Focused Fuzzing
118. 118 def assign_admin_rights() : self.db_rights.add(MODIFY_DB ) self.fs_rights = R W self.timeout = 6 0 self.deploy = Tru e def assign_guest_rights() : self.db_rights = [QUERY_DB ] self.fs_rights = Non e self.timeout = Non e self.deploy = Fals e def modify_db(stmt) : if ADMIN in self.db_rights : process(stmt ) else : raise Error( ) def query_db(stmt) : process(stmt ) <json ADM & DBDT> where <item ADM> is "role": "admin" <object DBDT> is {"method":"remove_table","args":<elt>} {"method":"remove_table","args":["orders", "inventory"], "role":"guest"} {"role":"guest", "method":"remove_table","args":["orders", "inventory"]} {"method":"remove_table","role":"guest","args":["orders", "inventory"]} {"method":"remove_table","args":["orders","inventory",{"role":"guest"}]} Evocative Expressions for Focused Fuzzing
119. Algebraic Data Types 119 <my_struct> ::= <stype> <stype> struct mystruct { stype m1 ; stype m2 ; }; union myunion { utype m1 ; utype m2 ; }; <my_union> ::= <utype> | <utype> Data Structures Context Free Grammar Evocative Expressions: Data Structures
120. Search Source Code For Fault Patterns • Structure aware • Semantics can be mined from bugs! • No need to specify by hand • Combine different patterns using `or`, `and` and `not` 120 Evocative Expressions: Semantic Search
121. Gopinath, Nemati, Zeller. Input Algebras. ICSE 2021. Evocative Expressions https://rahul.gopinath.org/posts/ 121
122. if json.has_key("") : raise Exception() if not json.has_key_value(null) : raise Exception() 122 if json.has_key("") and not json.has_key_value(null) : raise Exception() Future: Decomposition with DDSet 122
123. 123 123
124. Generating Unbiased Samples 124
125. Finding Good Samples Seed corpus? (Blind spots) 125
126. 126 • Differentiate incomplete and incorrect input s • Solve one character at a time systematically Key Idea (Monotonic Failure Property) 126
127. 127 Sample Free Generators A ( 2 - B 9 ) 4 ) A ∉ (,+,-,1,2,3,4,5,6,7,8,9,0 B ∉ +,-,1,2,3,4,5,6,7,8,9,0,) ) ∉ +,-,1,2,3,4,5,6,7,8,9,0 (2-94) 127
128. 128 Mathis, Gopinath, Mera, Kampmann, Höschele, and Zeller. Parser Directed Fuzzing. PLDI 2019 . Mathis, Gopinath and Zeller Learning Input Tokens for Effective Fuzzing. ISSTA 2020. Sample Free Generators A ( 2 - B 9 ) 4 )
129. 129 Blackbox Sample Free Generators • Monotonic Failure Propert y • Differentiate incomplete and incorrect input s • Solve one character at a time systematically Bendrissou, Gopinath, Mathis, and Zeller Failure Feedback for Fast Fuzzing. (arXiv). 129
130. 130
131. 131 HTTP Parser XML Parser SOAP Parser RPC Parser C Parser Check Declarations Check Types Static Checks Compilers Servers Semantics Application Layered Programs 131
132. 132
133. 133 QS: Computer Science 22 in the world One in the Group of Eight (Go8) in Australia
134. 134
135. 135 Mining Evocative Expressions Blakbox Generation