Share
Embed
Mutation Analysis: Answering the Fuzzing Challenge
University of Melbourne 2022
Rahul Gopinath
November 03, 2024
Transcript
96 slides
-
8 Input ✓ ✘ Testing @app.route('/admin') def admin(): username =
request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 -
HTTP/1.1 401 Not Authorized Content-Type: application/json { "Error": "Assignment of
internal role 'superadmin' is forbidden" } HTTP/1.1 200 OK Content-type: application/json { "result": "OK: Updated user 'exampleUser' with role 'superadmin'" } 9 Input ✓ ✘ Testing POST /user/update HTTP/1.1 { "user": "exampleUser", "roles": [ "superadmin" ] } @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 HTTP/1.1 500 Internal Server Error ✘ -
10 Can We Trust Our Tests? Tests are still predominantly
written manually (Patrick Lam 2014) Developer Tester -
11 • Modern tests -- • Are complex and have
non-deterministic control flow • Interact: network, file system ... • No rigorous quality control • 50% of test code is cut and paste (Lam 2014) • 65% of test assertions are inadequate or wrong (Zhi 2013) • Once a test is written, it is rarely looked at unless it fails (Coplian 2014) Can We Trust Our Tests? -
12 Can We Trust Our Tests? How to evaluate the
quality of test suites? -
@app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return
{"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 13 Thou Shalt Cover Thy Code But is coverage su ffi cient? @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 -
14 Code Coverage is Useful • Statement coverage vs effectiveness
• Developer written or organic test suites Results from 250 programs from Github Effectiveness = 0.9 x S.Coverage R2 = 0.939 Effectiveness Statement Coverage Effectiveness computed with faults produced by PIT 2014 ICSE -
15 But Maybe Misleading • Statement coverage against effectiveness •
Machine generated test suites (using Randoop) Results from 250 programs from Github Effectiveness = 0.6 x S.Coverage R2 = 0.72 Effectiveness Statement Coverage Effectiveness computed with faults produced by PIT 2014 ICSE -
16 Coverage Maybe Misleading More complex coverage measures are easier
to attain for automated tools using signi fi cantly weaker assertions. Effectiveness Branch Coverage Effectiveness Effectiveness Effectiveness Branch Coverage Path Coverage Path Coverage Gopinath, Jensen, and Groce “Code Coverage for Suite Evaluation by Developers” 2014 ICSE 2014 ICSE -
17 Can We Trust Our Tests? HTTP/1.1 401 Not Authorized
Content-Type: application/json { "Error": "Assignment of internal role 'superadmin' is forbidden" } HTTP/1.1 200 OK Content-type: application/json { "result": "OK: Updated user 'exampleUser' with role 'superadmin'" } Input ✓ ✘ POST /user/update HTTP/1.1 { "user": "exampleUser", "roles": [ "superadmin" ] } @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 HTTP/1.1 500 Internal Server Error ✘ 17 Unless backed by test assertions, code coverage is pointless -
19 A Simple Idea: Fault Seeding • Effectiveness = M/N
(here 5/10 = 0.5) • Seed the program with N random faults (here 10) • Run the test suite against the faults • Say M faults were caught (here 5) Typically used in benchmarks -
20 Problems with Fault Seeding • Relies on manual generation/curation
of faults • Hand produced faults are different from real-world faults (Andrews 2005) -
21 Mutation Testing To The Rescue! The “gold standard” for
measuring test suite effectiveness. -
22 What Is Mutation Testing? d = b^3 - 4
* a * c d = b^2 + 4 * a * c d = b^2 - 4 + a * c Mutants d = b^2 - 4 * a * c Original (a = 0, b = 0, c = 0) => (d = 0) (a = 1, b = 1, c = 1) => (d = -3) (a = 0, b = 2, c = 0) => (d = 4) Mutants killed by test cases Test cases = b2 4ac -
23 Some Possible Mutants d = b^0 - 4 *
a * c; d = b^1 - 4 * a * c; d = b^-1 - 4 * a * c; d = b^MAX - 4 * a * c; d = b^MIN - 4 * a * c; d = b - 4 * a * c; d = b ^ 4 * a * c; d = b^2 - 0 * a * c; d = b^2 - 1 * a * c; d = b^2 – (-1) * a * c; d = b^2 - MAX * a * c; d = b^2 - MIN * a * c; d = b^2 - 4 * a * c; d = b^2 - 4 * a * c; d = b^2 + 4 * a * c; d = b^2 * 4 * a * c; d = b^2 / 4 * a * c; d = b^2 ^ 4 * a * c; d = b^2 % 4 * a * c; d = b^2 << 4 * a * c; d = b^2 >> 4 * a * c; d = b^2 * 4 + a * c; d = b^2 * 4 - a * c; d = b^2 * 4 / a * c; d = b^2 * 4 ^ a * c; d = b^2 * 4 % a * c; d = b^2 * 4 << a * c; d = b^2 * 4 >> a * c; d = b^2 * 4 * a + c; d = b^2 * 4 * a - c; d = b^2 * 4 * a / c; d = b^2 * 4 * a ^ c; d = b^2 * 4 * a % c; d = b^2 * 4 * a << c; d = b^2 * 4 * a >> c; d = b + 2 - 4 * a * c; d = b - 2 - 4 * a * c; d = b * 2 - 4 * a * c; d = b / 2 - 4 * a * c; d = b % 2 - 4 * a * c; d = b << 2 - 4 * a * c; d = b >> 2 - 4 * a * c; … = b2 4ac -
24 Why Does It Work? • The simple mutations are
not the entire set of faults • n mutations can produce 2^n complex faults • So why does mutation testing work? -
25 Assumptions of Mutation Testing The finite neighborhood assumption Programmers
make simple mistakes. d = b^2 + 4 * a * c d = b^2 - 4 * a * c = b2 4ac -
26 The coupling effect • Faults rarely interact • If
they interact, they become easier to detect (kill) d = b^2 + 4 * a + c a=1,b=1,c=1 => d = 0 a=0,b=0,c=0 => d = 0 a=1,b=1,c=1 => d = 0 a=0,b=0,c=0 => d = 0 d = b^2 - 4 * a + c d = b^2 + 4 * a * c = b2 4ac Assumptions of Mutation Testing -
27 Finite Neighborhood Bug fi x patches analyzed from: 1850
C 1128 Java 1000 Python 1393 Haskell open source projects from Github 2014 ISSRE -
29 Can Mutation Score Predict Future Bugs? Statements with detected
mutants twice less likely to contain future bugs (1-Mutation score) is a good proxy for residual defects Boxplot of bug fi xes on covered program elements with killed mutants vs elements with no killed mutants FSE 2016 -
32 Debian 5 ~ 70 million lines Smart cars ~
100 million lines Google is ~ 2 Billion lines A million lines zoomed (informationisbeautiful.net) ‘70s ‘80s ‘90s 2000 onwards Software Complexity Size of software systems has a nasty habit of doubling every few years -
HTTP/1.1 401 Not Authorized Content-Type: application/json { "Error": "Assignment of
internal role 'superadmin' is forbidden" } HTTP/1.1 200 OK Content-type: application/json { "result": "OK: Created user 'exampleUser' with role 'superadmin\ud888'" } 34 Input ✓ ✘ E ff ective Testing? POST /user/create HTTP/1.1 { "user": "exampleUser", "roles": [ "superadmin\ud888" ] } @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 HTTP/1.1 500 Internal Server Error ✘ -
35 E ff ective Testing? 24 = 16 2 Testing
cannot keep up when components (layers, libraries, services) multiply. -
36 Verify Behavior Input ✓ ✘ Automatic Testing @app.route('/admin') def
admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 -
37 (Oracle) Input ✓ ✘ Automatic Testing Oracles require domain
expertise @app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return {"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 Verify Behavior -
@app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return
{"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 38 Fuzzing Trash deck technique: 1950s - Gerald Weinberg Crash? -
@app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return
{"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 39 Fuzzing Crash? • Memory Bounds Violation • Privilege Escalation • Safety Violations • Metamorphic Relations • Differential Execution ASAN,MSAN,TSAN,NSan,FuZZan,UBSan -
@app.route('/admin') def admin(): username = request.cookies.get("username") if not username: return
{"Error": "Specify username in Cookie"} username = urllib.quote(os.path.basename(username)) url = "http://permissions:5000/permissions/{}".format(username) resp = requests.request(method="GET", url=url) # "superadmin\ud888" will be simpli fi ed to "superadmin" ret = ujson.loads(resp.text) if resp.status_code == 200: if "superadmin" in ret["roles"]: return {"OK": "Superadmin Access granted"} else: e = u"Access denied. User has following roles: {}".format(ret["roles"]) return {"Error": e}, 401 else:return {"Error": ret["Error"]}, 500 [ ; x 1 - G P Z + w c c k c ] ; , N 9 J + ? # 6 ^ 6 \ e ? ] 9 l u 2 _ % ' 4 G X " 0 V U B [ E / r ~ f A p u 6 b 8 < { % s i q 8 Z h . 6 { V , h r ? ; {Ti.r3PIxMMMv6{xS^+'Hq!AxB"YXRS@! Kd6;wtAMefFWM(`|J_<1~o}z3K(CCzRH J I I v H z > _ * . \ > J r l U 3 2 ~ e G P ? lR=bF3+;y$3lodQ<B89!5"W2fK*vE7v{')KC- i,c{<[~m!]o;{.'}Gj\(X}EtYetrpbY@aGZ1{P! A Z U 7 x # 4 ( R t n ! q 4 n C w q o l ^ y 6 } 0 | Ko=*JK~;zMKV=9Nai:wxu{J&UV#HaU)*Bi C < ) , ` + t * g k a < W = Z . % T 5 W G H Z p I 3 0 D < P q > & ] B S 6 R & j ? # t P 7 i a V } - } ` \ ? [ _ [ Z ^ L B M P G - FKj'\xwuZ1=Q`^`5,$N$Q@[!CuRzJ2D|vBy! ^ z k h d f 3 C 5 P A k R ? V ( ( - % > < h n | 3='i2Qx]D$qs4O`1@fevnG'2\11Vf3piU37@ 5 : d f d 4 5 * ( 7 ^ % 5 a p \ z I y l " ' f , $ee,J4Gw:cgNKLie3nx9(`efSlg6#[K"@Wjh Z}r[Scun&sBCS,T[/3]KAeEnQ7lU)3Pn,0)G/ 6N-wyzj/MTd#A;r Program 41 https://www.fuzzingbook.org/html/Fuzzer.html Traditional Fuzzing -
• Insert Instrumentation • Generate inputs • Collect execution feedback
• Branches covered during execution • Slightly Mutate Input and try again Collect inputs obtaining new coverage 43 Feedback Driven Fuzzing 43 https://www.fuzzingbook.org/html/MutationFuzzer.html -
44 def triangle(a, b, c): if a == b: if
b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene def triangle(a, b, c): __probe_enter() if a == b: __probe_1() if b == c: __probe_2() return Equilateral else: __probe_3() return Isosceles else: __probe_4() if b == c: __probe_5() return Isosceles else: __probe_6() if a == c: __probe_7() return Isosceles else: __probe_8() return Scalene def triangle(a, b, c): if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene • Insert Instrumentation • Generate inputs • Collect execution feedback • Branches covered during execution • Slightly Mutate Input and try again Collect inputs obtaining new coverage 44 https://www.fuzzingbook.org/html/MutationFuzzer.html Feedback Driven Fuzzing -
45 Feedback Driven Fuzzing triangle (1,1,1) def triangle(a, b, c):
if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene • Insert Instrumentation • Generate inputs • Collect execution feedback • Branches covered during execution • Slightly Mutate Input and try again Collect inputs obtaining new coverage 45 https://www.fuzzingbook.org/html/MutationFuzzer.html -
46 Feedback Driven Fuzzing triangle (1,1,1) def triangle(a, b, c):
if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene • Insert Instrumentation • Generate inputs • Collect execution feedback • Branches covered during execution • Slightly Mutate Input and try again Collect inputs obtaining new coverage 46 https://www.fuzzingbook.org/html/MutationFuzzer.html -
triangle (1,1,1) 47 Feedback Driven Fuzzing triangle (1,1,2) def triangle(a,
b, c): if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene • Insert Instrumentation • Generate inputs • Collect execution feedback • Branches covered during execution • Slightly Mutate Input and try again Collect inputs obtaining new coverage Mutated 47 https://www.fuzzingbook.org/html/MutationFuzzer.html -
48 Feedback Driven Fuzzing triangle (1,1,3) def triangle(a, b, c):
if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene • Insert Instrumentation • Generate inputs • Collect execution feedback • Branches covered during execution • Slightly Mutate Input and try again Collect inputs obtaining new coverage Mutated 48 https://www.fuzzingbook.org/html/MutationFuzzer.html -
49 Feedback Driven Fuzzing triangle (1,1,2) def triangle(a, b, c):
if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene • Insert Instrumentation • Generate inputs • Collect execution feedback • Branches covered during execution • Slightly Mutate Input and try again Collect inputs obtaining new coverage 49 https://www.fuzzingbook.org/html/MutationFuzzer.html triangle (1,1,1) triangle (1,1,3) -
• Insert Instrumentation • Generate inputs • Collect execution feedback
• Branches covered during execution • Slightly Mutate Input and try again Collect inputs obtaining new coverage 50 Feedback Driven Fuzzing triangle (1,1,2) def triangle(a, b, c): if a == b: if b == c: return Equilateral else: return Isosceles else: if b == c: return Isosceles else: if a == c: return Isosceles else: return Scalene triangle (1,1,1) AFL 50 -
51 Feedback Driven Fuzzing Weakness: static int is_reserved_word_token(const char *s,
int len) { const char *reserved[] = { "break", "case", "catch", "continue", "debugger", "default", "delete", "do", "else", "false", "finally", "for", "function", "if", "in", "instanceof", "new", "null", "return", "switch", "this", "throw", "true", "try", "typeof", "var", "void", "while", "with", "let", "undefined", ((void *)0)}; int i; if (!mjs_is_alpha(s[0])) return 0; for (i = 0; reserved[i] != ((void *)0); i++) { if (len == (int)strlen(reserved[i]) && strncmp(s, reserved[i], len) == 0) return i + 1; } return 0; } Tokens if (x > 100) { } coverage: 20% if (x > 100) { } e coverage: 5% if (x > 100) { } el coverage: 5% if (x > 100) { } els coverage: 5% if (x > 100) { } else coverage: 25% No smooth coverage gradient in parsers 51 -
52 Feedback Driven Fuzzing def json_raw(stm): while True: stm.skipspaces() c
= stm.peek() if c == 't': return json_fixed(stm, 'true') elif c == 'f': return json_fixed(stm, 'false') elif c == 'n': return json_fixed(stm, 'null') elif c == '"': return json_string(stm) elif c == '{': return json_dict(stm) elif c == '[': return json_list(stm) elif c in NUMSTART: return json_number(stm) raise JSONError(E_MALF, stm, stm.pos) Weak points: • Need for smooth coverage gradient • Coverage only provides first level guidance 1. {"abc":[]} 2. [{"a":[]}, {"b":[]}, {"c":["ab","c"]}] 52 -
53 Feedback Driven Fuzzing def json_raw(stm): while True: stm.skipspaces() c
= stm.peek() if c == 't': return json_fixed(stm, 'true') elif c == 'f': return json_fixed(stm, 'false') elif c == 'n': return json_fixed(stm, 'null') elif c == '"': return json_string(stm) elif c == '{': return json_dict(stm) elif c == '[': return json_list(stm) elif c in NUMSTART: return json_number(stm) raise JSONError(E_MALF, stm, stm.pos) Weak points: • Need for smooth coverage gradient • Coverage only provides first level guidance 1. {"abc":[]} 2. [{"a":[]}, {"b":[]}, {"c":["ab","c"]}] 53 -
54 Feedback Driven Fuzzing def json_raw(stm): while True: stm.skipspaces() c
= stm.peek() if c == 't': return json_fixed(stm, 'true') elif c == 'f': return json_fixed(stm, 'false') elif c == 'n': return json_fixed(stm, 'null') elif c == '"': return json_string(stm) elif c == '{': return json_dict(stm) elif c == '[': return json_list(stm) elif c in NUMSTART: return json_number(stm) raise JSONError(E_MALF, stm, stm.pos) Weak points: • Need for smooth coverage gradient • Coverage only provides first level guidance 1. {"abc":[]} 2. [{"a":[]}, {"b":[]}, {"c":["ab","c"]}] 54 -
55 def json_raw(stm): while True: stm.skipspaces() c = stm.peek() if
c == 't': return json_fixed(stm, 'true') elif c == 'f': return json_fixed(stm, 'false') elif c == 'n': return json_fixed(stm, 'null') elif c == '"': return json_string(stm) elif c == '{': return json_dict(stm) elif c == '[': return json_list(stm) elif c in NUMSTART: return json_number(stm) raise JSONError(E_MALF, stm, stm.pos) 55 { '<json>' : [['<elt>']], '<elt>' : [['<object>'], ['<array>'], ['<string>'], ['<number>'], ['true'], ['false'], ['null']], '<object>' : [['{', '<items>','}'], ['{}']], '<items>' : [['<item>,',',<items>'], ['<item>']], '<item>' : [['<string>',':', '<elt>']], '<array>' : [['[', '<elts>', ']'], ['[]']], '<elts>' : [['<elt>,',',<elts>'], ['<elt>']], '<string>' : [['"', '<chars>', '"'], ['""']], '<chars>' : [['<char>','<chars>'], ['<char>']], '<number>' : [['<digits>']], '<digits>' : [['<digit>','<digits>'], ['<digit>']], '<char>' : [[c] for c in string.characters] '<digit>' : [[c] for c in string.digits] } Solution: Structure Aware Fuzzing JSON Grammar -
56 { '<json>' : [['<elt>']], '<elt>' : [['<object>'], ['<array>'], ['<string>'],
['<number>'], ['true'], ['false'], ['null']], '<object>' : [['{', '<items>','}'], ['{}']], '<items>' : [['<item>,',',<items>'], ['<item>']], '<item>' : [['<string>',':', '<elt>']], '<array>' : [['[', '<elts>', ']'], ['[]']], '<elts>' : [['<elt>,',',<elts>'], ['<elt>']], '<string>' : [['"', '<chars>', '"'], ['""']], '<chars>' : [['<char>','<chars>'], ['<char>']], '<number>' : [['<digits>']], '<digits>' : [['<digit>','<digits>'], ['<digit>']], '<char>' : [[c] for c in string.characters] '<digit>' : [[c] for c in string.digits] } Structure Aware Feedback Driven Fuzzer -
{ '<json>' : [['<elt>']], '<elt>' : [['<object>'], ['<array>'], ['<string>'], ['<number>'],
['true'], ['false'], ['null']], '<object>' : [['{', '<items>','}'], ['{}']], '<items>' : [['<item>,',',<items>'], ['<item>']], '<item>' : [['<string>',':', '<elt>']], '<array>' : [['[', '<elts>', ']'], ['[]']], '<elts>' : [['<elt>,',',<elts>'], ['<elt>']], '<string>' : [['"', '<chars>', '"'], ['""']], '<chars>' : [['<char>','<chars>'], ['<char>']], '<number>' : [['<digits>']], '<digits>' : [['<digit>','<digits>'], ['<digit>']], '<char>' : [[c] for c in string.characters] '<digit>' : [[c] for c in string.digits] } 57 Grammar Fuzzer -
General Parser { '<json>' : [['<elt>']], '<elt>' : [['<object>'], ['<array>'],
['<string>'], ['<number>'], ['true'], ['false'], ['null']], '<object>' : [['{', '<items>','}'], ['{}']], '<items>' : [['<item>,',',<items>'], ['<item>']], '<item>' : [['<string>',':', '<elt>']], '<array>' : [['[', '<elts>', ']'], ['[]']], '<elts>' : [['<elt>,',',<elts>'], ['<elt>']], '<string>' : [['"', '<chars>', '"'], ['""']], '<chars>' : [['<char>','<chars>'], ['<char>']], '<number>' : [['<digits>']], '<digits>' : [['<digit>','<digits>'], ['<digit>']], '<char>' : [[c] for c in string.characters] '<digit>' : [[c] for c in string.digits] } 58 -
Lang Fuzzer { '<json>' : [['<elt>']], '<elt>' : [['<object>'], ['<array>'],
['<string>'], ['<number>'], ['true'], ['false'], ['null']], '<object>' : [['{', '<items>','}'], ['{}']], '<items>' : [['<item>,',',<items>'], ['<item>']], '<item>' : [['<string>',':', '<elt>']], '<array>' : [['[', '<elts>', ']'], ['[]']], '<elts>' : [['<elt>,',',<elts>'], ['<elt>']], '<string>' : [['"', '<chars>', '"'], ['""']], '<chars>' : [['<char>','<chars>'], ['<char>']], '<number>' : [['<digits>']], '<digits>' : [['<digit>','<digits>'], ['<digit>']], '<char>' : [[c] for c in string.characters] '<digit>' : [[c] for c in string.digits] } https://www.fuzzingbook.org/html/LangFuzzer.html 59 -
def process_input(input): try: ✔val = parse(input) res = process(val) return
res except SyntaxError: return Error { '<json>' : [['<elt>']], '<elt>' : [['<object>'], ['<array>'], ['<string>'], ['<number>'], ['true'], ['false'], ['null']], '<object>' : [['{', '<items>','}'], ['{}']], '<items>' : [['<item>,',',<items>'], ['<item>']], '<item>' : [['<string>',':', '<elt>']], '<array>' : [['[', '<elts>', ']'], ['[]']], '<elts>' : [['<elt>,',',<elts>'], ['<elt>']], '<string>' : [['"', '<chars>', '"'], ['""']], '<chars>' : [['<char>','<chars>'], ['<char>']], '<number>' : [['<digits>']], '<digits>' : [['<digit>','<digits>'], ['<digit>']], '<char>' : [[c] for c in string.characters] '<digit>' : [[c] for c in string.digits] } 60 -
64 "Be liberal in what you accept, and conservative in
what you send" Postel's Law 64 -
QUIRK_ALLOW_ASCII_CONTROL_CODES QUIRK_ALLOW_BACKSLASH_A QUIRK_ALLOW_BACKSLASH_CAPITAL_U QUIRK_ALLOW_BACKSLASH_E QUIRK_ALLOW_BACKSLASH_NEW_LINE QUIRK_ALLOW_BACKSLASH_QUESTION_MARK QUIRK_ALLOW_BACKSLASH_SINGLE_QUOTE QUIRK_ALLOW_BACKSLASH_V QUIRK_ALLOW_BACKSLASH_X_AS_BYTES QUIRK_ALLOW_BACKSLASH_X_AS_CODE_POINTS
QUIRK_ALLOW_BACKSLASH_ZERO QUIRK_ALLOW_COMMENT_BLOCK QUIRK_ALLOW_COMMENT_LINE QUIRK_ALLOW_EXTRA_COMMA QUIRK_ALLOW_INF_NAN_NUMBERS QUIRK_ALLOW_LEADING_ASCII_RECORD_SEPARATOR QUIRK_ALLOW_LEADING_UNICODE_BYTE_ORDER_MARK QUIRK_ALLOW_TRAILING_FILLER QUIRK_EXPECT_TRAILING_NEW_LINE_OR_EOF QUIRK_JSON_POINTER_ALLOW_TILDE_N_TILDE_R_TILDE_T QUIRK_REPLACE_INVALID_UNICODE JSON common quirks from https://github.com/google/wuffs 65 -
"Be liberal in what you accept, and conservative in what
you send" Postel's Law The Specification The Implementation Extra "Features" Where to Get the Grammar From? 66 -
67 Where to Get the Grammar From? Hand-written parsers already
encode the grammar 67 -
def json_raw(stm): while True: stm.skipspaces() c = stm.peek() if c
== 't': return json_fixed(stm, 'true') elif c == 'f': return json_fixed(stm, 'false') elif c == 'n': return json_fixed(stm, 'null') elif c == '"': return json_string(stm) elif c == '{': return json_dict(stm) elif c == '[': return json_list(stm) elif c in NUMSTART: return json_number(stm) raise JSONError(E_MALF, stm, stm.pos) <json_raw>::= <json_object> | <json_list> | <json_string> | <json_number> | <json_fixed(true)> | <json_fixed(false)> | <json_fixed(null)> <json_string> ::= `"` <chars> `"` | `""` <chars> ::= <char><chars> | <char> <json_object> ::= `{`<items>`}` | `{}` <items> ::= <item>`,`<items> | <item> <item> ::= <string>`:`<elt> <json_list> ::= `[`<elts>`]` | `[]` <elts> ::= <elt>`,`<elts> | <elt> <json_number> ::= <digits> <digits> ::= <digit><digits> | <digit> https://github.com/phensley/microjson MicroJSON 68 68 -
def json_raw(stm): while True: stm.skipspaces() c = stm.peek() if c
== 't': return json_fixed(stm, 'true') elif c == 'f': return json_fixed(stm, 'false') elif c == 'n': return json_fixed(stm, 'null') elif c == '"': return json_string(stm) elif c == '{': return json_dict(stm) elif c == '[': return json_list(stm) elif c in NUMSTART: return json_number(stm) raise JSONError(E_MALF, stm, stm.pos) <json_raw>::= <json_object> | <json_list> | <json_string> | <json_number> | <json_fixed(true)> | <json_fixed(false)> | <json_fixed(null)> <json_string> ::= `"` <chars> `"` | `""` <chars> ::= <char><chars> | <char> <json_object> ::= `{`<items>`}` | `{}` <items> ::= <item>`,`<items> | <item> <item> ::= <string>`:`<elt> <json_list> ::= `[`<elts>`]` | `[]` <elts> ::= <elt>`,`<elts> | <elt> <json_number> ::= <digits> <digits> ::= <digit><digits> | <digit> https://github.com/phensley/microjson MicroJSON 69 -
<F> := <A> | <B> <F> := <A> <B> <C>
<Fs> := <B> <Fs> | <empty> Structured Control Flow to Grammar Sequence A B C [F] Selection cond A B [F] F T Iteration cond B [F] 70 -
def json_raw(stm): while True: stm.skipspaces() c = stm.peek() if c
== 't': return json_fixed(stm, 'true') elif c == 'f': return json_fixed(stm, 'false') elif c == 'n': return json_fixed(stm, 'null') elif c == '"': return json_string(stm) elif c == '{': return json_object(stm) elif c == '[': return json_array(stm) elif c in NUMSTART: return json_number(stm) raise JSONError(E_MALF, stm, stm.pos) <json_raw> ::= <loop1> <loop1> ::= <iter1> <loop1> | <empty> <iter1>::= <json_fixed(true)> | <json_fixed(false)> | <json_fixed(null)> | `"` <json_string> | `{` <json_object> | `[` <json_array> | [[1-9] <json_number> 71 -
def json_raw(stm): while True: stm.skipspaces() c = stm.peek() if c
== 't': return json_fixed(stm, 'true') elif c == 'f': return json_fixed(stm, 'false') elif c == 'n': return json_fixed(stm, 'null') elif c == '"': return json_string(stm) elif c == '{': return json_object(stm) elif c == '[': return json_array(stm) elif c in NUMSTART: return json_number(stm) raise JSONError(E_MALF, stm, stm.pos) <json_raw> ::= <loop1> <loop1> ::= <iter1> <loop1> | <empty> <iter1>::= <json_fixed(true)> | <json_fixed(false)> | <json_fixed(null)> | `"` <json_string> | `{` <json_object> | `[` <json_array> | [[1-9] <json_number> 72 -
def json_string(stm): # skip over '"' stm.next() r = []
while True: c = stm.next() if c == '': raise JSONError(E_TRUNC) elif c == '\\': c = stm.next() r.append(decode_escape(c, stm)) elif c == '"': return ''.join(r) else: r.append(c) <json_raw> ::= <loop1> ... <iter1>::= `"` <json_string> ... <json_string> ::= <loop2> <loop2> ::= <iter2> <loop2> | <empty> <iter2>::= `\\` <decode_escape> | `"` | <other_char> "ab" <json_string> ::= <loop2> <loop2_end> <loop2> ::= <iter2> <loop2> | <empty> <iter2>::= `\\` <decode_escape> | <other_char> <loop2_end> ::= <iter2_end> <iter2_end>::=`"` 73 -
74 <START> ::= <json_raw> <json_raw> ::= '"' <json_string'> | '['
<json_list'> | '{' <json_dict'> | <json_number'> | 'true' | 'false' | 'null' <json_number'> ::= <json_number>+ | <json_number>+ 'e' <json_number>+ <json_number> ::= '+' | '-' | '.' | [0-9] | 'E' | 'e' <json_string'> ::= <json_string>* '"' <json_list'> ::= ']' | <json_raw> (','<json_raw>)* ']' | ( ',' <json_raw>)+ (',' <json_raw>)* ']' <json_dict'> ::= '}' | ( '"' <json_string'> ':' <json_raw> ',' )* '"'<json_string'> ':' <json_raw> '}' <json_string> ::= ' ' | '!' | '#' | '$' | '%' | '&' | ''' | '*' | '+' | '-' | ',' | '.' | '/' | ':' | ';' | '<' | '=' | '>' | '?' | '@' | '[' | ']' | '^' | '_', ''',| '{' | '|' | '}' | '~' | '[A-Za-z0-9]' | '\' <decode_escape> <decode_escape> ::= '"' | '/' | 'b' | 'f' | 'n' | 'r' | 't' stm.next() if expect_key: raise JSONError(E_DKEY, stm, stm.pos) if c == '}': return result expect_key = 1 continue # parse out a key/value pair elif c == '"': key = _from_json_string(stm) stm.skipspaces() c = stm.next() if c != ':': raise JSONError(E_COLON, stm, stm.pos) stm.skipspaces() val = _from_json_raw(stm) result[key] = val expect_key = 0 continue raise JSONError(E_MALF, stm, stm.pos) def _from_json_raw(stm): while True: stm.skipspaces() c = stm.peek() if c == '"': return _from_json_string(stm) elif c == '{': return _from_json_dict(stm) elif c == '[': return _from_json_list(stm) elif c == 't': return _from_json_fixed(stm, 'true', True, E_BOOL) elif c == 'f': return _from_json_fixed(stm, 'false', False, E_BOOL) elif c == 'n': return _from_json_fixed(stm, 'null', None, E_NULL) elif c in NUMSTART: return _from_json_number(stm) raise JSONError(E_MALF, stm, stm.pos) def from_json(data): stm = JSONStream(data) return _from_json_raw(stm) microjson.py Recovered JSON grammar 74 -
Recall Subjects Mimid calc.py 100.0% mathexpr.py 87.5% cgidecode.py 100.0% urlparse.py
100.0% microjson.py 98.7% parseclisp.py 99.3% jsonparser.c 100.0% tiny.c 100.0% mjs.c 95.4% Inputs generated by inferred grammar that were accepted by the program Subjects Mimid calc.py 100.0% mathexpr.py 92.7% cgidecode.py 100.0% urlparse.py 96.4% microjson.py 93.0% parseclisp.py 80.6% jsonparser.c 83.8% tiny.c 92.8% mjs.c 95.9% Inputs generated by golden grammar that were accepted by the inferred grammar parser Precision 75 75 Evaluation: Accuracy -
76 <START> ::= <json_raw> <json_raw> ::= '"' <json_string'> | '['
<json_list'> | '{' <json_dict'> | <json_number'> | 'true' | 'false' | 'null' <json_number'> ::= <json_number>+ | <json_number>+ 'e' <json_number>+ <json_number> ::= '+' | '-' | '.' | [0-9] | 'E' | 'e' <json_string'> ::= <json_string>* '"' <json_list'> ::= ']' | <json_raw> (','<json_raw>)* ']' | ( ',' <json_raw>)+ (',' <json_raw>)* ']' <json_dict'> ::= '}' | ( '"' <json_string'> ':' <json_raw> ',' )* '"'<json_string'> ':' <json_raw> '}' <json_string> ::= ' ' | '!' | '#' | '$' | '%' | '&' | ''' | '*' | '+' | '-' | ',' | '.' | '/' | ':' | ';' | '<' | '=' | '>' | '?' | '@' | '[' | ']' | '^' | '_', ''',| '{' | '|' | '}' | '~' | '[A-Za-z0-9]' | '\' <decode_escape> <decode_escape> ::= '"' | '/' | 'b' | 'f' | 'n' | 'r' | 't' stm.next() if expect_key: raise JSONError(E_DKEY, stm, stm.pos) if c == '}': return result expect_key = 1 continue # parse out a key/value pair elif c == '"': key = _from_json_string(stm) stm.skipspaces() c = stm.next() if c != ':': raise JSONError(E_COLON, stm, stm.pos) stm.skipspaces() val = _from_json_raw(stm) result[key] = val expect_key = 0 continue raise JSONError(E_MALF, stm, stm.pos) def _from_json_raw(stm): while True: stm.skipspaces() c = stm.peek() if c == '"': return _from_json_string(stm) elif c == '{': return _from_json_dict(stm) elif c == '[': return _from_json_list(stm) elif c == 't': return _from_json_fixed(stm, 'true', True, E_BOOL) elif c == 'f': return _from_json_fixed(stm, 'false', False, E_BOOL) elif c == 'n': return _from_json_fixed(stm, 'null', None, E_NULL) elif c in NUMSTART: return _from_json_number(stm) raise JSONError(E_MALF, stm, stm.pos) def from_json(data): stm = JSONStream(data) return _from_json_raw(stm) microjson.py Recovered JSON grammar 76 ESEC/FSE 2020. Mimid -
77 HTTP Parser XML Parser SOAP Parser RPC Parser C
Parser Check Declarations Check Types Static Checks Challenges Compilers Servers Semantics Application -
79 Which Fuzzer Should We Use? Exponential growth in fuzzing
literature Cumulative publications Publications per year -
82 Structural Coverage? Effectiveness Branch Coverage Effectiveness Effectiveness Effectiveness Branch
Coverage Path Coverage Path Coverage Insufficient -
84 Seeded Fault Benchmarks? • Easy to fine-tune a fuzzer
to overfit • Faults are rarely similar to real faults • Based on bugs we know about! • Human bias in bug curation • Limited supply • Bug interactions requiring deduplication -
85 Seeded Fault Benchmarks? • Easy to fine-tune a fuzzer
to overfit • Faults are rarely similar to real faults • Based on bugs we know about! • Human bias in bug curation • Limited supply Mutation Analysis? -
86 Mutation Analysis • Easy to fine-tune a fuzzer to
overfit • Faults are rarely similar to real faults • Based on bugs we know about! • Human bias in bug curation • Limited supply • Bug interactions requiring deduplication -
87 Mutation Analysis • Easy to fine-tune a fuzzer to
overfit • Faults are rarely similar to real faults • Based on bugs we know about! • Human bias in bug curation • Limited supply • Bug interactions requiring deduplication -
88 Mutation Analysis • Easy to fine-tune a fuzzer to
overfit • Faults are rarely similar to real faults • Based on bugs we know about! • Human bias in bug curation • Limited supply • Bug interactions requiring deduplication -
89 Mutation Analysis • Easy to fine-tune a fuzzer to
overfit • Faults are rarely similar to real faults • Based on bugs we know about! • Human bias in bug curation • Limited supply • Bug interactions requiring deduplication -
90 Mutation Analysis • Easy to fine-tune a fuzzer to
overfit • Faults are rarely similar to real faults • Based on bugs we know about! • Human bias in bug curation • Limited supply • Bug interactions requiring deduplication -
91 Mutation Analysis • Easy to fine-tune a fuzzer to
overfit • Faults are rarely similar to real faults • Based on bugs we know about! • Human bias in bug curation • Limited supply • Bug interactions requiring deduplication -
92 Mutation Analysis • Easy to fine-tune a fuzzer to
overfit • Faults are rarely similar to real faults • Based on bugs we know about! • Human bias in bug curation • Limited supply • Bug interactions requiring deduplication -
94 Mutation Analysis Is Costly Deterministically insert all simple faults,
and try to fi nd each. = b2 4ac d = b^0 - 4 * a * c; d = b^1 - 4 * a * c; d = b^-1 - 4 * a * c; d = b^MAX - 4 * a * c; d = b^MIN - 4 * a * c; d = b - 4 * a * c; d = b ^ 4 * a * c; d = b^2 - 0 * a * c; d = b^2 - 1 * a * c; d = b^2 – (-1) * a * c; d = b^2 - MAX * a * c; d = b^2 - MIN * a * c; d = b^2 - 4 * a * c; d = b^2 - 4 * a * c; d = b^2 + 4 * a * c; d = b^2 * 4 * a * c; d = b^2 / 4 * a * c; d = b^2 ^ 4 * a * c; d = b^2 % 4 * a * c; d = b^2 << 4 * a * c; d = b^2 >> 4 * a * c; d = b^2 * 4 + a * c; d = b^2 * 4 - a * c; d = b^2 * 4 / a * c; d = b^2 * 4 ^ a * c; d = b^2 * 4 % a * c; d = b^2 * 4 << a * c; d = b^2 * 4 >> a * c; d = b^2 * 4 * a + c; d = b^2 * 4 * a - c; d = b^2 * 4 * a / c; d = b^2 * 4 * a ^ c; d = b^2 * 4 * a % c; d = b^2 * 4 * a << c; d = b^2 * 4 * a >> c; d = b + 2 - 4 * a * c; d = b - 2 - 4 * a * c; d = b * 2 - 4 * a * c; d = b / 2 - 4 * a * c; d = b % 2 - 4 * a * c; d = b << 2 - 4 * a * c; d = b >> 2 - 4 * a * c; … and more … -
95 Mutation Analysis Is Costly M mutants Lines of code
Mutation points Lines of code Number of tests T tests Program size Effort for mutation testing = MxT test runs Test cases are no longer static -
96 Ongoing Research • Non interacting Faults with Higher Order
Mutants • Separate Coverage Analysis • Parallel Executions