Fuzzing -- From Alchemy to a Science

Fuzzing
Rahul Gopinath

View Slide

Fuzzing
from Alchemy
Rahul Gopinath

View Slide

Fuzzing
from Alchemy
to Science
Rahul Gopinath

View Slide

4

View Slide

4
The story begins 2500 years ago.
500 BC

View Slide

4
Vedic Sanskrit
500 BC

View Slide

4
Vedic Sanskrit Classical Sanskrit
500 BC

View Slide

5
500 BC

View Slide

5
500 BC
Aṣṭādhyāyī Dakṣiputra Pāṇini

View Slide

5
500 BC
Aṣṭādhyāyī Dakṣiputra Pāṇini
Ad hoc rules Formal specification

View Slide

6

View Slide

6
2500 years later....
2021 CE The world is governed by software

View Slide

6
2500 years later....
2021 CE
We have a crisis
The world is governed by software

View Slide

7
Debian 5 ~ 70 million lines
Smart cars ~ 100 million lines
Google is ~ 2 Billion lines
5 M
10 M
15 M
20 M
25 M
30 M
35 M
(Source: Wikipedia)
#Linux Kernel

Size in Millions

of Lines of Code
Linux Releases
Growth in software complexity

View Slide

7
Debian 5 ~ 70 million lines
Smart cars ~ 100 million lines
Google is ~ 2 Billion lines
1994
1996
1.0.0 2.0.0 2.1.0 2.2.0 2.4.0 2.6.0
2003
2011
3.0
2015
4.0
2019
5.0
2019
5.7
2001
5 M
10 M
15 M
20 M
25 M
30 M
35 M
(Source: Wikipedia)
#Linux Kernel

Size in Millions

of Lines of Code
Linux Releases
Growth in software complexity

View Slide

8
Growth in vulnerabilities
#Vulnerabilities
Year
16k
14k
12k
10k
8k
6k
4k
2k
(Source: NIST)

View Slide

8
Growth in vulnerabilities
#Vulnerabilities
Year
2019
2018
2017
2016
2015
2014
2013
2012
2011
2010
2009
2008
2007
2006
2005
2004
2003
2002
2001
16k
14k
12k
10k
8k
6k
4k
2k
(Source: NIST)

View Slide

Fuzzing
Program
Trash deck technique: 1950s - Gerald Weinberg

View Slide

Fuzzing
Crash?
Program
Trash deck technique: 1950s - Gerald Weinberg

View Slide

10
Fuzzing
Random Inputs
Program
Automatic Checks

View Slide

10
Fuzzing
• Memory Bounds Violation

• Privilege Escalation

• Safety Violations

• Metamorphic Relations

• Diﬀerential Execution
Random Inputs
Program
Automatic Checks

View Slide

11
Random Fuzzing
Program

View Slide

11
Random Fuzzing
$ ./fuzz
[;x1-GPZ+wcckc];,N9J+?#6^6\e?]9lu 
2_%'4GX"0VUB[E/r ~fApu6b8<{%siq8Z 
h.6{V,hr?;{Ti.r3PIxMMMv6{xS^+'Hq! 
AxB"[email protected]!Kd6;wtAMefFWM(`|J_<1~o} 
z3K(CCzRH JIIvHz>_*.\>JrlU32~eGP? 
lR=bF3+;y$3lodQ)KC-i,c{<[~m!]o;{.'}Gj\(X}EtYetrp 
[email protected]{P!AZU7x#4(Rtn!q4nCwqol^y6 
}0|Ko=*JK~;zMKV=9Nai:wxu{J&UV#HaU 
)*BiC<),`+t*gkaPq>&]BS6R&j?#tP7iaV}-}`\?[_[Z^LBM 
PG-FKj'\xwuZ1=Q`^`5,[email protected][!CuRzJ2 
D|vBy!^zkhdf3C5PAkR?V((-%>i2Qx]D$qs4O`[email protected]'2\[email protected] 
5:dfd45*(7^%5ap\zIyl"'f,$ee,J4Gw: 
cgNKLie3nx9(`efSlg6#[K"@WjhZ}r[Sc 
un&sBCS,T[/3]KAeEnQ7lU)3Pn,0)G/6N 
-wyzj/MTd#A;r*(ds./df3r8Odaf?/<#r
Program
✘

View Slide

$ ./fuzz -int | program
634111569742810193727424069509
741355925061499451162464719526
615957331924826555590537407605
181400079803446874252046374716
740973770255348279425601333144
152724057932073828569041216191
099859446496509919024810271242
622974988671421938012464630138
735355134599327240920259675263
574528613057084231370741920902
794677842164654990353575580453
777282305855352378119038096476
699871306655084953377039862387
924957554389878352934547664240
082431556093837288597262675598
630851919061829885048834738832
677022429414980917053939970795
722006987916088650168665471731 yes
12
Random Fuzzing
def is_prime(n: int) -> bool:
"""Primality test using 6k+-1 optimization."""
if n <= 3:
return n > 1
if n % 2 == 0 or n % 3 == 0:
return False
i = 5
while i ** 2 <= n:
if n % i == 0 or n % (i + 2) == 0:
return False
i += 6
return True
def main():
num = stdin.read()
print(num, is_prime(num))

View Slide

$ ./fuzz -int | program
634111569742810193727424069509
741355925061499451162464719526
615957331924826555590537407605
181400079803446874252046374716
740973770255348279425601333144
152724057932073828569041216191
099859446496509919024810271242
622974988671421938012464630138
735355134599327240920259675263
574528613057084231370741920902
794677842164654990353575580453
777282305855352378119038096476
699871306655084953377039862387
924957554389878352934547664240
082431556093837288597262675598
630851919061829885048834738832
677022429414980917053939970795
722006987916088650168665471731 yes
12
Random Fuzzing
def is_prime(n: int) -> bool:
"""Primality test using 6k+-1 optimization."""
if n <= 3:
return n > 1
if n % 2 == 0 or n % 3 == 0:
return False
i = 5
while i ** 2 <= n:
if n % i == 0 or n % (i + 2) == 0:
return False
i += 6
return True
def main():
num = stdin.read()
print(num, is_prime(num))
Weak point: unequal divisions in input space
Input space
n > 3
n <= 3

View Slide

13
Advanced Fuzzing: Instrumentation

View Slide

14
def triangle(a, b, c):
if a == b:
if b == c:
return Equilateral
else:
return Isosceles
else:
if b == c:
return Isosceles
else:
if a == c:
return Isosceles
else:
return Scalene
if a == b:
if b == c:
return Equilateral
else:
return Isosceles
else:
if b == c:
return Isosceles
else:
if a == c:
return Isosceles
else:
return Scalene

View Slide

14
__probe_enter()
if a == b:
__probe_1()
if b == c:
__probe_2()
return Equilateral
else:
__probe_3()
return Isosceles
else:
__probe_4()
if b == c:
__probe_5()
return Isosceles
else:
__probe_6()
if a == c:
__probe_7()
return Isosceles
else:
__probe_8()
return Scalene
if a == b:
if b == c:
return Equilateral
else:
return Isosceles
else:
if b == c:
return Isosceles
else:
if a == c:
return Isosceles
else:
return Scalene

View Slide

15
Instrumentation Based Fuzzing
if a == b:
if b == c:
return Equilateral
else:
return Isosceles
else:
if b == c:
return Isosceles
else:
if a == c:
return Isosceles
else:
return Scalene

View Slide

15
if a == b:
if b == c:
return Equilateral
else:
return Isosceles
else:
if b == c:
return Isosceles
else:
if a == c:
return Isosceles
else:
return Scalene
triangle (1,1,1)

View Slide

16
if a == b:
if b == c:
return Equilateral
else:
return Isosceles
else:
if b == c:
return Isosceles
else:
if a == c:
return Isosceles
else:
return Scalene

View Slide

16
if a == b:
if b == c:
return Equilateral
else:
return Isosceles
else:
if b == c:
return Isosceles
else:
if a == c:
return Isosceles
else:
return Scalene
triangle (1,1,2)

View Slide

17
Instrumentation Guided Fuzzing
• Coverage Guided
• Solver Directed

View Slide

18
Coverage Guided Fuzzing
• Randomly generate inputs
• Choose seeds with new coverage

View Slide

18
AFL

View Slide

19
AFL
First Release: 2013
AFL Trophy Case

View Slide

20
AFL
Pulling JPEGs out of thin air
https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-thin-air.html
Valid JPEG in 6 hours in an 8 core machine!

View Slide

21
AFL
static int is_reserved_word_token(const char *s, int len) {
const char *reserved[] = {
"break", "case", "catch", "continue", "debugger", "default",
"delete", "do", "else", "false", "finally", "for",
"function", "if", "in", "instanceof", "new", "null",
"return", "switch", "this", "throw", "true", "try",
"typeof", "var", "void", "while", "with", "let",
"undefined", ((void *)0)};
int i;
if (!mjs_is_alpha(s[0]))
return 0;
for (i = 0; reserved[i] != ((void *)0); i++) {
if (len == (int)strlen(reserved[i]) && strncmp(s, reserved[i], len) == 0)
return i + 1;
}
return 0;
}

View Slide

21
AFL
static int is_reserved_word_token(const char *s, int len) {
const char *reserved[] = {
"break", "case", "catch", "continue", "debugger", "default",
"delete", "do", "else", "false", "finally", "for",
"function", "if", "in", "instanceof", "new", "null",
"return", "switch", "this", "throw", "true", "try",
"typeof", "var", "void", "while", "with", "let",
"undefined", ((void *)0)};
int i;
if (!mjs_is_alpha(s[0]))
return 0;
for (i = 0; reserved[i] != ((void *)0); i++) {
if (len == (int)strlen(reserved[i]) && strncmp(s, reserved[i], len) == 0)
return i + 1;
}
return 0;
}
Weak point: Magic bytes

View Slide

22

View Slide

23
Solver Directed Fuzzing
• Collect path constraints
• Solve negated constraints for new inputs

View Slide

23
(a == b)
(b == c)

View Slide

23
(a == b)
(b == c)
(b != c)

View Slide

23
(a == b)
(b == c)
(b != c)
triangle(1,2,1)

View Slide

24
• Solve constraints for new inputs
void next_sym() {
while(1) {
switch (ch){
case '{': next_ch(); sym = LBRA; return;
case '}': next_ch(); sym = RBRA; return;
case '(': next_ch(); sym = LPAR; return;
case ')': next_ch(); sym = RPAR; return;
case '+': next_ch(); sym = PLUS; return;
case '-': next_ch(); sym = MINUS; return;
case '<': next_ch(); sym = LESS; return;
case ';': next_ch(); sym = SEMI; return;
case '=': next_ch(); sym = EQUAL; return;
default:
if (ch >= '0' && ch <= '9') {
int_val = 0; /* missing overflow check */
while (ch >= '0' && ch <= '9') {
int_val = int_val*10 + (ch - '0'); next_ch(); }
sym = INT;
} else if (ch >= 'a' && ch <= 'z') {
int i = 0; /* missing overflow check */
while ((ch >= 'a' && ch <= 'z') || ch == '_'){
id_name[i++] = ch; next_ch(); }
id_name[i] = '\0';
sym = 0;
while (words[sym] != NULL && strcmp(words[sym], id_name) != 0)
sym++;
if (words[sym] == NULL)
if (id_name[1] == '\0') sym = ID; else syntax_error();
}
else syntax_error();
return;
}

View Slide

24
• Solve constraints for new inputs
void next_sym() {
while(1) {
switch (ch){
case '{': next_ch(); sym = LBRA; return;
case '}': next_ch(); sym = RBRA; return;
case '(': next_ch(); sym = LPAR; return;
case ')': next_ch(); sym = RPAR; return;
case '+': next_ch(); sym = PLUS; return;
case '-': next_ch(); sym = MINUS; return;
case '<': next_ch(); sym = LESS; return;
case ';': next_ch(); sym = SEMI; return;
case '=': next_ch(); sym = EQUAL; return;
default:
if (ch >= '0' && ch <= '9') {
int_val = 0; /* missing overflow check */
while (ch >= '0' && ch <= '9') {
int_val = int_val*10 + (ch - '0'); next_ch(); }
sym = INT;
} else if (ch >= 'a' && ch <= 'z') {
int i = 0; /* missing overflow check */
while ((ch >= 'a' && ch <= 'z') || ch == '_'){
id_name[i++] = ch; next_ch(); }
id_name[i] = '\0';
sym = 0;
while (words[sym] != NULL && strcmp(words[sym], id_name) != 0)
sym++;
if (words[sym] == NULL)
if (id_name[1] == '\0') sym = ID; else syntax_error();
}
else syntax_error();
return;
}
Weak point: Path explosion

View Slide

25

View Slide

26
Fuzzing Parsers
$ ./fuzz
Interpreter

View Slide

26
Fuzzing Parsers
$ ./fuzz
Parser
Syntax Error
Interpreter
#

View Slide

27
Advanced Fuzzing: Specialized Generators
• Specialize generation for a domain

View Slide

27
Advanced Fuzzing: Specialized Generators
• Specialize generation for a domain
80,000 lines of code

View Slide

28
The Holy Grail of Fuzzing

View Slide

28
The Holy Grail of Fuzzing
Parsers

View Slide

Overcoming Parsers

View Slide

30
Overcoming Parsers

View Slide

30
Monolithic
Overcoming Parsers

View Slide

31

View Slide

32
The Missing Piece: Formal Specification

View Slide

33
Grammar

View Slide

34
Formal Languages
Formal Language Descriptions

View Slide

34
Formal Languages
3. Regular
(Chomsky,1956)

View Slide

34
Formal Languages
3. Regular
Context Free
(Chomsky,1956)
Argument Stack

View Slide

34
Formal Languages
3. Regular
Context Free
Recursively Enumerable
(Chomsky,1956)
Argument Stack
Return Stack

View Slide

34
Formal Languages
3. Regular
Context Free
Recursively Enumerable
(Chomsky,1956)
Easy to produce and parse
Argument Stack
Return Stack

View Slide

35
Grammar
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
Arithmetic expression grammar

View Slide

35
Grammar
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
key

View Slide

35
Grammar
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
Definition for
key

View Slide

36
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
Grammar

View Slide

36
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
Grammar
Expansion Rule

View Slide

36
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
Grammar
Expansion Rule Terminal Symbol

View Slide

36
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
Grammar
Expansion Rule Terminal Symbol
Nonterminal Symbol

View Slide

37
Grammars
As recognizers
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]

View Slide

37
Grammars
As recognizers
(8 / 3) * 49
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]

View Slide

38
Grammars
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
As producers (Hanford 1970)

(Purdom 1972)

View Slide

38
Grammars8.2 - 27 - -9 / +((+9 * --2 + --+-+-
((-1 * +(8 - 5 - 6)) * (-(a-+(((+(4)
)))) - ++4) / +(-+---((5.6 - --(3 *
-1.8 * +(6 * +-(((-(-6) * ---+6)) /
+--(+-+-7 * (-0 * (+(((((2)) + 8 - 3
- ++9.0 + ---(--+7 / (1 / +++6.37)
+ (1) / 482) / +++-+0)))) + 8.2 - 27
- -9 / +((+9 * --2 + --+-+-((-1 * +
(8 - 5 - 6)) * (-(a-+(((+(4))))) - +
+4) / +(-+---((5.6 - --(3 * -1.8 * +
(6 * +-(((-(-6) * ---+6)) / +--(+-+-
7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0
+ ---(--+7 / (1 / +++6.37) + (1) /
482) / +++-+0)))) * -+5 + 7.513))))
- (+1 / ++((-84)))))))) * ++5 / +-(-
-2 - -++-9.0)))) / 5 * --++090 + * -
+5 + 7.513)))) - (+1 / ++((-84))))))
)) * 8.2 - 27 - -9 / +((+9 * --2 + -
-+-+-((-1 * +(8 - 5 - 6)) * (-(a-+((
(+(4))))) - ++4) / +(-+---((5.6 - --
(3 * -1.8 * +(6 * +-(((-(-6) * ---+6
)) / +--(+-+-7 * (-0 * (+(((((2)) +
8 - 3 - ++9.0 + ---(--+7 / (1 / +++6
.37) + (1) / 482) / +++-+0)))) * -+5
+ 7.513)))) - (+1 / ++((-84))))))))
* ++5 / +-(--2 - -++-9.0)))) / 5 *
--++090 ++5 / +-(--2 - -++-9.0)))) /
5 * --++090
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
As producers (Hanford 1970)

(Purdom 1972)

View Slide

39
Grammars
As effective producers
8.2 - 27 - -9 / +((+9 * --2 + --+-+-
((-1 * +(8 - 5 - 6)) * (-(a-+(((+(4)
)))) - ++4) / +(-+---((5.6 - --(3 *
-1.8 * +(6 * +-(((-(-6) * ---+6)) /
+--(+-+-7 * (-0 * (+(((((2)) + 8 - 3
- ++9.0 + ---(--+7 / (1 / +++6.37)
+ (1) / 482) / +++-+0)))) + 8.2 - 27
- -9 / +((+9 * --2 + --+-+-((-1 * +
(8 - 5 - 6)) * (-(a-+(((+(4))))) - +
+4) / +(-+---((5.6 - --(3 * -1.8 * +
(6 * +-(((-(-6) * ---+6)) / +--(+-+-
7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0
+ ---(--+7 / (1 / +++6.37) + (1) /
482) / +++-+0)))) * -+5 + 7.513))))
- (+1 / ++((-84)))))))) * ++5 / +-(-
-2 - -++-9.0)))) / 5 * --++090 + * -
+5 + 7.513)))) - (+1 / ++((-84))))))
)) * 8.2 - 27 - -9 / +((+9 * --2 + -
-+-+-((-1 * +(8 - 5 - 6)) * (-(a-+((
(+(4))))) - ++4) / +(-+---((5.6 - --
(3 * -1.8 * +(6 * +-(((-(-6) * ---+6
)) / +--(+-+-7 * (-0 * (+(((((2)) +
8 - 3 - ++9.0 + ---(--+7 / (1 / +++6
.37) + (1) / 482) / +++-+0)))) * -+5
+ 7.513)))) - (+1 / ++((-84))))))))
* ++5 / +-(--2 - -++-9.0)))) / 5 *
--++090 ++5 / +-(--2 - -++-9.0)))) /
5 * --++090

View Slide

39
Grammars
As effective producers
Interpreter
Parser
✘
✔
8.2 - 27 - -9 / +((+9 * --2 + --+-+-
((-1 * +(8 - 5 - 6)) * (-(a-+(((+(4)
)))) - ++4) / +(-+---((5.6 - --(3 *
-1.8 * +(6 * +-(((-(-6) * ---+6)) /
+--(+-+-7 * (-0 * (+(((((2)) + 8 - 3
- ++9.0 + ---(--+7 / (1 / +++6.37)
+ (1) / 482) / +++-+0)))) + 8.2 - 27
- -9 / +((+9 * --2 + --+-+-((-1 * +
(8 - 5 - 6)) * (-(a-+(((+(4))))) - +
+4) / +(-+---((5.6 - --(3 * -1.8 * +
(6 * +-(((-(-6) * ---+6)) / +--(+-+-
7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0
+ ---(--+7 / (1 / +++6.37) + (1) /
482) / +++-+0)))) * -+5 + 7.513))))
- (+1 / ++((-84)))))))) * ++5 / +-(-
-2 - -++-9.0)))) / 5 * --++090 + * -
+5 + 7.513)))) - (+1 / ++((-84))))))
)) * 8.2 - 27 - -9 / +((+9 * --2 + -
-+-+-((-1 * +(8 - 5 - 6)) * (-(a-+((
(+(4))))) - ++4) / +(-+---((5.6 - --
(3 * -1.8 * +(6 * +-(((-(-6) * ---+6
)) / +--(+-+-7 * (-0 * (+(((((2)) +
8 - 3 - ++9.0 + ---(--+7 / (1 / +++6
.37) + (1) / 482) / +++-+0)))) * -+5
+ 7.513)))) - (+1 / ++((-84))))))))
* ++5 / +-(--2 - -++-9.0)))) / 5 *
--++090 ++5 / +-(--2 - -++-9.0)))) /
5 * --++090

View Slide

40
Where to Get the Grammar From?

View Slide

41
•Reference Specification?

View Slide

41
The standard spec

View Slide

41
The standard spec
Buggy Implementation

View Slide

41
The standard spec
"Extra" Features

View Slide

41
The standard spec
"Extra" Features
"Be liberal in what you accept, and conservative in what you send" 
(the cause of trouble) Postel's Law
"Accepted" Bugs

View Slide

42
https://www.json.org

View Slide

42
object
{ }
{ members }
members
pair
pair , members
pair
string : value
array
[ ]
[ elements ]
elements
value
value , elements
value
string
number
object
array
true
false
null
string
" "
" chars "
chars
char
char chars
char
UNICODE \ [",\,CTRL]
\" \\ \/ \b \f \n \r \t
\u hex hex hex hex
number
int
int frac
int exp
int frac exp
int
digit
onenine digits
- digit
- onenine digits
frac
. digits
exp
e digits
hex
digit
A - F
a - f
digits
digit
digit digits
e
e e+ e-
E E+ E-
https://www.json.org

View Slide

43
Parsing JSON is a Minefield
http://seriot.ch/

View Slide

43
Parsing JSON is a Minefield
http://seriot.ch/
Expected
Parse Fail (Expect Success)
Parse Success (Expect Fail)
Parse Success (Undefined)
Parse Fail (Undefined)
Parser Crash
Timeout

View Slide

View Slide

45

View Slide

45
Hand-written parsers already encode the grammar

View Slide

46
How to Extract This Grammar?

View Slide

46
• Inputs -> Dynamic Control Dependence Trees

View Slide

46
• Inputs -> Dynamic Control Dependence Trees
• DCD Trees -> Context Free Grammar

View Slide

47
Control Dependence Graph
Statement B is control dependent on A if A determines whether B executes.
def parse_csv(s,i):
while s[i:]:
if is_digit(s[i]):
n,j = num(s[i:])
i = i+j
else:
comma(s[i])
i += 1

View Slide

47
def parse_csv(s,i):
while s[i:]:
if is_digit(s[i]):
n,j = num(s[i:])
i = i+j
else:
comma(s[i])
i += 1
CDG for parse_csv

View Slide

47
def parse_csv(s,i):
while s[i:]:
if is_digit(s[i]):
n,j = num(s[i:])
i = i+j
else:
comma(s[i])
i += 1
CDG for parse_csv
while: determines

whether

if: executes

View Slide

48
def parse_csv(s,i):
while s[i:]:
if is_digit(s[i]):
n,j = num(s[i:])
i = i+j
else:
comma(s[i])
i += 1
CDG for parse_csv
Dynamic Control Dependence Tree
Each statement execution is represented as a separate node

View Slide

48
def parse_csv(s,i):
while s[i:]:
if is_digit(s[i]):
n,j = num(s[i:])
i = i+j
else:
comma(s[i])
i += 1
CDG for parse_csv
Dynamic Control Dependence Tree
Each statement execution is represented as a separate node
DCD Tree for call parse_csv()

View Slide

49
def parse_csv(s,i):
while s[i:]:
if is_digit(s[i]):
n,j = num(s[i:])
i = i+j
else:
comma(s[i])
i += 1
DCD Tree ~ Parse Tree
•No tracking beyond input buﬀer

•Characters are attached to nodes where they are accessed last
"12,"
"12,"

View Slide

49
def parse_csv(s,i):
while s[i:]:
if is_digit(s[i]):
n,j = num(s[i:])
i = i+j
else:
comma(s[i])
i += 1
'1' '2' ','
DCD Tree ~ Parse Tree
•No tracking beyond input buﬀer

•Characters are attached to nodes where they are accessed last
"12,"
"12,"

View Slide

50
def is_digit(i): return i in '0123456789'
def parse_num(s,i):
n = ''
while s[i:] and is_digit(s[i]):
n += s[i]
i = i +1
return i,n
def parse_paren(s, i):
assert s[i] == '('
i, v = parse_expr(s, i+1)
if s[i:] == '': raise Ex(s, i)
assert s[i] == ')'
return i+1, v
def parse_expr(s, i = 0):
expr, is_op = [], True
while s[i:]:
c = s[i]
if isdigit(c):
if not is_op: raise Ex(s,i)
i,num = parse_num(s,i)
expr.append(num)
is_op = False
elif c in ['+', '-', '*', '/']:
if is_op: raise Ex(s,i)
expr.append(c)
is_op, i = True, i + 1
elif c == '(':
i, cexpr = parse_paren(s, i)
expr.append(cexpr)
is_op = False
elif c == ')': break
else: raise Ex(s,i)
return i, expr
9+3/4
Parse tree for parse_expr('9+3/4')

View Slide

51
def parse_num(s,i):
n = ''
n += s[i]
i = i +1
return i,n
assert s[i] == '('
assert s[i] == ')'
return i+1, v
while s[i:]:
c = s[i]
if isdigit(c):
expr.append(num)
is_op = False
elif c in ['+', '-', '*', '/']:
expr.append(c)
elif c == '(':
expr.append(cexpr)
is_op = False
else: raise Ex(s,i)
return i, expr
9+3/4
Identifying Compatible Nodes
Which nodes correspond to the same nonterminal

View Slide

52
3 * (9 + 1)

View Slide

52
(9 + 1) * 3
3 * (9 + 1)

View Slide

53
3 * (9 + 1)

View Slide

53
9 + 1
3 * (9 + 1)

View Slide

54
3 * (9 + 1)

View Slide

54
3 (9 + 1) *
3 * (9 + 1)

View Slide

55
3*(1)
1

View Slide

56
3*(1)
1

View Slide

56
3*(1)
1
:=

View Slide

56
3*(1)
1
:=

:=

View Slide

:=
|
|
|
:=
:=
|
:=
:= '3' | '1'
:= '(' ')'
:=
:= '*'
57

View Slide

:=
:=
|
:=
|
|
|
:=
:=
|
:=
:= '3' | '1'
:= '(' ')'
:=
:= '*'
57

View Slide

58
def parse_num(s,i):
n = ''
n += s[i]
i = i +1
return i,n
assert s[i] == '('
assert s[i] == ')'
return i+1, v
while s[i:]:
c = s[i]
if isdigit(c):
expr.append(num)
is_op = False
elif c in ['+', '-', '*', '/']:
expr.append(c)
elif c == '(':
expr.append(cexpr)
is_op = False
else: raise Ex(s,i)
return i, expr
:=
:=
|
:=
|
:= '(' ')'
|
:= '*' | '+' | '-' | '/'
:=
|
: [0-9]
calc.py Recovered Arithmetic Grammar

View Slide

59
:=
:=
|
:=
|
:= '(' ')'
|
:= '*' | '+' | '-' | '/'
:=
|
: [0-9]

View Slide

59
8.2 - 27 - -9 / +((+9 * --2 + --+-+-
((-1 * +(8 - 5 - 6)) * (-(a-+(((+(4)
)))) - ++4) / +(-+---((5.6 - --(3 *
-1.8 * +(6 * +-(((-(-6) * ---+6)) /
+--(+-+-7 * (-0 * (+(((((2)) + 8 - 3
- ++9.0 + ---(--+7 / (1 / +++6.37)
+ (1) / 482) / +++-+0)))) + 8.2 - 27
- -9 / +((+9 * --2 + --+-+-((-1 * +
(8 - 5 - 6)) * (-(a-+(((+(4))))) - +
+4) / +(-+---((5.6 - --(3 * -1.8 * +
(6 * +-(((-(-6) * ---+6)) / +--(+-+-
7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0
+ ---(--+7 / (1 / +++6.37) + (1) /
482) / +++-+0)))) * -+5 + 7.513))))
- (+1 / ++((-84)))))))) * ++5 / +-(-
-2 - -++-9.0)))) / 5 * --++090 + * -
+5 + 7.513)))) - (+1 / ++((-84))))))
)) * 8.2 - 27 - -9 / +((+9 * --2 + -
-+-+-((-1 * +(8 - 5 - 6)) * (-(a-+((
(+(4))))) - ++4) / +(-+---((5.6 - --
(3 * -1.8 * +(6 * +-(((-(-6) * ---+6
)) / +--(+-+-7 * (-0 * (+(((((2)) +
8 - 3 - ++9.0 + ---(--+7 / (1 / +++6
.37) + (1) / 482) / +++-+0)))) * -+5
+ 7.513)))) - (+1 / ++((-84))))))))
* ++5 / +-(--2 - -++-9.0)))) / 5 *
--++090 ++5 / +-(--2 - -++-9.0)))) /
5 * --++090
:=
:=
|
:=
|
:= '(' ')'
|
:= '*' | '+' | '-' | '/'
:=
|
: [0-9]

View Slide

60
::=
::= '"'
| '['
| '{'
|
| 'true'
| 'false'
| 'null'
::= +
| + 'e' +
::= '+' | '-' | '.' | [0-9] | 'E' | 'e'
::= * '"'
::= ']'
| (',')* ']'
| ( ',' )+ (',' )* ']'
::= '}'
| ( '"' ':' ',' )*
'"' ':' '}'
::= ' ' | '!' | '#' | '$' | '%' | '&' | '''
| '*' | '+' | '-' | ',' | '.' | '/' | ':' | ';'
| '<' | '=' | '>' | '?' | '@' | '[' | ']' | '^'
| '_', ''',| '{' | '|' | '}' | '~'
| '[A-Za-z0-9]'
| '\'
::= '"' | '/' | 'b' | 'f' | 'n' | 'r' | 't'
stm.next()
if expect_key:
raise JSONError(E_DKEY, stm, stm.pos)
if c == '}':
return result
expect_key = 1
continue
# parse out a key/value pair
elif c == '"':
key = _from_json_string(stm)
stm.skipspaces()
c = stm.next()
if c != ':':
raise JSONError(E_COLON, stm, stm.pos)
stm.skipspaces()
val = _from_json_raw(stm)
result[key] = val
expect_key = 0
continue
raise JSONError(E_MALF, stm, stm.pos)
def _from_json_raw(stm):
while True:
stm.skipspaces()
c = stm.peek()
if c == '"':
return _from_json_string(stm)
elif c == '{':
return _from_json_dict(stm)
elif c == '[':
return _from_json_list(stm)
elif c == 't':
return _from_json_fixed(stm, 'true', True, E_BOOL)
elif c == 'f':
return _from_json_fixed(stm, 'false', False, E_BOOL)
elif c == 'n':
return _from_json_fixed(stm, 'null', None, E_NULL)
elif c in NUMSTART:
return _from_json_number(stm)
def from_json(data):
stm = JSONStream(data)
return _from_json_raw(stm)
microjson.py Recovered JSON grammar

View Slide

61
::=
::= '"'
| '['
| '{'
|
| 'true'
| 'false'
| 'null'
::= +
| + 'e' +
::= '+' | '-' | '.' | [0-9] | 'E' | 'e'
::= * '"'
::= ']'
| (',')* ']'
| ( ',' )+ (',' )* ']'
::= '}'
| ( '"' ':' ',' )*
'"' ':' '}'
::= ' ' | '!' | '#' | '$' | '%' | '&' | '''
| '*' | '+' | '-' | ',' | '.' | '/' | ':' | ';'
| '<' | '=' | '>' | '?' | '@' | '[' | ']' | '^'
| '_', ''',| '{' | '|' | '}' | '~'
| '[A-Za-z0-9]'
| '\'
::= '"' | '/' | 'b' | 'f' | 'n' | 'r' | 't'
stm.next()
if expect_key:
raise JSONError(E_DKEY, stm, stm.pos)
if c == '}':
return result
expect_key = 1
continue
# parse out a key/value pair
elif c == '"':
key = _from_json_string(stm)
stm.skipspaces()
c = stm.next()
if c != ':':
raise JSONError(E_COLON, stm, stm.pos)
stm.skipspaces()
val = _from_json_raw(stm)
result[key] = val
expect_key = 0
continue
def _from_json_raw(stm):
while True:
stm.skipspaces()
c = stm.peek()
if c == '"':
return _from_json_string(stm)
elif c == '{':
return _from_json_dict(stm)
elif c == '[':
return _from_json_list(stm)
elif c == 't':
return _from_json_fixed(stm, 'true', True, E_BOOL)
elif c == 'f':
return _from_json_fixed(stm, 'false', False, E_BOOL)
elif c == 'n':
return _from_json_fixed(stm, 'null', None, E_NULL)
elif c in NUMSTART:
return _from_json_number(stm)
def from_json(data):
stm = JSONStream(data)
return _from_json_raw(stm)
microjson.py Recovered JSON grammar
Mimid
Gopinath, Mathis, and Zeller. Mining Input Grammars from Dynamic Control Flow. ESEC/FSE 2020.
•Javascript
•C
•Lisp
•JSON
•URL
•CGI

View Slide

Actual Specification

View Slide

Dynamic Approximation
(Mimid)

View Slide

(Mimid)
Static Approximation
(Static Mimid)

View Slide

(Mimid)
Static Approximation
(Static Mimid)
IOT
&
Embedded

View Slide

63

View Slide

Generating Unbiased Samples

View Slide

Finding Good Samples
Seed corpus?

View Slide

Finding Good Samples
Seed corpus?
(Blind spots)

View Slide

• Differentiate incomplete and incorrect inputs
• Solve one input symbol at a time systematically
Key Idea

View Slide

67
Sample Free Generators

View Slide

67
A

View Slide

67
A
A ∉ (,+,-,1,2,3,4,5,6,7,8,9,0

View Slide

67
A
(
A ∉ (,+,-,1,2,3,4,5,6,7,8,9,0

View Slide

67
A
( 2
A ∉ (,+,-,1,2,3,4,5,6,7,8,9,0

View Slide

67
A
( 2
-
B
9
)
4 )
A ∉ (,+,-,1,2,3,4,5,6,7,8,9,0
B ∉ +,-,1,2,3,4,5,6,7,8,9,0,)
) ∉ +,-,1,2,3,4,5,6,7,8,9,0

View Slide

67
A
( 2
-
B
9
)
4 )
A ∉ (,+,-,1,2,3,4,5,6,7,8,9,0
B ∉ +,-,1,2,3,4,5,6,7,8,9,0,)
) ∉ +,-,1,2,3,4,5,6,7,8,9,0
(2-94)

View Slide

Mathis, Gopinath, Mera, Kampmann, Höschele, and Zeller. Parser Directed Fuzzing. PLDI 2019.
Mathis, Gopinath and Zeller Learning Input Tokens for Effective Fuzzing. ISSTA 2020.
Gopinath, Bendrissou, Mathis, and Zeller Black-box Testing with Monotonic Preﬁxes. ISSRE 2021 (submitted).
A
( 2
-
B
9
)
4 )

View Slide

:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
Fast Fuzzing with Grammars
fuzz(expr_grammar, '')

View Slide

:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
def fuzz(grammar, key):
return collapse_tree(gen_key(grammar, key))

View Slide

:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
def gen_key(grammar, key): 
if is_terminal_symbol(key): 
return key
else:
next_rule = random.choice(rules)
return gen_rule(grammar, grammar[key][next_rule])

View Slide

:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
return key
else:
def gen_rule(grammar, rule):
return [gen_key(token) for token in rule]

View Slide

return key
else:

View Slide


+

1 8
return key
else:

View Slide


+

1 8
def collapse(tree):
key, children = tree
if not children: return tree
return ''.join([collapse(c) for c in children])
return key
else:

View Slide


+

1 8
1 8
+
"1 + 8"
def collapse(tree):
key, children = tree
if not children: return tree
return ''.join([collapse(c) for c in children])
return key
else:

View Slide

:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
Compiling the Grammar

View Slide

:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
def start():
expr()
def expr():
match (random() % 6):
case 0: expr(); print('+'); expr()
case 1: expr(); print('-'); expr()
case 2: expr(); print('/'); expr()
case 3: expr(); print('*'); expr()
case 4: print('('); expr(); print(')')
case 5: number()
def number():
case 0: integer()
case 1: integer(); print('.'); integer()
def integer():
case 0: digit(); integer()
case 1: digit()
def digit():
case 0: print('0')
case 1: print('1')
case 2: print('2')
case 3: print('3')
case 4: print('4')
case 5: print('5')
case 6: print('6')
case 7: print('7')

View Slide

:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
def start():
expr()
def expr():
case 5: number()
def number():
case 0: integer()
def integer():
case 1: digit()
def digit():
case 0: print('0')
case 1: print('1')
case 2: print('2')
case 3: print('3')
case 4: print('4')
case 5: print('5')
case 6: print('6')
case 7: print('7')
def start(rops):
expr(rops)
def expr(rops):
match (rops.next % 6):
case 0: expr(rops); print('+'); e
case 1: expr(rops); print('-'); e
case 2: expr(rops); print('/'); e
case 3: expr(rops); print('*'); e
case 4: print('('); expr(rops); p
case 5: number(rops)
def number(rops):
case 0: integer(rops)
case 1: integer(rops); print('.')
def integer(rops):
case 0: digit(rops); integer(rops
case 1: digit(rops)
def digit(rops):
case 0: print('0')
case 1: print('1')
case 2: print('2')
case 3: print('3')
case 4: print('4')
case 5: print('5')
case 6: print('6')
case 7: print('7')

View Slide

Grammar Fuzzer
grammar = """ 
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9] 
"""
generate(grammar)
Fast Fuzzers
def start():
expr()
def expr():
case 5: number()
def number():
case 0: integer()
def integer():
case 1: digit()
def digit():
case 0: print('0')
case 1: print('1')
case 2: print('2')
case 3: print('3')
case 4: print('4')
case 5: print('5')
case 6: print('6')
case 7: print('7')
Compiled Grammar (F1)
Building Fast Fuzzers Gopinath and Zeller 2019
def start_0(rops):
r = next(rops)
if 0 <= r < 43: expr_0()
elif 43 <= r < 85: expr_1()
elif 85 <= r < 128: expr_2()
elif 128 <= r < 171: expr_3()
elif 171 <= r < 213: expr_4()
else: expr_5()
def expr_0(rops):
r = next(rops)
if 0 <= r < 43: expr_0()
elif 43 <= r < 85: expr_1()
elif 85 <= r < 128: expr_2()
elif 128 <= r < 171: expr_3()
elif 171 <= r < 213: expr_4()
else: expr_5()
print('+')
r = next(rops)
if 0 <= r < 43: expr_0()
elif 43 <= r < 85: expr_1()
elif 85 <= r < 128: expr_2()
elif 128 <= r < 171: expr_3()
elif 171 <= r < 213: expr_4()
else: expr_5()
Grammar VM (F1)

View Slide

73
The Fuzzing Pipeline
Program Under Test

View Slide

73
The Fuzzing Pipeline
Program Under Test
pFuzzer
Active
Guidance
F1 Grammar Fuzzer
Grammar
Inputs
Grammar Miner
Samples
Active
Learning

View Slide

74
The Fuzzing Synergy
Mimid Grammar Miner
Parser Directed pFuzzer
F1 Fuzzer VM
Mathis, Gopinath, Mera, Kampmann, Höschele, and Zeller. Mining Input Grammars from Dynamic Control Flow. PLDI 2019.
Mathis, Gopinath and Zeller Learning Input Tokens for Effective Fuzzing. ISSTA 2020.
Gopinath, Bendrissou, Mathis, and Andreas Zeller Black-box Testing with Monotonic Preﬁxes. ISSTA 2021 (submitted).
Gopinath and Zeller Building Fast Fuzzers 2019 (unpublished)
Gopinath, Mathis, and Zeller Mining Input Grammars with Dynamic Control Flow. FSE 2020.

View Slide

View Slide

76
Challenge: Multilevel Envelopes
POST /InStock HTTP/1.1
Host: www.stock.org
Content-Type: application/soap+xml; charset=utf-8
Content-Length: 312

xmlns:soap="http://www.w3.org/2001/12/soap-envelope"
soap:encodingStyle="http://w3.org/2001/12/soap-encoding">

IBM

View Slide

76
Host: www.stock.org
Content-Length: 312


IBM

HTTP POST

View Slide

76
Host: www.stock.org
Content-Length: 312


IBM

HTTP POST
XML PAYLOAD

View Slide

76
Host: www.stock.org
Content-Length: 312


IBM

HTTP POST
XML PAYLOAD
SOAP

View Slide

76
Host: www.stock.org
Content-Length: 312


IBM

HTTP POST
XML PAYLOAD
SOAP
RPC Call

View Slide

77
Future Challenge: Multilevel Envelopes

View Slide

77
Future Challenge: Multilevel Envelopes
Mimid

View Slide

78
#include
int main() {
int number1, number2, number3;
number1 = 10;
number2 = 20;
number3 = sum(number1, number2);
if (number3 > 100) return 0;
return 1;
}
$ cc example.c -o example
example.c
1.Syntactically correct
2.Variables declared before use
3.Use correct types
4.Statically conforming
5.Dynamically conforming
6.Model conforming
Challenge: Semantic Envelopes
(Mckeeman 1998)
(parse)
(compile)
(link)
(run)
(synthesis)

View Slide

79
#include
int main() {
int number1, number2, number3;
number1 = 10;
number2 = 20;
number3 = sum(number1, number2);
if (number3 > 100) return 0;
return 1;
}
$ cc example.c -o example
example.c
1.Syntactically correct
2.Variables declared before use
3.Use correct types
4.Statically conforming
5.Dynamically conforming
6.Model conforming
Challenge: Semantic Envelopes
(Mckeeman 1998)
(parse)
(compile)
(link)
(run)
(synthesis)

View Slide

80

View Slide

81
Fuzzing
Inputs
Program
Behavior

View Slide

81
Fuzzing
Inputs
Program
Behavior
✔

View Slide

82

View Slide

83
We Found A Crash

View Slide

Why Did My Program Crash?

View Slide

8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 -
5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.
6 - --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +-
-(+-+-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(
--+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) +
8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 -
5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.6
- --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +--
(+-+-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(-
-+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) * -
+5 + 7.513)))) - (+1 / ++((-84)))))))) * ++5 / +-
(--2 - -++-9.0)))) / 5 * --++090 + * -+5 + 7.513)
))) - (+1 / ++((-84)))))))) * 8.2 - 27 - -9 / +((
+9 * --2 + --+-+-((-1 * +(8 - 5 - 6)) * (-(a-+
(((+(4))))) - ++4) / +(-+---((5.6 - --(3 * -1.8 *
+(6 * +-(((-(-6) * ---+6)) / +--(+-+-7 * (-0 * (
+(((((2)) + 8 - 3 - ++9.0 + ---(--+7 / (1 / +++6.
37) + (1) / 482) / +++-+0)))) * -+5 + 7.513)))) -
(+1 / ++((-84)))))))) * ++5 / +-(--2 - -++-9.0)))
) / 5 * --++090 ++5 / +-(--2 - -++-9.0)))) / 5 *
--++090

View Slide

8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 -
5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.
6 - --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +-
-(+-+-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(
--+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) +
8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 -
5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.6
- --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +--
(+-+-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(-
-+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) * -
+5 + 7.513)))) - (+1 / ++((-84)))))))) * ++5 / +-
(--2 - -++-9.0)))) / 5 * --++090 + * -+5 + 7.513)
))) - (+1 / ++((-84)))))))) * 8.2 - 27 - -9 / +((
+9 * --2 + --+-+-((-1 * +(8 - 5 - 6)) * (-(a-+
(((+(4))))) - ++4) / +(-+---((5.6 - --(3 * -1.8 *
+(6 * +-(((-(-6) * ---+6)) / +--(+-+-7 * (-0 * (
+(((((2)) + 8 - 3 - ++9.0 + ---(--+7 / (1 / +++6.
37) + (1) / 482) / +++-+0)))) * -+5 + 7.513)))) -
(+1 / ++((-84)))))))) * ++5 / +-(--2 - -++-9.0)))
) / 5 * --++090 ++5 / +-(--2 - -++-9.0)))) / 5 *
--++090
DD Minimized Input
((4))

View Slide

8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 -
5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.
6 - --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +-
-(+-+-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(
--+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) +
8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 -
5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.6
- --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +--
(+-+-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(-
-+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) * -
+5 + 7.513)))) - (+1 / ++((-84)))))))) * ++5 / +-
(--2 - -++-9.0)))) / 5 * --++090 + * -+5 + 7.513)
))) - (+1 / ++((-84)))))))) * 8.2 - 27 - -9 / +((
+9 * --2 + --+-+-((-1 * +(8 - 5 - 6)) * (-(a-+
(((+(4))))) - ++4) / +(-+---((5.6 - --(3 * -1.8 *
+(6 * +-(((-(-6) * ---+6)) / +--(+-+-7 * (-0 * (
+(((((2)) + 8 - 3 - ++9.0 + ---(--+7 / (1 / +++6.
37) + (1) / 482) / +++-+0)))) * -+5 + 7.513)))) -
(+1 / ++((-84)))))))) * ++5 / +-(--2 - -++-9.0)))
) / 5 * --++090 ++5 / +-(--2 - -++-9.0)))) / 5 *
--++090
DD Minimized Input
((4))
00000 ?

View Slide

8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 -
5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.
6 - --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +-
-(+-+-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(
--+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) +
8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 -
5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.6
- --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +--
(+-+-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(-
-+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) * -
+5 + 7.513)))) - (+1 / ++((-84)))))))) * ++5 / +-
(--2 - -++-9.0)))) / 5 * --++090 + * -+5 + 7.513)
))) - (+1 / ++((-84)))))))) * 8.2 - 27 - -9 / +((
+9 * --2 + --+-+-((-1 * +(8 - 5 - 6)) * (-(a-+
(((+(4))))) - ++4) / +(-+---((5.6 - --(3 * -1.8 *
+(6 * +-(((-(-6) * ---+6)) / +--(+-+-7 * (-0 * (
+(((((2)) + 8 - 3 - ++9.0 + ---(--+7 / (1 / +++6.
37) + (1) / 482) / +++-+0)))) * -+5 + 7.513)))) -
(+1 / ++((-84)))))))) * ++5 / +-(--2 - -++-9.0)))
) / 5 * --++090 ++5 / +-(--2 - -++-9.0)))) / 5 *
--++090
DD Minimized Input
((4))
00000 ?
((5)) ?

View Slide

8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 -
5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.
6 - --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +-
-(+-+-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(
--+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) +
8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 -
5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.6
- --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +--
(+-+-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(-
-+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) * -
+5 + 7.513)))) - (+1 / ++((-84)))))))) * ++5 / +-
(--2 - -++-9.0)))) / 5 * --++090 + * -+5 + 7.513)
))) - (+1 / ++((-84)))))))) * 8.2 - 27 - -9 / +((
+9 * --2 + --+-+-((-1 * +(8 - 5 - 6)) * (-(a-+
(((+(4))))) - ++4) / +(-+---((5.6 - --(3 * -1.8 *
+(6 * +-(((-(-6) * ---+6)) / +--(+-+-7 * (-0 * (
+(((((2)) + 8 - 3 - ++9.0 + ---(--+7 / (1 / +++6.
37) + (1) / 482) / +++-+0)))) * -+5 + 7.513)))) -
(+1 / ++((-84)))))))) * ++5 / +-(--2 - -++-9.0)))
) / 5 * --++090 ++5 / +-(--2 - -++-9.0)))) / 5 *
--++090
DD Minimized Input
((4))
00000 ?
((5)) ?
(++5) ?

View Slide

85
Issue 386 from Rhino
var A = class extends (class {}){};
Issue 2937 from Closure
const [y,y] = [];
var {baz:{} = baz => {}} = baz => {};
{while ((l_0)){ if ((l_0)) {break;;var l_0; continue }0}}

View Slide

85
const [y,y] = [];
var {baz:{} = baz => {}} = baz => {};
Delta Minimization is useful but not suﬃcient

View Slide

( ( 4 ) )

View Slide

( ( 4 ) )
:=
:= ' + '
| ' - '
|
:= ' * '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '.'
|
:=
|
:= [0-9]

View Slide

( ( 4 ) )

View Slide

( ( 4 ) )
:=
:= ' + '
| ' - '
|
:= ' * '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '.'
|
:=
|
:= [0-9]

View Slide

( ( 4 ) )
:=
:= ' + '
| ' - '
|
:= ' * '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '.'
|
:=
|
:= [0-9]
✓ Did not reproduce the failure
1 * (2 - 3)

View Slide

( ( 4 ) )
:=
:= ' + '
| ' - '
|
:= ' * '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '.'
|
:=
|
:= [0-9]

View Slide

( ( 4 ) )
:=
:= ' + '
| ' - '
|
:= ' * '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '.'
|
:=
|
:= [0-9]
c

View Slide

( ( 4 ) )
:=
:= ' + '
| ' - '
|
:= ' * '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '.'
|
:=
|
:= [0-9]
c
1 + 3 + 4

View Slide

( ( 4 ) )
:=
:= ' + '
| ' - '
|
:= ' * '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '.'
|
:=
|
:= [0-9]
c
c

View Slide

3 * 4
:=
:= ' + '
| ' - '
|
:= ' * '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '.'
|
:=
|
:= [0-9]
c
c

View Slide

( ( 4 ) )
:=
:= ' + '
| ' - '
|
:= ' * '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '.'
|
:=
|
:= [0-9]
c
c
c
c
c
c
c

View Slide

( ( 1 - 2 ) )
:=
:= ' + '
| ' - '
|
:= ' * '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '.'
|
:=
|
:= [0-9]
c
c
c
c
c
c
c
( ( 1 - 2 ) )

View Slide

( ( 1 - 2 ) )
:=
:= ' + '
| ' - '
|
:= ' * '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '.'
|
:=
|
:= [0-9]
c
c
c
c
c
c
c
✘ reproduced the failure
( ( 1 - 2 ) )

View Slide

( ( 1 - 2 ) )
c
c
c
c
c
c
c
( ( 1 - 2 ) )

View Slide

( ( 1 - 2 ) )
c
c
c
c
c
c
c
✘
( ( 1 - 2 ) )

View Slide

( ( 1 - 2 ) )
c
c
c
c
c
c
c
✘
( ( 1 - 2 ) )
( ( 2 * 3 + 4 ) )

View Slide

( ( 1 - 2 ) )
c
c
c
c
c
c
c
✘
( ( 1 - 2 ) )
✘
( ( 2 * 3 + 4 ) )

View Slide

( ( 1 - 2 ) )
c
c
c
c
c
c
c
✘
( ( 1 - 2 ) )
✘
( ( 2 * 3 + 4 ) )
( ( - 2 / 1 ) )

View Slide

( ( 1 - 2 ) )
c
c
c
c
c
c
c
✘
( ( 1 - 2 ) )
✘
( ( 2 * 3 + 4 ) )
✘
( ( - 2 / 1 ) )

View Slide

( ( 1 - 2 ) )
c
c
c
c
c
c
c
✘
( ( 1 - 2 ) )
✘
( ( 2 * 3 + 4 ) )
✘
( ( - 2 / 1 ) )
( ( 98 - 0 ) )

View Slide

( ( 1 - 2 ) )
c
c
c
c
c
c
c
✘
( ( 1 - 2 ) )
✘
( ( 2 * 3 + 4 ) )
✘
( ( - 2 / 1 ) )
✘
( ( 98 - 0 ) )

View Slide

)
(
( )
( ( )
4 )
( ( 4 ) )
c
c
c
c
c
c
c
A

View Slide

( ( 4 ) )
c
c
c
c
c
c
c
A
( ( ) )

( ( ) )
4
Minimized Input
Abstract Failure Inducing Input
def check(parsed):
if parsed.is_nested() and parsed.child.is_nested():
raise Exception()
return input

View Slide


View Slide

= class extends (class {}){}

View Slide

var {baz:{} = baz => {}} = baz => {};

View Slide

var {baz:{} = baz => {}} = baz => {};
var {<$Id1>:{} = <$Id1> => {}} ;

View Slide

const [y,y] = [];

View Slide

const [y,y] = [];
const [<$Id1>,<$Id1>] = []

View Slide


View Slide

{while ((<$Id1>)){ if ((<$Id1>)) {break;;var <$Id1>; continue }0}}

View Slide

( ( 4 ) )
c
c
c
c
c
c
c
A
( ( ) )

( ( ) )
4
Minimized Input
Abstract Failure Inducing Input
• Eﬀectively abstracts a minimized input

• The abstraction identiﬁes where the problem lies

• Decompose complex program behaviors
DDSET
Gopinath, Kampmann, Havrikov, Soremekun, and Zeller. Abstracting Failure Inducing Inputs. ISSTA 2020.
def check(parsed):
if parsed.is_nested() and parsed.child.is_nested():
raise Exception()
return input
ISSTA 2020 Distinguished Award

View Slide

108
:=
:= ' + '
| ' + '
| ' - '
| ' - '
|
:= ' * '
| ' * '
| ' / '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '(' ')'
:=
:=
:= '(' ')'
Specialized Grammar
is (())

View Slide

108
:=
:= ' + '
| ' + '
| ' - '
| ' - '
|
:= ' * '
| ' * '
| ' / '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '(' ')'
:=
:=
:= '(' ')'
((1)) + 2
(23 * ((3)) - 34)
(344- 4 + ((223)))
(1) - 3 * 773 + (-22 + 1)
1798 - 889 / ((333-1)) * 2 / 3 + 1
34 + ((4)) -334 + (334 - (22) + 919 * 0 + 1
98435747+ 88 + (((0))) + (1) - 1 * 7 / 4 * 889 - 2
8 + ((8)) + --1 + 11223 / 344 - 39 + (1) - 456 + 134 / 45
437 + 8 - 1 * ((9 + ((1))) - 1 + 99111948 + 3 --1 + (112) - 2 + 445) + 0
74 + 334 + ((178 - 88 / (3393-1) * 1002 / 3 + 1+ 3439)) * 223 - 1233 + 334672
2 * ((9)) - (1798 - 889 / (333-1) * 2 / 3 + 100012 + 3434392 + 234 ----6 * 1798 - 889 / (33
778 - (((1) - 3 * 773 + (-22 + 1) * (4545) - 23 - ((2)) * 773 + (-22 + 1) / 3434 + ---1 + 1 / 34343 + 112
349 + (((1) - 3 * 3 + (-22 + 1) ((+ (-22 + 1) * (4545) - 23 - (2) * 773 + ((-22 + 1)) / 3434 + ---1 + 1 / 34343 + 1123
8 + ((8)) + --1 + / 1 - 39 + (1) - 456 + 134 / 45 ))(((1) - 2334 + ((((1)) - 3 * 773 + (-22 + 1) * (2) - 23 - (2) * 773 + (-22 + 1) / 3
74 + 3 + ((178 - 88 / (3393-1) * 1002 / 3 + 1+ 3439)) * - 1233 + 334672)) ((8 + ((8)) + --1 + / 344 - 39 + (1) - 456 + 134 / 45 ))(((1) - 3 * 773
1+ 33+ 24343433 +23343 - ((74 + 334 + ((178 - 88 / (3393-1) * 1002 / 3 + 1+ 3439)) * - 1233 + 334672)) ((8 + ((8)) + --1 + / 344 - 39 + (1) - 456 + 134 / 4
✘
Specialized Grammar
is (())
✘

View Slide

Algebra of Grammar Specializations

View Slide

is = class extends (class {}){} Closure 2937
is var {<$Id2>:{} = <$Id2> => {}} ; Rhino 385
is const [<$Id3>,<$Id3>] = [] Rhino 386
is {while ((<$Id1>)){ if ((<$Id1>)) {break;;var <$Id1>; continue }0}} Closure 2842
where

View Slide

Gopinath, Nemati, Zeller. Input Algebras. ICSE 2021.
Mechanized proofs are available

View Slide

Gopinath, Nemati, Zeller. Input Algebras. ICSE 2021.

where
is = class extends (class {}){}
is {while ((<$Id1>)){ if ((<$Id1>)) {break;;var <$Id1>; continue }0}}
is var {<$Id2>:{} = <$Id2> => {}} ;
is const [<$Id3>,<$Id3>] = []
Mechanized proofs are available

View Slide

View Slide

Science of Focused Fuzzing

where is (())
is / 0

View Slide

where is (())
is / 0

View Slide

where is (())
is / 0
Isolating & Decomposing Program Behaviors

View Slide

where is (())
is / 0
Algebra of Program Behaviors

View Slide

where is (())
is / 0
Algebra of Program Behaviors
Science of Program Behaviors

View Slide

View Slide

115
insert into tbl values (1,2,3)
select b from tbl
drop table tbl
Input Behavior
Program
Challenge: Identify Behavior Divergence

View Slide

115
select b from tbl
drop table tbl
update($file)
read($file)
rm($file)
Input Behavior
action='read'
$action('tbl')
Program
assert invoked read: 'tbl.data' ✔

View Slide

115
select b from tbl
drop table tbl
update($file)
read($file)
rm($file)
Input Behavior
action='read'
$action('tbl')
Program
assert invoked read: 'tbl.data' ✔
action='rm'

View Slide

if a == b:
if b == c:
return Equilateral
else:
return Isosceles
else:
if b == c:
return Isosceles
else:
if a == c:
return Isosceles
else:
return Scalene

View Slide

LockB()
LockA()
DoAB()
UnlockB()
UnlockA()

View Slide

LockB()
LockA()
DoAB()
UnlockB()
UnlockA()
LockB()
LockA()
DoAB()
UnlockA()
UnlockB()

View Slide

LockB()
LockA()
DoAB()
UnlockB()
UnlockA()
LockB()
LockA()
DoAB()
UnlockA()
UnlockB()
UnLockA()
LockA()
DoAB()
LockB()
UnlockB()

View Slide

118
if a == b:
if b == c:
return Equilateral
else:
return Isosceles
else:
if b == c:
return Isosceles
else:
if a == c:
return Isosceles
else:
return Scalene

View Slide

119

View Slide

119
:=

View Slide

120
:=
:= (a==b)
| (a!=b)

View Slide

121
:=
:= (a==b)
| (a!=b)
:= (b==c)
| (b!=c)

View Slide

122
:=
:= (a==b)
| (a!=b)
:= (b==c)
| (b!=c)
:= "return Equilateral"
:= "return Isosceles"

View Slide

123
:=
:= (a==b)
| (a!=b)
:= (b==c)
| (b!=c)
:= "return Equilateral"
:= (b==c)
| (b!=c)
:= (a==c)
| (a!=c)
:= "return Scalene"

View Slide

LockB()
LockA()
DoAB()
UnlockB()
UnlockA()
"lockA" "lockB" "UnlockB" "UnlockA"

View Slide

View Slide

126
The Science of Inputs
Program
The Science of Behaviors
✔

View Slide

126
The Science of Fuzzing
The Science of Inputs
Program
The Science of Behaviors
✔

View Slide

View Slide

128
Oracles for Fuzzing
Focused Fuzzing
Automatic Repair
Fault Localization
Beyond Syntax
Grammar Mining

View Slide

128
Oracles for Fuzzing
Focused Fuzzing
Automatic Repair
Fault Localization
Beyond Syntax
Grammar Mining
Onward

View Slide

Program Under Test
pFuzzer
F1 Fuzzer
Inputs
Grammar Miner
https://rahul.gopinath.org

View Slide

Program Under Test
pFuzzer
F1 Fuzzer
Inputs
Grammar Miner
Problem: How to Fuzz Parsers

View Slide

Program Under Test
pFuzzer
F1 Fuzzer
Inputs
Grammar Miner
Generalize

View Slide

Program Under Test
pFuzzer
F1 Fuzzer
Inputs
Grammar Miner
Generalize
Combine

View Slide

Future Work:
Program Under Test
pFuzzer
F1 Fuzzer
Inputs
Grammar Miner
Generalize
Combine

View Slide

Future Work:
Program Under Test
pFuzzer
F1 Fuzzer
Inputs
Grammar Miner
Debugging
Combine

View Slide

Fuzzing -- From Alchemy to a Science

Fuzzing -- From Alchemy to a Science

Rahul Gopinath

More Decks by Rahul Gopinath

Other Decks in Research

Featured

Transcript