Learning And Refining Input Grammars For Effective Fuzzing

Learning and Refining Input Grammars
for
Effective Fuzzing
Rahul Gopinath
1

View Slide

Prerequisites
https://github.com/vrthra/SBST22-tutorial#readme
• Install Python 3.10

• Install Graphviz

• Install Jupyter

• Start Jupyter
2

View Slide

Prerequisites
http://localhost:8888/tree
• Install Python 3.10

• Install Graphviz

• Install Jupyter

• Start Jupyter
3

View Slide

http://localhost:8888/notebooks/RoadMap.ipynb
4

View Slide

http://localhost:8888/notebooks/x0_0_Prerequisites.ipynb
5

View Slide

Search
6

View Slide

Fuzzing
Program
Trash deck technique: 1950s - Gerald Weinberg
7

View Slide

Fuzzing
Crash?
Program
Trash deck technique: 1950s - Gerald Weinberg
7

View Slide

8
Random Fuzzing
Program

View Slide

8
Random Fuzzing
$ ./fuzz
[;x1-GPZ+wcckc];,N9J+?#6^6\e?]9lu 
2_%'4GX"0VUB[E/r ~fApu6b8<{%siq8Z 
h.6{V,hr?;{Ti.r3PIxMMMv6{xS^+'Hq! 
AxB"[email protected]!Kd6;wtAMefFWM(`|J_<1~o} 
z3K(CCzRH JIIvHz>_*.\>JrlU32~eGP? 
lR=bF3+;y$3lodQ)KC-i,c{<[~m!]o;{.'}Gj\(X}EtYetrp 
[email protected]{P!AZU7x#4(Rtn!q4nCwqol^y6 
}0|Ko=*JK~;zMKV=9Nai:wxu{J&UV#HaU 
)*BiC<),`+t*gkaPq>&]BS6R&j?#tP7iaV}-}`\?[_[Z^LBM 
PG-FKj'\xwuZ1=Q`^`5,[email protected][!CuRzJ2 
D|vBy!^zkhdf3C5PAkR?V((-%>i2Qx]D$qs4O`[email protected]'2\[email protected] 
5:dfd45*(7^%5ap\zIyl"'f,$ee,J4Gw: 
cgNKLie3nx9(`efSlg6#[K"@WjhZ}r[Sc 
un&sBCS,T[/3]KAeEnQ7lU)3Pn,0)G/6N 
-wyzj/MTd#A;r*(ds./df3r8Odaf?/<#r
Program
✘

View Slide

$ ./fuzz -int | program

634111569742810193727424069509
 
741355925061499451162464719526
 
615957331924826555590537407605
 
181400079803446874252046374716
 
740973770255348279425601333144
 
152724057932073828569041216191
 
099859446496509919024810271242
 
622974988671421938012464630138
 
735355134599327240920259675263
 
574528613057084231370741920902
 
794677842164654990353575580453
 
777282305855352378119038096476
 
699871306655084953377039862387
 
924957554389878352934547664240
 
082431556093837288597262675598
 
630851919061829885048834738832
 
677022429414980917053939970795
 
722006987916088650168665471731 yes
9
Feedback Driven Fuzzing
def is_prime(n: int) -> bool:

"""Primality test using 6k+-1 optimization."""

if n <= 3:

return n > 1

if n % 2 == 0 or n % 3 == 0:

return False

i = 5

while i ** 2 <= n:

if n % i == 0 or n % (i + 2) == 0:

return False

i += 6

return True

def main():

num = stdin.read()

print(num, is_prime(num))

View Slide

http://localhost:8888/notebooks/x1_0_GeneratingSamples.ipynb
10

View Slide

11
Fuzzing Parsers
$ ./fuzz
Interpreter

View Slide

11
Fuzzing Parsers
$ ./fuzz
Parser
Syntax Error
Interpreter
#

View Slide

12
Sepcification Free Generators

View Slide

12
A

View Slide

12
A
A ∉ (,+,-,1,2,3,4,5,6,7,8,9,0

View Slide

12
A
(
A ∉ (,+,-,1,2,3,4,5,6,7,8,9,0

View Slide

12
A
( 2
A ∉ (,+,-,1,2,3,4,5,6,7,8,9,0

View Slide

12
A
( 2
-
B
9
)
4 )
A ∉ (,+,-,1,2,3,4,5,6,7,8,9,0
B ∉ +,-,1,2,3,4,5,6,7,8,9,0,)
) ∉ +,-,1,2,3,4,5,6,7,8,9,0

View Slide

12
A
( 2
-
B
9
)
4 )
A ∉ (,+,-,1,2,3,4,5,6,7,8,9,0
B ∉ +,-,1,2,3,4,5,6,7,8,9,0,)
) ∉ +,-,1,2,3,4,5,6,7,8,9,0
(2-94)

View Slide

http://localhost:8888/notebooks/x1_1_TrackingAccess.ipynb
13

View Slide

14
Limitation: Lack of control

View Slide

15
Constraining the Search Space

with

Input Grammars

View Slide

16
Grammar

View Slide

17
Formal Languages
Formal Language Descriptions

View Slide

17
Formal Languages
3. Regular
(Chomsky,1956)

View Slide

17
Formal Languages
3. Regular
Context Free
(Chomsky,1956)
Argument Stack

View Slide

17
Formal Languages
3. Regular
Context Free
Recursively Enumerable
(Chomsky,1956)
Argument Stack
Return Stack

View Slide

17
Formal Languages
3. Regular
Context Free
Recursively Enumerable
(Chomsky,1956)
Easy to produce and parse
Argument Stack
Return Stack

View Slide

18
Grammar
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
Arithmetic expression grammar

View Slide

18
Grammar
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
key

View Slide

18
Grammar
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
De
f
inition for
key

View Slide

19
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
Grammar

View Slide

19
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
Grammar
Expansion Rule

View Slide

19
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
Grammar
Expansion Rule Terminal Symbol

View Slide

19
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
Grammar
Expansion Rule Terminal Symbol
Nonterminal Symbol

View Slide

http://localhost:8888/notebooks/x0_1_Grammars.ipynb
20

View Slide

21
Grammars
For Parsing
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]

View Slide

21
Grammars
For Parsing
(8 / 3) * 49
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]

View Slide

http://localhost:8888/notebooks/x0_3_Parser.ipynb
22

View Slide

23
Grammars
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
For Fuzzing (Hanford 1970)

(Purdom 1972)

View Slide

23
Grammars8.2 - 27 - -9 / +((+9 * --2 + --+-+-
 
((-1 * +(8 - 5 - 6)) * (-(a-+(((+(4)
 
)))) - ++4) / +(-+---((5.6 - --(3 *
 
-1.8 * +(6 * +-(((-(-6) * ---+6)) /
 
+--(+-+-7 * (-0 * (+(((((2)) + 8 - 3
 
- ++9.0 + ---(--+7 / (1 / +++6.37)
 
+ (1) / 482) / +++-+0)))) + 8.2 - 27
 
- -9 / +((+9 * --2 + --+-+-((-1 * +
 
(8 - 5 - 6)) * (-(a-+(((+(4))))) - +
+4) / +(-+---((5.6 - --(3 * -1.8 * +
 
(6 * +-(((-(-6) * ---+6)) / +--(+-+-
 
7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0
 
+ ---(--+7 / (1 / +++6.37) + (1) /
 
482) / +++-+0)))) * -+5 + 7.513))))
 
- (+1 / ++((-84)))))))) * ++5 / +-(-
 
-2 - -++-9.0)))) / 5 * --++090 + * -
 
+5 + 7.513)))) - (+1 / ++((-84))))))
 
)) * 8.2 - 27 - -9 / +((+9 * --2 + -
 
-+-+-((-1 * +(8 - 5 - 6)) * (-(a-+((
 
(+(4))))) - ++4) / +(-+---((5.6 - --
 
(3 * -1.8 * +(6 * +-(((-(-6) * ---+6
 
)) / +--(+-+-7 * (-0 * (+(((((2)) +
 
8 - 3 - ++9.0 + ---(--+7 / (1 / +++6
 
.37) + (1) / 482) / +++-+0)))) * -+5
 
+ 7.513)))) - (+1 / ++((-84))))))))
 
* ++5 / +-(--2 - -++-9.0)))) / 5 *
 
--++090 ++5 / +-(--2 - -++-9.0)))) /
 
5 * --++090
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
For Fuzzing (Hanford 1970)

(Purdom 1972)

View Slide

24
Grammars
As effective producers
8.2 - 27 - -9 / +((+9 * --2 + --+-+-
 
((-1 * +(8 - 5 - 6)) * (-(a-+(((+(4)
 
)))) - ++4) / +(-+---((5.6 - --(3 *
 
-1.8 * +(6 * +-(((-(-6) * ---+6)) /
 
+--(+-+-7 * (-0 * (+(((((2)) + 8 - 3
 
- ++9.0 + ---(--+7 / (1 / +++6.37)
 
+ (1) / 482) / +++-+0)))) + 8.2 - 27
 
- -9 / +((+9 * --2 + --+-+-((-1 * +
 
(8 - 5 - 6)) * (-(a-+(((+(4))))) - +
+4) / +(-+---((5.6 - --(3 * -1.8 * +
 
(6 * +-(((-(-6) * ---+6)) / +--(+-+-
 
7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0
 
+ ---(--+7 / (1 / +++6.37) + (1) /
 
482) / +++-+0)))) * -+5 + 7.513))))
 
- (+1 / ++((-84)))))))) * ++5 / +-(-
 
-2 - -++-9.0)))) / 5 * --++090 + * -
 
+5 + 7.513)))) - (+1 / ++((-84))))))
 
)) * 8.2 - 27 - -9 / +((+9 * --2 + -
 
-+-+-((-1 * +(8 - 5 - 6)) * (-(a-+((
 
(+(4))))) - ++4) / +(-+---((5.6 - --
 
(3 * -1.8 * +(6 * +-(((-(-6) * ---+6
 
)) / +--(+-+-7 * (-0 * (+(((((2)) +
 
8 - 3 - ++9.0 + ---(--+7 / (1 / +++6
 
.37) + (1) / 482) / +++-+0)))) * -+5
 
+ 7.513)))) - (+1 / ++((-84))))))))
 
* ++5 / +-(--2 - -++-9.0)))) / 5 *
 
--++090 ++5 / +-(--2 - -++-9.0)))) /
 
5 * --++090

View Slide

24
Grammars
As effective producers
Interpreter
Parser
✘
✔
8.2 - 27 - -9 / +((+9 * --2 + --+-+-
 
((-1 * +(8 - 5 - 6)) * (-(a-+(((+(4)
 
)))) - ++4) / +(-+---((5.6 - --(3 *
 
-1.8 * +(6 * +-(((-(-6) * ---+6)) /
 
+--(+-+-7 * (-0 * (+(((((2)) + 8 - 3
 
- ++9.0 + ---(--+7 / (1 / +++6.37)
 
+ (1) / 482) / +++-+0)))) + 8.2 - 27
 
- -9 / +((+9 * --2 + --+-+-((-1 * +
 
(8 - 5 - 6)) * (-(a-+(((+(4))))) - +
+4) / +(-+---((5.6 - --(3 * -1.8 * +
 
(6 * +-(((-(-6) * ---+6)) / +--(+-+-
 
7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0
 
+ ---(--+7 / (1 / +++6.37) + (1) /
 
482) / +++-+0)))) * -+5 + 7.513))))
 
- (+1 / ++((-84)))))))) * ++5 / +-(-
 
-2 - -++-9.0)))) / 5 * --++090 + * -
 
+5 + 7.513)))) - (+1 / ++((-84))))))
 
)) * 8.2 - 27 - -9 / +((+9 * --2 + -
 
-+-+-((-1 * +(8 - 5 - 6)) * (-(a-+((
 
(+(4))))) - ++4) / +(-+---((5.6 - --
 
(3 * -1.8 * +(6 * +-(((-(-6) * ---+6
 
)) / +--(+-+-7 * (-0 * (+(((((2)) +
 
8 - 3 - ++9.0 + ---(--+7 / (1 / +++6
 
.37) + (1) / 482) / +++-+0)))) * -+5
 
+ 7.513)))) - (+1 / ++((-84))))))))
 
* ++5 / +-(--2 - -++-9.0)))) / 5 *
 
--++090 ++5 / +-(--2 - -++-9.0)))) /
 
5 * --++090

View Slide

25
Grammars
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
As efficient producers

View Slide

25
Grammars
:=
:= '+'
| '-'
| '/'
| '*'
| '(' ')'
|
:=
| '.'
:=
|
:= [0-9]
As efficient producers
def start():
expr()
def expr():
match (random() % 6):
case 0: expr(); print('+'); expr()
case 1: expr(); print('-'); expr()
case 2: expr(); print('/'); expr()
case 3: expr(); print('*'); expr()
case 4: print('('); expr(); print(')')
case 5: number()
def number():
case 0: integer()
case 1: integer(); print('.'); integer()
def integer():
case 0: digit(); integer()
case 1: digit()
def digit():
case 0: print('0')
case 1: print('1')
case 2: print('2')
case 3: print('3')
case 4: print('4')
case 5: print('5')
case 6: print('6')
case 7: print('7')
Compiled Grammar (F1)

View Slide

http://localhost:8888/notebooks/x0_2_GrammarFuzzer.ipynb
26

View Slide

27
Where to Get the Input Grammar From?

View Slide

http://localhost:8888/notebooks/x2_0_MiningGrammar.ipynb
28

View Slide

29
Where to Get the Grammar From?
Hand-written parsers already encode the grammar

View Slide

29
Where to Get the Grammar From?
1. Extract the input string accesses

2. Attach control
fl
ow information (context-managers)
Hand-written parsers already encode the grammar

View Slide

30
How to Extract This Grammar?

View Slide

30
• Inputs + control
fl
ow -> Dynamic Control Dependence Trees

View Slide

30
• Inputs + control
fl
ow -> Dynamic Control Dependence Trees
• DCD Trees -> Parse Tree

View Slide

31
Control Dependence Graph
Statement B is control dependent on A if A determines whether B executes.
def parse_csv(s,i):

while s[i:]:

if is_digit(s[i]):

n,j = num(s[i:])

i = i+j

else:

comma(s[i])

i += 1

View Slide

31
def parse_csv(s,i):

while s[i:]:

if is_digit(s[i]):

n,j = num(s[i:])

i = i+j

else:

comma(s[i])

i += 1
CDG for parse_csv

View Slide

31
def parse_csv(s,i):

while s[i:]:

if is_digit(s[i]):

n,j = num(s[i:])

i = i+j

else:

comma(s[i])

i += 1
CDG for parse_csv
while: determines

whether

if: executes

View Slide

32
def parse_csv(s,i):

while s[i:]:

if is_digit(s[i]):

n,j = num(s[i:])

i = i+j

else:

comma(s[i])

i += 1
CDG for parse_csv
Dynamic Control Dependence Tree
Each statement execution is represented as a separate node

View Slide

32
def parse_csv(s,i):

while s[i:]:

if is_digit(s[i]):

n,j = num(s[i:])

i = i+j

else:

comma(s[i])

i += 1
CDG for parse_csv
Dynamic Control Dependence Tree
Each statement execution is represented as a separate node
DCD Tree for call parse_csv()

View Slide

33
def parse_csv(s,i):

while s[i:]:

if is_digit(s[i]):

n,j = num(s[i:])

i = i+j

else:

comma(s[i])

i += 1
DCD Tree ~ Parse Tree
•No tracking beyond input bu
ff
er

•Characters are attached to nodes where they are accessed last
"12,"
"12,"

View Slide

33
def parse_csv(s,i):

while s[i:]:

if is_digit(s[i]):

n,j = num(s[i:])

i = i+j

else:

comma(s[i])

i += 1
'1' '2' ','
DCD Tree ~ Parse Tree
•No tracking beyond input bu
ff
er

•Characters are attached to nodes where they are accessed last
"12,"
"12,"

View Slide

34
def is_digit(i): return i in '0123456789'

def parse_num(s,i):

n = ''

while s[i:] and is_digit(s[i]):

n += s[i]

i = i +1

return i,n

def parse_paren(s, i):

assert s[i] == '('

i, v = parse_expr(s, i+1)

if s[i:] == '': raise Ex(s, i)

assert s[i] == ')'

return i+1, v

def parse_expr(s, i = 0):

expr, is_op = [], True

while s[i:]:

c = s[i]

if isdigit(c):

if not is_op: raise Ex(s,i)

i,num = parse_num(s,i)

expr.append(num)

is_op = False

elif c in ['+', '-', '*', '/']:

if is_op: raise Ex(s,i)

expr.append(c)

is_op, i = True, i + 1

elif c == '(':

if not is_op: raise Ex(s,i)

i, cexpr = parse_paren(s, i)

expr.append(cexpr)

is_op = False

elif c == ')': break

else: raise Ex(s,i)

if is_op: raise Ex(s,i)

return i, expr
9+3/4
Parse tree for parse_expr('9+3/4')

View Slide

35

View Slide

36

View Slide

37
3 * (9 + 1)

View Slide

37
(9 + 1) * 3
3 * (9 + 1)

View Slide

38
3 * (9 + 1)

View Slide

38
9 + 1
3 * (9 + 1)

View Slide

39
3 * (9 + 1)

View Slide

39
3 (9 + 1) *
3 * (9 + 1)

View Slide

40

View Slide

41
3*(1)
1

View Slide

42
3*(1)
1

View Slide

42
3*(1)
1
:=

View Slide

42
3*(1)
1
:=

:=

View Slide

:=
|
|
|
:=
:=
|
:=
:= '3' | '1'
:= '(' ')'
:=
:= '*'
43

View Slide

:=
:=
|
:=
|
|
|
:=
:=
|
:=
:= '3' | '1'
:= '(' ')'
:=
:= '*'
43

View Slide

44

:=

|

:=

|

:= '(' ')'

|

:= '*' | '+' | '-' | '/'

:=

|

: [0-9]

calc.py Recovered Arithmetic Grammar

View Slide

45
:=

:=

|

:=

|

:= '(' ')'

|

:= '*' | '+' | '-' | '/'

:=

|

: [0-9]

View Slide

45
8.2 - 27 - -9 / +((+9 * --2 + --+-+-
 
((-1 * +(8 - 5 - 6)) * (-(a-+(((+(4)
 
)))) - ++4) / +(-+---((5.6 - --(3 *
 
-1.8 * +(6 * +-(((-(-6) * ---+6)) /
 
+--(+-+-7 * (-0 * (+(((((2)) + 8 - 3
 
- ++9.0 + ---(--+7 / (1 / +++6.37)
 
+ (1) / 482) / +++-+0)))) + 8.2 - 27
 
- -9 / +((+9 * --2 + --+-+-((-1 * +
 
(8 - 5 - 6)) * (-(a-+(((+(4))))) - +
+4) / +(-+---((5.6 - --(3 * -1.8 * +
 
(6 * +-(((-(-6) * ---+6)) / +--(+-+-
 
7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0
 
+ ---(--+7 / (1 / +++6.37) + (1) /
 
482) / +++-+0)))) * -+5 + 7.513))))
 
- (+1 / ++((-84)))))))) * ++5 / +-(-
 
-2 - -++-9.0)))) / 5 * --++090 + * -
 
+5 + 7.513)))) - (+1 / ++((-84))))))
 
)) * 8.2 - 27 - -9 / +((+9 * --2 + -
 
-+-+-((-1 * +(8 - 5 - 6)) * (-(a-+((
 
(+(4))))) - ++4) / +(-+---((5.6 - --
 
(3 * -1.8 * +(6 * +-(((-(-6) * ---+6
 
)) / +--(+-+-7 * (-0 * (+(((((2)) +
 
8 - 3 - ++9.0 + ---(--+7 / (1 / +++6
 
.37) + (1) / 482) / +++-+0)))) * -+5
 
+ 7.513)))) - (+1 / ++((-84))))))))
 
* ++5 / +-(--2 - -++-9.0)))) / 5 *
 
--++090 ++5 / +-(--2 - -++-9.0)))) /
 
5 * --++090
:=

:=

|

:=

|

:= '(' ')'

|

:= '*' | '+' | '-' | '/'

:=

|

: [0-9]

View Slide

46
::=
::= '"'
| '['
| '{'
|
| 'true'
| 'false'
| 'null'
::= +
| + 'e' +
::= '+' | '-' | '.' | [0-9] | 'E' | 'e'
::= * '"'
::= ']'
| (',')* ']'
| ( ',' )+ (',' )* ']'
::= '}'
| ( '"' ':' ',' )*
'"' ':' '}'
::= ' ' | '!' | '#' | '$' | '%' | '&' | '''
| '*' | '+' | '-' | ',' | '.' | '/' | ':' | ';'
| '<' | '=' | '>' | '?' | '@' | '[' | ']' | '^'
| '_', ''',| '{' | '|' | '}' | '~'
| '[A-Za-z0-9]'
| '\'
::= '"' | '/' | 'b' | 'f' | 'n' | 'r' | 't'
stm.next()

View Slide

47

View Slide

48

View Slide

49
We Found A Crash

View Slide

Why Did My Program Crash?
50

View Slide

8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 -
 
5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.
 
6 - --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +-
 
-(+-+-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(
 
--+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) +
 
8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 -
 
5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.6
 
- --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +--
 
(+-+-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(-
 
-+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) * -
 
+5 + 7.513)))) - (+1 / ++((-84)))))))) * ++5 / +-
 
(--2 - -++-9.0)))) / 5 * --++090 + * -+5 + 7.513)
 
))) - (+1 / ++((-84)))))))) * 8.2 - 27 - -9 / +((
 
+9 * --2 + --+-+-((-1 * +(8 - 5 - 6)) * (-(a-+
(((+(4))))) - ++4) / +(-+---((5.6 - --(3 * -1.8 *
 
+(6 * +-(((-(-6) * ---+6)) / +--(+-+-7 * (-0 * (
 
+(((((2)) + 8 - 3 - ++9.0 + ---(--+7 / (1 / +++6.
 
37) + (1) / 482) / +++-+0)))) * -+5 + 7.513)))) -
 
(+1 / ++((-84)))))))) * ++5 / +-(--2 - -++-9.0)))
 
) / 5 * --++090 ++5 / +-(--2 - -++-9.0)))) / 5 *
 
--++090
50

View Slide

8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 -
 
5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.
 
6 - --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +-
 
-(+-+-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(
 
--+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) +
 
8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 -
 
5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.6
 
- --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +--
 
(+-+-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(-
 
-+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) * -
 
+5 + 7.513)))) - (+1 / ++((-84)))))))) * ++5 / +-
 
(--2 - -++-9.0)))) / 5 * --++090 + * -+5 + 7.513)
 
))) - (+1 / ++((-84)))))))) * 8.2 - 27 - -9 / +((
 
+9 * --2 + --+-+-((-1 * +(8 - 5 - 6)) * (-(a-+
(((+(4))))) - ++4) / +(-+---((5.6 - --(3 * -1.8 *
 
+(6 * +-(((-(-6) * ---+6)) / +--(+-+-7 * (-0 * (
 
+(((((2)) + 8 - 3 - ++9.0 + ---(--+7 / (1 / +++6.
 
37) + (1) / 482) / +++-+0)))) * -+5 + 7.513)))) -
 
(+1 / ++((-84)))))))) * ++5 / +-(--2 - -++-9.0)))
 
) / 5 * --++090 ++5 / +-(--2 - -++-9.0)))) / 5 *
 
--++090
DD Minimized Input
((4))
50

View Slide

8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 -
 
5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.
 
6 - --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +-
 
-(+-+-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(
 
--+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) +
 
8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 -
 
5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.6
 
- --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +--
 
(+-+-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(-
 
-+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) * -
 
+5 + 7.513)))) - (+1 / ++((-84)))))))) * ++5 / +-
 
(--2 - -++-9.0)))) / 5 * --++090 + * -+5 + 7.513)
 
))) - (+1 / ++((-84)))))))) * 8.2 - 27 - -9 / +((
 
+9 * --2 + --+-+-((-1 * +(8 - 5 - 6)) * (-(a-+
(((+(4))))) - ++4) / +(-+---((5.6 - --(3 * -1.8 *
 
+(6 * +-(((-(-6) * ---+6)) / +--(+-+-7 * (-0 * (
 
+(((((2)) + 8 - 3 - ++9.0 + ---(--+7 / (1 / +++6.
 
37) + (1) / 482) / +++-+0)))) * -+5 + 7.513)))) -
 
(+1 / ++((-84)))))))) * ++5 / +-(--2 - -++-9.0)))
 
) / 5 * --++090 ++5 / +-(--2 - -++-9.0)))) / 5 *
 
--++090
DD Minimized Input
((4))
00000 ?
50

View Slide

8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 -
 
5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.
 
6 - --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +-
 
-(+-+-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(
 
--+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) +
 
8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 -
 
5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.6
 
- --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +--
 
(+-+-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(-
 
-+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) * -
 
+5 + 7.513)))) - (+1 / ++((-84)))))))) * ++5 / +-
 
(--2 - -++-9.0)))) / 5 * --++090 + * -+5 + 7.513)
 
))) - (+1 / ++((-84)))))))) * 8.2 - 27 - -9 / +((
 
+9 * --2 + --+-+-((-1 * +(8 - 5 - 6)) * (-(a-+
(((+(4))))) - ++4) / +(-+---((5.6 - --(3 * -1.8 *
 
+(6 * +-(((-(-6) * ---+6)) / +--(+-+-7 * (-0 * (
 
+(((((2)) + 8 - 3 - ++9.0 + ---(--+7 / (1 / +++6.
 
37) + (1) / 482) / +++-+0)))) * -+5 + 7.513)))) -
 
(+1 / ++((-84)))))))) * ++5 / +-(--2 - -++-9.0)))
 
) / 5 * --++090 ++5 / +-(--2 - -++-9.0)))) / 5 *
 
--++090
DD Minimized Input
((4))
00000 ?
((5)) ?
50

View Slide

8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 -
 
5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.
 
6 - --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +-
 
-(+-+-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(
 
--+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) +
 
8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 -
 
5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.6
 
- --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +--
 
(+-+-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(-
 
-+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) * -
 
+5 + 7.513)))) - (+1 / ++((-84)))))))) * ++5 / +-
 
(--2 - -++-9.0)))) / 5 * --++090 + * -+5 + 7.513)
 
))) - (+1 / ++((-84)))))))) * 8.2 - 27 - -9 / +((
 
+9 * --2 + --+-+-((-1 * +(8 - 5 - 6)) * (-(a-+
(((+(4))))) - ++4) / +(-+---((5.6 - --(3 * -1.8 *
 
+(6 * +-(((-(-6) * ---+6)) / +--(+-+-7 * (-0 * (
 
+(((((2)) + 8 - 3 - ++9.0 + ---(--+7 / (1 / +++6.
 
37) + (1) / 482) / +++-+0)))) * -+5 + 7.513)))) -
 
(+1 / ++((-84)))))))) * ++5 / +-(--2 - -++-9.0)))
 
) / 5 * --++090 ++5 / +-(--2 - -++-9.0)))) / 5 *
 
--++090
DD Minimized Input
((4))
00000 ?
((5)) ?
(++5) ?
50

View Slide

51

View Slide

52
Issue 386 from Rhino
var A = class extends (class {}){};
Issue 2937 from Closure
const [y,y] = [];
var {baz:{} = baz => {}} = baz => {};
{while ((l_0)){ if ((l_0)) {break;;var l_0; continue }0}}

View Slide

52
var A = class extends (class {}){};
const [y,y] = [];
var {baz:{} = baz => {}} = baz => {};
{while ((l_0)){ if ((l_0)) {break;;var l_0; continue }0}}
Delta Minimization is useful but not su
ff
i
cient

View Slide

( ( 4 ) )
53

View Slide

( ( 4 ) )
:=
:= ' + '
| ' - '
|
:= ' * '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '.'
|
:=
|
:= [0-9]
53

View Slide

( ( 4 ) )
54

View Slide

( ( 4 ) )
:=
:= ' + '
| ' - '
|
:= ' * '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '.'
|
:=
|
:= [0-9]
54

View Slide

( ( 4 ) )
:=
:= ' + '
| ' - '
|
:= ' * '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '.'
|
:=
|
:= [0-9]
55

View Slide

( ( 4 ) )
:=
:= ' + '
| ' - '
|
:= ' * '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '.'
|
:=
|
:= [0-9]
✓ Did not reproduce the failure
1 * (2 - 3)
55

View Slide

( ( 4 ) )
:=
:= ' + '
| ' - '
|
:= ' * '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '.'
|
:=
|
:= [0-9]
56

View Slide

( ( 4 ) )
:=
:= ' + '
| ' - '
|
:= ' * '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '.'
|
:=
|
:= [0-9]
c
57

View Slide

( ( 4 ) )
:=
:= ' + '
| ' - '
|
:= ' * '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '.'
|
:=
|
:= [0-9]
c
58

View Slide

( ( 4 ) )
:=
:= ' + '
| ' - '
|
:= ' * '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '.'
|
:=
|
:= [0-9]
c
1 + 3 + 4
58

View Slide

( ( 4 ) )
:=
:= ' + '
| ' - '
|
:= ' * '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '.'
|
:=
|
:= [0-9]
c
c
59

View Slide

3 * 4
:=
:= ' + '
| ' - '
|
:= ' * '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '.'
|
:=
|
:= [0-9]
c
c
60

View Slide

( ( 4 ) )
:=
:= ' + '
| ' - '
|
:= ' * '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '.'
|
:=
|
:= [0-9]
c
c
c
c
c
c
c
61

View Slide

( ( 1 - 2 ) )
:=
:= ' + '
| ' - '
|
:= ' * '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '.'
|
:=
|
:= [0-9]
c
c
c
c
c
c
c
( ( 1 - 2 ) )
62

View Slide

( ( 1 - 2 ) )
:=
:= ' + '
| ' - '
|
:= ' * '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '.'
|
:=
|
:= [0-9]
c
c
c
c
c
c
c
✘ reproduced the failure
( ( 1 - 2 ) )
62

View Slide

( ( 1 - 2 ) )
c
c
c
c
c
c
c
( ( 1 - 2 ) )
63

View Slide

( ( 1 - 2 ) )
c
c
c
c
c
c
c
✘
( ( 1 - 2 ) )
63

View Slide

( ( 1 - 2 ) )
c
c
c
c
c
c
c
✘
( ( 1 - 2 ) )
( ( 2 * 3 + 4 ) )
64

View Slide

( ( 1 - 2 ) )
c
c
c
c
c
c
c
✘
( ( 1 - 2 ) )
✘
( ( 2 * 3 + 4 ) )
64

View Slide

( ( 1 - 2 ) )
c
c
c
c
c
c
c
✘
( ( 1 - 2 ) )
✘
( ( 2 * 3 + 4 ) )
( ( - 2 / 1 ) )
65

View Slide

( ( 1 - 2 ) )
c
c
c
c
c
c
c
✘
( ( 1 - 2 ) )
✘
( ( 2 * 3 + 4 ) )
✘
( ( - 2 / 1 ) )
65

View Slide

( ( 1 - 2 ) )
c
c
c
c
c
c
c
✘
( ( 1 - 2 ) )
✘
( ( 2 * 3 + 4 ) )
✘
( ( - 2 / 1 ) )
( ( 98 - 0 ) )
66

View Slide

( ( 1 - 2 ) )
c
c
c
c
c
c
c
✘
( ( 1 - 2 ) )
✘
( ( 2 * 3 + 4 ) )
✘
( ( - 2 / 1 ) )
✘
( ( 98 - 0 ) )
66

View Slide

)
(
( )
( ( )
4 )
( ( 4 ) )
c
c
c
c
c
c
c
A
67

View Slide

)
(
( )
( ( )
4 )
( ( 4 ) )
c
c
c
c
c
c
c
A
68

View Slide

( ( 4 ) )
c
c
c
c
c
c
c
A
( ( ) )

( ( ) )
4
Minimized Input
Abstract Failure Inducing Input
def check(parsed):
if parsed.is_nested() and parsed.child.is_nested():
raise Exception()
return input
69

View Slide

70
:=
:= ' + '
| ' + '
| ' - '
| ' - '
|
:= ' * '
| ' * '
| ' / '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '(' ')'
:=
:=
:= '(' ')'
Specialized Grammar
is (())

View Slide

70
:=
:= ' + '
| ' + '
| ' - '
| ' - '
|
:= ' * '
| ' * '
| ' / '
| ' / '
|
:= '+'
| '-'
| '(' ')'
| '(' ')'
:=
:=
:= '(' ')'
((1)) + 2
(23 * ((3)) - 34)
(344- 4 + ((223)))
(1) - 3 * 773 + (-22 + 1)
1798 - 889 / ((333-1)) * 2 / 3 + 1
34 + ((4)) -334 + (334 - (22) + 919 * 0 + 1
98435747+ 88 + (((0))) + (1) - 1 * 7 / 4 * 889 - 2
8 + ((8)) + --1 + 11223 / 344 - 39 + (1) - 456 + 134 / 45
437 + 8 - 1 * ((9 + ((1))) - 1 + 99111948 + 3 --1 + (112) - 2 + 445) + 0
74 + 334 + ((178 - 88 / (3393-1) * 1002 / 3 + 1+ 3439)) * 223 - 1233 + 334672
2 * ((9)) - (1798 - 889 / (333-1) * 2 / 3 + 100012 + 3434392 + 234 ----6 * 1798 - 889 / (33
778 - (((1) - 3 * 773 + (-22 + 1) * (4545) - 23 - ((2)) * 773 + (-22 + 1) / 3434 + ---1 + 1 / 34343 + 112
349 + (((1) - 3 * 3 + (-22 + 1) ((+ (-22 + 1) * (4545) - 23 - (2) * 773 + ((-22 + 1)) / 3434 + ---1 + 1 / 34343 + 1123
8 + ((8)) + --1 + / 1 - 39 + (1) - 456 + 134 / 45 ))(((1) - 2334 + ((((1)) - 3 * 773 + (-22 + 1) * (2) - 23 - (2) * 773 + (-22 + 1) / 3
74 + 3 + ((178 - 88 / (3393-1) * 1002 / 3 + 1+ 3439)) * - 1233 + 334672)) ((8 + ((8)) + --1 + / 344 - 39 + (1) - 456 + 134 / 45 ))(((1) - 3 * 773
1+ 33+ 24343433 +23343 - ((74 + 334 + ((178 - 88 / (3393-1) * 1002 / 3 + 1+ 3439)) * - 1233 + 334672)) ((8 + ((8)) + --1 + / 344 - 39 + (1) - 456 + 134 / 4
✘
Specialized Grammar
is (())
✘

View Slide

Input Algebras
71

View Slide

72

View Slide

def jsoncheck(json):
...
{...}
73

View Slide

if any_key_has_null_value(json):
fail(’key value must not be null’)
process(json)
{"abc": null}
✘
74

View Slide

process(json)
{"abc": null}
✘
is : null
74

View Slide

{"abc": []}
✔
process(json)
75

View Slide

{"abc": []}
✔
no is : null
process(json)
75

View Slide

{"abc": 124}
✘
no is "" :
if no_key_is_empty_string(json):
fail(’one key must be empty’)
process(json)
76

View Slide

process(json)
{"": 124}
✔
is "" :
77

View Slide

process(json)
78

View Slide

process(json)
{"": 124}
✔
is "" :
no is : null
&
79

View Slide

process(json)
{"abc": null}
✘
is : null
Start Symbol
80

View Slide

process(json)
{"abc": []}
no is : null
✔
Start Symbol
81

View Slide

process(json)
{"abc": 124}
✘
is "" :
Start Symbol
82

View Slide

is "" :
no is : null
&
process(json)
{"": 124}
✔
Start Symbol
83

View Slide

process(json)
Evogram
84

View Slide

process(json)
Evogram
Automatically Derived
84

View Slide

process(json)
85

View Slide

process(json)
Automatically Derived
85

View Slide

Supercharged Pattern Matchers

where

is "":

is :null

where

is (())

is / 0

where

is "0"

is "0x"

where

is ";;"

is "()"

is "()"

Alternative to Regular Expressions
86

View Slide

87

View Slide

Learning And Refining Input Grammars For Effective Fuzzing

Learning And Refining Input Grammars For Effective Fuzzing

Rahul Gopinath

More Decks by Rahul Gopinath

Other Decks in Research

Featured

Transcript