Mining Input Grammars from Dynamic Control Flow

Slide 1

Slide 1 text

Mining Input Grammars from Dynamic Control Flow Rahul Gopinath Björn Mathis Andreas Zeller CISPA Helmholtz Center for Information Security

Slide 2

Slide 2 text

Why do We Need a Grammar?

Slide 3

Slide 3 text

Traditional Fuzzing Program

Slide 4

Slide 4 text

Traditional Fuzzing $ ./fuzz [;x1-GPZ+wcckc];,N9J+?#6^6\e?]9lu  2_%'4GX"0VUB[E/r ~fApu6b8<{%siq8Z  h.6{V,hr?;{Ti.r3PIxMMMv6{xS^+'Hq!  AxB"YXRS@!Kd6;wtAMefFWM(`|J_<1~o}  z3K(CCzRH JIIvHz>_*.\>JrlU32~eGP?  lR=bF3+;y$3lodQ&]BS6R&j?#tP7iaV}-}`\?[_[Z^LBM  PG-FKj'\xwuZ1=Q`^`5,$N$Q@[!CuRzJ2  D|vBy!^zkhdf3C5PAkR?V((-%>

Slide 5

Slide 5 text

Structured Inputs $ ./fuzz [;x1-GPZ+wcckc];,N9J+?#6^6\e?]9lu  2_%'4GX"0VUB[E/r ~fApu6b8<{%siq8Z  h.6{V,hr?;{Ti.r3PIxMMMv6{xS^+'Hq!  AxB"YXRS@!Kd6;wtAMefFWM(`|J_<1~o}  z3K(CCzRH JIIvHz>_*.\>JrlU32~eGP?  lR=bF3+;y$3lodQ&]BS6R&j?#tP7iaV}-}`\?[_[Z^LBM  PG-FKj'\xwuZ1=Q`^`5,$N$Q@[!CuRzJ2  D|vBy!^zkhdf3C5PAkR?V((-%>

Slide 6

Slide 6 text

Slide 7

Slide 7 text

def process_input(input): try: ast = parse(input) res = evaluate(ast) return res except SyntaxError: return Error

Slide 8

Slide 8 text

def process_input(input): try: ast = parse(input) res = evaluate(ast) return res except SyntaxError: return Error The Core

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

SYNTAX ERROR

Slide 11

Slide 11 text

Use an Input Grammar

Slide 12

Slide 12 text

Slide 13

Slide 13 text

9 := := ' + ' | ' - ' | := ' * ' | ' / ' | := '+' | '-' | '(' ')' | '.' | := | := [0-9]

Slide 14

Slide 14 text

10 := := ' + ' | ' - ' | := ' * ' | ' / ' | := '+' | '-' | '(' ')' | '.' | := | := [0-9]

Slide 15

Slide 15 text

10 := := ' + ' | ' - ' | := ' * ' | ' / ' | := '+' | '-' | '(' ')' | '.' | := | := [0-9] 8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 - 5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.6 - --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +--(+- +-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(--+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) + 8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 - 5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.6 - --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +--(+- +-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(--+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) * - +5 + 7.513)))) - (+1 / ++((-84)))))))) * ++5 / +-(--2 - -++-9.0)))) / 5 * --++090 + * -+5 + 7.513)))) - (+1 / ++((-84)))))))) * 8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 - 5 - 6)) * (-(a- +(((+(4))))) - ++4) / +(-+---((5.6 - --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +--(+-+-7 * (-0 * (+ (((((2)) + 8 - 3 - ++9.0 + ---(--+7 / (1 / ++ +6.37) + (1) / 482) / +++-+0)))) * -+5 + 7.513)))) - (+1 / ++((-84)))))))) * ++5 / +-(--2 - -++-9.0)))) / 5 * --++090 ++5 / +-(--2 - -+ +-9.0)))) / 5 * --++090

Slide 16

Slide 16 text

def process_input(input): try: ast = parse(input) res = evaluate(ast) return res except SyntaxError: return Error

Slide 17

Slide 17 text

def process_input(input): try: ast = parse(input) res = evaluate(ast) return res except SyntaxError: return Error The Core

Slide 18

Slide 18 text

Where to Get the Grammar From?

Slide 19

Slide 19 text

•Reference Specification?

Slide 20

Slide 20 text

The standard spec •Reference Specification?

Slide 21

Slide 21 text

The standard spec Buggy Implementation •Reference Specification?

Slide 22

Slide 22 text

The standard spec Buggy Implementation "Extra" Features •Reference Specification?

Slide 23

Slide 23 text

The standard spec Buggy Implementation "Extra" Features "Be liberal in what you accept, and conservative in what you send"  Postel's Law "Accepted" Bugs •Reference Specification?

Slide 24

Slide 24 text

Where to Get a Grammar From? • Design Documentation?

Slide 25

Slide 25 text

Where to Get a Grammar From? • Design Documentation? What design documentation?

Slide 26

Slide 26 text

https://www.json.org

Slide 27

Slide 27 text

object { } { members } members pair pair , members pair string : value array [ ] [ elements ] elements value value , elements value string number object array true false null string " " " chars " chars char char chars char UNICODE \ [",\,CTRL] \" \\ \/ \b \f \n \r \t \u hex hex hex hex number int int frac int exp int frac exp int digit onenine digits - digit - onenine digits frac . digits exp e digits hex digit A - F a - f digits digit digit digits e e e+ e- E E+ E- https://www.json.org

Slide 28

Slide 28 text

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Where to Get a Grammar From? Hand-written parsers already encode the grammar

Slide 31

Slide 31 text

:= := '+' | '-' | := '*' | '/' | := '+' | '-' | '(' ')' | '.' | := | := [0-9] def start_parse(): parse_expr() def parse_expr(): parse_term() match lookahead(): case '+': consume(); parse_expr() case '-': consume(); parse_expr() default: pass def parse_term(): parse_factor() match lookahead(): case '*': consume(); parse_term() case '/': consume(); parse_term() default: pass

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Slide 39

Slide 39 text

Slide 40

Slide 40 text

Slide 41

Slide 41 text

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Slide 44

Slide 44 text

Slide 45

Slide 45 text

Slide 46

Slide 46 text

Slide 47

Slide 47 text

Slide 48

Slide 48 text

Slide 49

Slide 49 text

Slide 50

Slide 50 text

How to Extract This Grammar?

Slide 51

Slide 51 text

How to Extract This Grammar? • Inputs -> Dynamic Control Dependence trees

Slide 52

Slide 52 text

How to Extract This Grammar? • Inputs -> Dynamic Control Dependence trees • DCD Trees -> Context Free Grammar

Slide 53

Slide 53 text

Control Dependence Graph Statement B is control dependent on A if A determines whether B executes. def parse_csv(s,i): while s[i:]: if is_digit(s[i]): n,j = num(s[i:]) i = i+j else: comma(s[i]) i += 1

Slide 54

Slide 54 text

Slide 55

Slide 55 text

Slide 56

Slide 56 text

def parse_csv(s,i): while s[i:]: if is_digit(s[i]): n,j = num(s[i:]) i = i+j else: comma(s[i]) i += 1 CDG for parse_csv Dynamic Control Dependence Tree Each statement execution is represented as a separate node

Slide 57

Slide 57 text

Slide 58

Slide 58 text

def parse_csv(s,i): while s[i:]: if is_digit(s[i]): n,j = num(s[i:]) i = i+j else: comma(s[i]) i += 1 DCD Tree ~ Parse Tree •No tracking beyond input buﬀer •Characters are attached to nodes where they are accessed last "12," "12,"

Slide 59

Slide 59 text

def parse_csv(s,i): while s[i:]: if is_digit(s[i]): n,j = num(s[i:]) i = i+j else: comma(s[i]) i += 1 '1' '2' ',' DCD Tree ~ Parse Tree •No tracking beyond input buﬀer •Characters are attached to nodes where they are accessed last "12," "12,"

Slide 60

Slide 60 text

def is_digit(i): return i in '0123456789' def parse_num(s,i): n = '' while s[i:] and is_digit(s[i]): n += s[i] i = i +1 return i,n def parse_paren(s, i): assert s[i] == '(' i, v = parse_expr(s, i+1) if s[i:] == '': raise Ex(s, i) assert s[i] == ')' return i+1, v def parse_expr(s, i = 0): expr, is_op = [], True while s[i:]: c = s[i] if isdigit(c): if not is_op: raise Ex(s,i) i,num = parse_num(s,i) expr.append(num) is_op = False elif c in ['+', '-', '*', '/']: if is_op: raise Ex(s,i) expr.append(c) is_op, i = True, i + 1 elif c == '(': if not is_op: raise Ex(s,i) i, cexpr = parse_paren(s, i) expr.append(cexpr) is_op = False elif c == ')': break else: raise Ex(s,i) if is_op: raise Ex(s,i) return i, expr 9+3/4 Parse tree for parse_expr('9+3/4')

Slide 61

Slide 61 text

Slide 62

Slide 62 text

Slide 63

Slide 63 text

Slide 64

Slide 64 text

3 * (9 + 1)

Slide 65

Slide 65 text

3 * (9 + 1)

Slide 66

Slide 66 text

(9 + 1) * 3 3 * (9 + 1)

Slide 67

Slide 67 text

(9 + 1) * 3 3 * (9 + 1)

Slide 68

Slide 68 text

3 * (9 + 1)

Slide 69

Slide 69 text

3 * (9 + 1)

Slide 70

Slide 70 text

9 + 1 3 * (9 + 1)

Slide 71

Slide 71 text

9 + 1 3 * (9 + 1)

Slide 72

Slide 72 text

3 * (9 + 1)

Slide 73

Slide 73 text

3 * (9 + 1)

Slide 74

Slide 74 text

3 (9 + 1) * 3 * (9 + 1)

Slide 75

Slide 75 text

3 (9 + 1) * 3 * (9 + 1)

Slide 76

Slide 76 text

3*(1) 1

Slide 77

Slide 77 text

3*(1) 1

Slide 78

Slide 78 text

3*(1) 1 :=

Slide 79

Slide 79 text

3*(1) 1 := :=

Slide 80

Slide 80 text

:= | | | := := | := := '3' | '1' := '(' ')' := := '*'

Slide 81

Slide 81 text

:= := | := | | | := := | := := '3' | '1' := '(' ')' := := '*'

Slide 82

Slide 82 text

Generalization of Lexical Tokens

Slide 83

Slide 83 text

Widening Lexical Tokens with Active Learning := 19458790 | 809451 | 243 | 48095284094435 := 1.0043 | 34.343 | 232.232 | 2988.343 := "" "dfa 39&*(" "0989" "0._3"

Slide 84

Slide 84 text

Widening Lexical Tokens with Active Learning := 19458790 | 809451 | 243 | 48095284094435 := 1.0043 | 34.343 | 232.232 | 2988.343 := "" "dfa 39&*(" "0989" "0._3" := [0-9]+ := [0-9]+[.][0-9]+ := "[:alphanum:]+"

Slide 85

Slide 85 text

Slide 86

Slide 86 text

::= ::= '"' | '[' | '{' | | 'true' | 'false' | 'null' ::= + | + 'e' + ::= '+' | '-' | '.' | [0-9] | 'E' | 'e' ::= * '"' ::= ']' | (',')* ']' | ( ',' )+ (',' )* ']' ::= '}' | ( '"' ':' ',' )* '"' ':' '}' ::= ' ' | '!' | '#' | '$' | '%' | '&' | ''' | '*' | '+' | '-' | ',' | '.' | '/' | ':' | ';' | '<' | '=' | '>' | '?' | '@' | '[' | ']' | '^' | '_', ''',| '{' | '|' | '}' | '~' | '[A-Za-z0-9]' | '\' ::= '"' | '/' | 'b' | 'f' | 'n' | 'r' | 't' # parse out a key/value pair elif c == '"': key = _from_json_string(stm) stm.skipspaces() c = stm.next() if c != ':': raise JSONError(E_COLON, stm, stm.pos) stm.skipspaces() val = _from_json_raw(stm) result[key] = val expect_key = 0 continue raise JSONError(E_MALF, stm, stm.pos) def _from_json_raw(stm): while True: stm.skipspaces() c = stm.peek() if c == '"': return _from_json_string(stm) elif c == '{': return _from_json_dict(stm) elif c == '[': return _from_json_list(stm) elif c == 't': return _from_json_fixed(stm, 'true', True, E_BOOL) elif c == 'f': return _from_json_fixed(stm, 'false', False, E_BOOL) elif c == 'n': return _from_json_fixed(stm, 'null', None, E_NULL) elif c in NUMSTART: return _from_json_number(stm) raise JSONError(E_MALF, stm, stm.pos) def from_json(data): stm = JSONStream(data) return _from_json_raw(stm) microjson.py Recovered JSON grammar

Slide 87

Slide 87 text

Evaluation

Slide 88

Slide 88 text

• Languages: C, Python • Style: Adhoc, Text book, Parser Combinators • Evaluation: Precision and Recall

Slide 89

Slide 89 text

Subjects Language Grammar Style calc.py Python CFG textbook parser mathexpr.py Python CFG textbook OO cgidecode.py Python RG automata urlparse.py Python RG ad hoc microjson.py Python CFG optimized parser parseclisp.py Python CFG parser combinator jsonparser.c C CFG optimized parser tiny.c C CFG lexer + parser mjs.c C CFG lexer + parser Evaluation: Subjects

Slide 90

Slide 90 text

Evaluation: Recall Subjects AUTOGRAM Mimid calc.py 36.5 % 100.0 % mathexpr.py 30.3 % 87.5 % cgidecode.py 47.8 % 100.0 % urlparse.py 100.0 % 100.0 % microjson.py 53.8 % 98.7 % parseclisp.py 100.0 % 99.3 % jsonparser.c n/a 100.0 % tiny.c n/a 100.0 % mjs.c n/a 95.4 % Inputs generated by inferred grammar that were accepted by the program

Slide 91

Slide 91 text

Subjects AUTOGRAM Mimid calc.py 0.0 % 100.0 % mathexpr.py 0.0 % 92.7 % cgidecode.py 35.1 % 100.0 % urlparse.py 100.0 % 96.4 % microjson.py 0.0 % 93.0 % parseclisp.py 37.6 % 80.6 % jsonparser.c n/a 83.8 % tiny.c n/a 92.8 % mjs.c n/a 95.9 % Inputs generated by golden grammar that were accepted by the inferred grammar parser Evaluation: Precision

Slide 92

Slide 92 text

• Grammar Generation with light weight instrumentation • Can be applied to multiple styles of parsers • Generates accurate and readable grammars • Replication package is available as s Jupyter notebook Mimid