Abstracting Failure Inducing Inputs

Slide 1

Slide 1 text

Abstracting Failure Inducing Inputs Rahul Gopinath Alexander Kampmann Nikolas Havrikov Ezekiel Soremekun Andreas Zeller CISPA Helmholtz Center for Information Security

Slide 2

Slide 2 text

Abstracting Failure Inducing Inputs Rahul Gopinath Alexander Kampmann Nikolas Havrikov Ezekiel Soremekun Andreas Zeller CISPA Helmholtz Center for Information Security

Slide 3

Slide 3 text

3 (1 + 2 + 334) ✓ Program

Slide 4

Slide 4 text

4 (1 + 2 + 334) Program (( 442 / 3 )) - 1 ✘

Slide 5

Slide 5 text

5 (1 + 2 + 334) Program (( 442 / 3 )) - 1 ✓ 2 -( 19 - 34 ) + 9 - 7

Slide 6

Slide 6 text

6 (1 + 2 + 334) Program (( 442 / 3 )) - 1 2 -( 19 - 34 ) + 9 - 7 (1) + ((3 + 1 / 334)) + 2 ✘

Slide 7

Slide 7 text

7 (1 + 2 + 334) (( 442 / 3 )) - 1 2 -( 19 - 34 ) + 9 - 7 (1) + ((3 + 1 / 334)) + 2 if '((' in input and '))' in input: raise Exception() Program

Slide 8

Slide 8 text

8 8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 - 5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.6 - --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +--(+- +-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(--+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) + 8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 - 5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.6 - --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +--(+- +-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(--+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) * - +5 + 7.513)))) - (+1 / ++((-84)))))))) * ++5 / +-(--2 - -++-9.0)))) / 5 * --++090 + * -+5 + 7.513)))) - (+1 / ++((-84)))))))) * 8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 - 5 - 6)) * (-(a- +(((+(4))))) - ++4) / +(-+---((5.6 - --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +--(+-+-7 * (-0 * (+ (((((2)) + 8 - 3 - ++9.0 + ---(--+7 / (1 / ++ +6.37) + (1) / 482) / +++-+0)))) * -+5 + 7.513)))) - (+1 / ++((-84)))))))) * ++5 / +-(--2 - -++-9.0)))) / 5 * --++090 ++5 / +-(--2 - -+ +-9.0)))) / 5 * --++090 Program ✘

Slide 9

Slide 9 text

Why did my program fail?

Slide 10

Slide 10 text

Delta Debugging 8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 - 5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.6 - --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +--(+- +-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(--+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) + 8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 - 5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.6 - --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +--(+- +-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(--+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) * - +5 + 7.513)))) - (+1 / ++((-84)))))))) * ++5 / +-(--2 - -++-9.0)))) / 5 * --++090 + * -+5 + 7.513)))) - (+1 / ++((-84)))))))) * 8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 - 5 - 6)) * (-(a- +(((+(4))))) - ++4) / +(-+---((5.6 - --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +--(+-+-7 * (-0 * (+ (((((2)) + 8 - 3 - ++9.0 + ---(--+7 / (1 / ++ +6.37) + (1) / 482) / +++-+0)))) * -+5 + 7.513)))) - (+1 / ++((-84)))))))) * ++5 / +-(--2 - -++-9.0)))) / 5 * --++090 ++5 / +-(--2 - -+ +-9.0)))) / 5 * --++090 Program

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Context Free Grammar Structured Inputs See also: "Learning Input Tokens for Effective Fuzzing" ISSTA '20 https://www.slideshare.net/BjrnMathis/lfuzzer-learning-input-tokens-for-effective-fuzzing-237085021

Slide 18

Slide 18 text

8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 - 5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.6 - --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +--(+- +-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(--+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) + 8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 - 5 - 6)) * (-(a-+(((+(4))))) - ++4) / +(-+---((5.6 - --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +--(+- +-7 * (-0 * (+(((((2)) + 8 - 3 - ++9.0 + ---(--+7 / (1 / +++6.37) + (1) / 482) / +++-+0)))) * - +5 + 7.513)))) - (+1 / ++((-84)))))))) * ++5 / +-(--2 - -++-9.0)))) / 5 * --++090 + * -+5 + 7.513)))) - (+1 / ++((-84)))))))) * 8.2 - 27 - -9 / +((+9 * --2 + --+-+-((-1 * +(8 - 5 - 6)) * (-(a- +(((+(4))))) - ++4) / +(-+---((5.6 - --(3 * -1.8 * +(6 * +-(((-(-6) * ---+6)) / +--(+-+-7 * (-0 * (+ (((((2)) + 8 - 3 - ++9.0 + ---(--+7 / (1 / ++ +6.37) + (1) / 482) / +++-+0)))) * -+5 + 7.513)))) - (+1 / ++((-84)))))))) * ++5 / +-(--2 - -++-9.0)))) / 5 * --++090 ++5 / +-(--2 - -+ +-9.0)))) / 5 * --++090 Interpreter ✘ Reproduced the failure Structured Inputs SYNTAX CHECK

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Slide 21

Slide 21 text

SYNTAX ERROR

Slide 22

Slide 22 text

Solution: Work on the Parse Tree

Slide 23

Slide 23 text

23 := := ' + ' | ' - ' | := ' * ' | ' / ' | := '+' | '-' | '(' ')' | '.' | := | := [0-9] 1 + (( 2 * 3 / 4 ))

Slide 24

Slide 24 text

24 := := ' + ' | ' - ' | := ' * ' | ' / ' | := '+' | '-' | '(' ')' | '.' | := | := [0-9] 1 + (( 2 * 3 / 4 ))

Slide 25

Slide 25 text

25 := := ' + ' | ' - ' | := ' * ' | ' / ' | := '+' | '-' | '(' ')' | '.' | := | := [0-9] 1 + (( 2 * 3 / 4 )) ✘

Slide 26

Slide 26 text

26 := := ' + ' | ' - ' | := ' * ' | ' / ' | := '+' | '-' | '(' ')' | '.' | := | := [0-9] (( 2 * 3 / 4 )) ✘

Slide 27

Slide 27 text

27 := := ' + ' | ' - ' | := ' * ' | ' / ' | := '+' | '-' | '(' ')' | '.' | := | := [0-9] (( 3 / 4 )) ✘

Slide 28

Slide 28 text

28 := := ' + ' | ' - ' | := ' * ' | ' / ' | := '+' | '-' | '(' ')' | '.' | := | := [0-9] (( 4 )) ✘

Slide 29

Slide 29 text

29 ( 1 + (( 2 * 3 / 4 )) ( 4 ))

Slide 30

Slide 30 text

30 ( 1 + (( 2 * 3 / 4 )) (4)) What caused this failure? Does the failure occur in other inputs?

Slide 31

Slide 31 text

31 ( 1 + (( 2 * 3 / 4 )) (4)) 4444 ? ()() ? ((-4)) ? ((29)) ? ((v)) ? +++1 ?

Slide 32

Slide 32 text

32 var A = class extends (class {}){}; Issue 2937 from Closure

Slide 33

Slide 33 text

33 { while ((l_0)){ if ((l_0)) { break;;var l_0; continue }0 } } Issue 2842 from Closure

Slide 34

Slide 34 text

34 var {baz:{} = baz => {}} = baz => {}; Issue 385 from Rhino

Slide 35

Slide 35 text

35 const [y,y] = []; Issue 386 from Rhino

Slide 36

Slide 36 text

36 Abstraction with DDSet

Slide 37

Slide 37 text

37 ( ( 4 ) ) := := ' + ' | ' - ' | := ' * ' | ' / ' | := '+' | '-' | '(' ')' | '.' | := | := [0-9]

Slide 38

Slide 38 text

38 ( ( 4 ) ) := := ' + ' | ' - ' | := ' * ' | ' / ' | := '+' | '-' | '(' ')' | '.' | := | := [0-9] ✓ Did not reproduce the failure 1 * (2 - 3)

Slide 39

Slide 39 text

39 ( ( 4 ) ) := := ' + ' | ' - ' | := ' * ' | ' / ' | := '+' | '-' | '(' ')' | '.' | := | := [0-9]

Slide 40

Slide 40 text

40 ( ( 4 ) ) := := ' + ' | ' - ' | := ' * ' | ' / ' | := '+' | '-' | '(' ')' | '.' | := | := [0-9] c

Slide 41

Slide 41 text

41 ( ( 4 ) ) := := ' + ' | ' - ' | := ' * ' | ' / ' | := '+' | '-' | '(' ')' | '.' | := | := [0-9] c ✓ Did not reproduce the failure 1 + 3 + 4

Slide 42

Slide 42 text

42 ( ( 4 ) ) := := ' + ' | ' - ' | := ' * ' | ' / ' | := '+' | '-' | '(' ')' | '.' | := | := [0-9] c c

Slide 43

Slide 43 text

43 3 * 4 := := ' + ' | ' - ' | := ' * ' | ' / ' | := '+' | '-' | '(' ')' | '.' | := | := [0-9] c c ✓ Did not reproduce the failure

Slide 44

Slide 44 text

44 ( ( 4 ) ) := := ' + ' | ' - ' | := ' * ' | ' / ' | := '+' | '-' | '(' ')' | '.' | := | := [0-9] c c c c c c c

Slide 45

Slide 45 text

45 ( ( 1 - 2 ) ) := := ' + ' | ' - ' | := ' * ' | ' / ' | := '+' | '-' | '(' ')' | '.' | := | := [0-9] c c c c c c c ✘ reproduced the failure ( ( 1 - 2 ) )

Slide 46

Slide 46 text

46 ( ( 1 - 2 ) ) c c c c c c c ✘ ( ( 1 - 2 ) )

Slide 47

Slide 47 text

47 ( ( 1 - 2 ) ) c c c c c c c ✘ ( ( 1 - 2 ) ) ✘ ( ( 2 * 3 + 4 ) )

Slide 48

Slide 48 text

48 ( ( 1 - 2 ) ) c c c c c c c ✘ ( ( 1 - 2 ) ) ✘ ( ( 2 * 3 + 4 ) ) ✘ ( ( - 2 / 1 ) )

Slide 49

Slide 49 text

49 ( ( 1 - 2 ) ) c c c c c c c ✘ ( ( 1 - 2 ) ) ✘ ( ( 2 * 3 + 4 ) ) ✘ ( ( - 2 / 1 ) ) ✘ ( ( 98 - 0 ) )

Slide 50

Slide 50 text

) ( ( ) ( ( ) 4 ) 50 ( ( 4 ) ) c c c c c c c A

Slide 51

Slide 51 text

) ( ( ) ( ( ) 4 ) 51 ( ( 4 ) ) c c c c c c c A

Slide 52

Slide 52 text

52 ( ( 4 ) ) c c c c c c c A ( ( ) ) ( ( ) ) 4 Minimized Input Abstract Failure Inducing Input ((1 + 2)) ((23 * 3 - 34)) ((344- 4 + (223))) (((1) - 3 * 773 + (-22 + 1))) ((1798 - 889 / (333-1) * 2 / 3 + 1)) ((34 + 4 --334 + (334 - (22) + 919 * 0 + 1)) ((98435747+ 88 + (((0))) + (1) - 1 * 7 / 4 * 889 - 2)) ((8 + ((8)) + --1 + 11223 / 344 - 39 + (1) - 456 + 134 / 45 )) ((437 + 8 - 1 * ((9 + 1) - 1 + 99111948 + 3 --1 + (112) - 2 + 445) + 0)) ((74 + 334 + ((178 - 88 / (3393-1) * 1002 / 3 + 1+ 3439)) * 223 - 1233 + 334672 (( 2 * 9 - (1798 - 889 / (333-1) * 2 / 3 + 100012 + 3434392 + 234 ----6 * 1798 - 889 / (333 ((778 - (((1) - 3 * 773 + (-22 + 1) * (4545) - 23 - (2) * 773 + (-22 + 1) / 3434 + ---1 + 1 / 34343 + 112 ((349 + (((1) - 3 * 3 + (-22 + 1) ((+ (-22 + 1) * (4545) - 23 - (2) * 773 + (-22 + 1) / 3434 + ---1 + 1 / 34343 + 1123 ((8 + ((8)) + --1 + / 1 - 39 + (1) - 456 + 134 / 45 ))(((1) - 2334 + (((1) - 3 * 773 + (-22 + 1) * (2) - 23 - (2) * 773 + (-22 + 1) / 3 ((74 + 3 + ((178 - 88 / (3393-1) * 1002 / 3 + 1+ 3439)) * - 1233 + 334672)) ((8 + ((8)) + --1 + / 344 - 39 + (1) - 456 + 134 / 45 ))(((1) - 3 * 77 ((1+ 33+ 24343433 +23343 - ((74 + 334 + ((178 - 88 / (3393-1) * 1002 / 3 + 1+ 3439)) * - 1233 + 334672)) ((8 + ((8)) + --1 + / 344 - 39 + (1) - 456 + 134 / ✘ ✘

Slide 53

Slide 53 text

= class extends (class {}){} 53 var A = class extends (class {}){}; Issue 2937 from Closure

Slide 54

Slide 54 text

54 var A = class extends (class {}){}; Issue 2937 from Closure = class extends (class {}){}

Slide 55

Slide 55 text

var {baz:{} = baz => {}} = 55 var {baz:{} = baz => {}} = baz => {}; Issue 385 from Rhino

Slide 56

Slide 56 text

56 var {baz:{} = baz => {}} = baz => {}; Issue 385 from Rhino var {baz:{} = baz => {}} =

Slide 57

Slide 57 text

const [y,y] = []; 57 const [y,y] = []; Issue 386 from Rhino

Slide 58

Slide 58 text

58 const [y,y] = []; Issue 386 from Rhino const [y,y] = [];

Slide 59

Slide 59 text

v = 0; v = v Co-varying Fragments

Slide 60

Slide 60 text

v = 0; v = v Co-varying Fragments

Slide 61

Slide 61 text

v = 0; v = v

Slide 62

Slide 62 text

v = 0; v = v x = 0; v = v ✓

Slide 63

Slide 63 text

v = 0; v = v v = 0; r = v ✓

Slide 64

Slide 64 text

v = 0; v = v v = 0; v = 0 ✓

Slide 65

Slide 65 text

v = 0; v = v • Identify matching nonterminals of concrete nodes • Modify them together

Slide 66

Slide 66 text

v = 0; v = v z = 0; z = z ✘ • Identify matching nonterminals of concrete nodes • Modify them together

Slide 67

Slide 67 text

v = 0; v = v z = 0; z = z ✘ • Identify matching nonterminals of concrete nodes • Modify them together p = 0; p = p ✘

Slide 68

Slide 68 text

v = 0; v = v z = 0; z = z ✘ • Identify matching nonterminals of concrete nodes • Modify them together p = 0; p = p ✘ c = 0; c = c ✘

Slide 69

Slide 69 text

v = 0; v = v = ; = V1 V1 V1

Slide 70

Slide 70 text

const [y,y] = []; Issue 386 from Rhino

Slide 71

Slide 71 text

const [y,y] = []; Issue 386 from Rhino const [<$Id1>,<$Id1>] = []

Slide 72

Slide 72 text

72 var {baz:{} = baz => {}} = baz => {}; Issue 385 from Rhino

Slide 73

Slide 73 text

73 var {baz:{} = baz => {}} = baz => {}; Issue 385 from Rhino var {<$Id1>:{} = <$Id1> => {}} ;

Slide 74

Slide 74 text

74 {while ((l_0)){ if ((l_0)) {break;;var l_0; continue }0}} Issue 2842 from Closure

Slide 75

Slide 75 text

75 {while ((l_0)){ if ((l_0)) {break;;var l_0; continue }0}} Issue 2842 from Closure {while ((<$Id1>)){ if ((<$Id1>)) {break;;var <$Id1>; continue }0}}

Slide 76

Slide 76 text

Program %Valid %Fail lua-5.3.5 4 100.0 100.0 clj-2092 100.0 100.0 clj-2345 100.0 100.0 clj-2450 62.0 100.0 clj-2473 40.0 100.0 clj-2518 100.0 100.0 clj-2521 100.0 100.0 closure.1978 76.0 100.0 closure.2808 100.0 100.0 closure.2842 100.0 99.0 closure.2937 36.0 100.0 closure.3178 57.0 100.0 closure.3379 84.0 100.0 rhino 385 49.0 100.0 rhino 386 100.0 100.0 find 07b941b1 100.0 100.0 find 93623752 100.0 100.0 find c8491c11 100.0 100.0 find dbcb10e9 100.0 100.0 grep 3c3bdace 100.0 100.0 grep 54d55bba 100.0 100.0 grep 9c45c193 100.0 100.0 Mean 86.54 100.0 • Lua • lua-5.3.5 (1 bug) • Javascript • rhino-1.7.7.2 (2 bugs) • closure 20151216 (1 bug) • closure 20171203 (3 bugs) • closure 20200101 (2 bugs) • Clojure • clojure-1.11.0 (6 bug) • Unix Utilities (dbgbench) • find (4 bugs) • grep (3 bugs) Experimental Results

Slide 77

Slide 77 text

Where do the grammars come from? https://rahul.gopinath.org/post/2020/07/15/ddset/ https://github.com/vrthra/ddset Stay tuned for our FSE 2020 Paper Mining Input Grammars from Dynamic Control Flow

Slide 78

Slide 78 text

Future... Algebra of Behavior Inducing Patterns Grammar refinement: generate(Grefined): 123 + ((34 + 5)) - 244 After fixing a failure, I want to produce numerous new inputs that • Induces the same failure (R1) • Covers what I just checked in (R2) • Does not go through validation (R3) • ... Grefined = R1 & R2 & !R3 ...

Slide 79

Slide 79 text

https://rahul.gopinath.org/post/2020/07/15/ddset/ https://github.com/vrthra/ddset

Slide 80

Slide 80 text

80 https://rahul.gopinath.org/post/2020/07/15/ddset/ https://github.com/vrthra/ddset