Slide 1

Slide 1 text

What the fuzz? Alexey Palazhchenko

Slide 2

Slide 2 text

Types of testing

Slide 3

Slide 3 text

Types of testing • Unit

Slide 4

Slide 4 text

Types of testing • Unit • Integration

Slide 5

Slide 5 text

Types of testing • Unit • Integration • System

Slide 6

Slide 6 text

Types of testing • Unit • Integration • System • …

Slide 7

Slide 7 text

Types of testing • Unit • Integration • System • … • Destructive

Slide 8

Slide 8 text

Types of testing • Unit • Integration • System • … • Destructive • Sanity "

Slide 9

Slide 9 text

Types of testing data

Slide 10

Slide 10 text

Types of testing data • Well-known normal data

Slide 11

Slide 11 text

Types of testing data • Well-known normal data • Well-known corner cases

Slide 12

Slide 12 text

Types of testing data • Well-known normal data • Well-known corner cases • Well-known fixed bugs

Slide 13

Slide 13 text

Types of testing data • Well-known normal data • Well-known corner cases • Well-known fixed bugs • Unknown random data

Slide 14

Slide 14 text

Testing with random data

Slide 15

Slide 15 text

Testing with random data • Cheap • Fast • Not biased

Slide 16

Slide 16 text

• May not be deterministic • Requires good spec
 («it’s a feature, not a bug») • Requires a lot of brute force Testing with random data • Cheap • Fast • Not biased

Slide 17

Slide 17 text

Testing with random data • testing/quick • github.com/google/gofuzz • github.com/leanovate/gopter
 (Go Property Tester)

Slide 18

Slide 18 text

No examples yet We can do better

Slide 19

Slide 19 text

Quiz question! How «go test -cover» works?

Slide 20

Slide 20 text

How can we do better?

Slide 21

Slide 21 text

• We can rewrite our code just like cover tool does How can we do better?

Slide 22

Slide 22 text

• We can rewrite our code just like cover tool does • We can use code coverage information to reduce input while keeping the same coverage, then mutate it until we get new coverage How can we do better?

Slide 23

Slide 23 text

• We can rewrite our code just like cover tool does • We can use code coverage information to reduce input while keeping the same coverage, then mutate it until we get new coverage • We can use MORE POWER!!1 How can we do better? © Randall Munroe https://what-if.xkcd.com/13/

Slide 24

Slide 24 text

In a nutshell Instrument program Collect initial corpus, reduce it for { Apply random mutation to input Check coverage If gives new coverage, try to reduce input, then add to corpus }

Slide 25

Slide 25 text

github.com/dvyukov/go-fuzz

Slide 26

Slide 26 text

Get it

Slide 27

Slide 27 text

Get it • $ go get github.com/dvyukov/go-fuzz/go-fuzz

Slide 28

Slide 28 text

Get it • $ go get github.com/dvyukov/go-fuzz/go-fuzz • ☕

Slide 29

Slide 29 text

Get it • $ go get github.com/dvyukov/go-fuzz/go-fuzz • ☕ • $ go get github.com/dvyukov/go-fuzz/go-fuzz-build

Slide 30

Slide 30 text

// +build gofuzz func Fuzz(data []byte) int { _, err := handle(data) if err != nil { return 0 } return 1 }

Slide 31

Slide 31 text

Fuzz: basic rules

Slide 32

Slide 32 text

Fuzz: basic rules • Return 1 if data happens to be correct

Slide 33

Slide 33 text

Fuzz: basic rules • Return 1 if data happens to be correct • Panic in case of unexpected error

Slide 34

Slide 34 text

Fuzz: basic rules • Return 1 if data happens to be correct • Panic in case of unexpected error • Return 0 otherwise

Slide 35

Slide 35 text

Fuzz: basic rules • Return 1 if data happens to be correct • Panic in case of unexpected error • Return 0 otherwise • No external state!

Slide 36

Slide 36 text

Fuzz: basic rules • Return 1 if data happens to be correct • Panic in case of unexpected error • Return 0 otherwise • No external state! • Log only before panic

Slide 37

Slide 37 text

Fuzz: basic rules • Return 1 if data happens to be correct • Panic in case of unexpected error • Return 0 otherwise • No external state! • Log only before panic • That’s ok for function to hang or eat a lot of memory

Slide 38

Slide 38 text

Fuzz: basic rules • Return 1 if data happens to be correct • Panic in case of unexpected error • Return 0 otherwise • No external state! • Log only before panic • That’s ok for function to hang or eat a lot of memory • Use build tag

Slide 39

Slide 39 text

Initial corpus collection

Slide 40

Slide 40 text

Initial corpus collection • Small and diverse

Slide 41

Slide 41 text

Initial corpus collection • Small and diverse • Diversity is more important

Slide 42

Slide 42 text

Initial corpus collection • Small and diverse • Diversity is more important • Use data from unit tests!

Slide 43

Slide 43 text

Run it

Slide 44

Slide 44 text

Run it • $ go-fuzz-build package

Slide 45

Slide 45 text

Run it • $ go-fuzz-build package • Fill workdir/corpus with files

Slide 46

Slide 46 text

Run it • $ go-fuzz-build package • Fill workdir/corpus with files • $ go-fuzz -bin=package-fuzz.zip -workdir=workdir

Slide 47

Slide 47 text

Run it • $ go-fuzz-build package • Fill workdir/corpus with files • $ go-fuzz -bin=package-fuzz.zip -workdir=workdir • See results in workdir/crashes

Slide 48

Slide 48 text

Run it • $ go-fuzz-build package • Fill workdir/corpus with files • $ go-fuzz -bin=package-fuzz.zip -workdir=workdir • See results in workdir/crashes • Consider adding workdir/corpus to VCS

Slide 49

Slide 49 text

Fuzz: logic checks

Slide 50

Slide 50 text

Fuzz: logic checks • Decode, encode, compare bytes

Slide 51

Slide 51 text

Fuzz: logic checks • Decode, encode, compare bytes • Decode, encode, decode, compare objects

Slide 52

Slide 52 text

Fuzz: logic checks • Decode, encode, compare bytes • Decode, encode, decode, compare objects • Cross-check with different implementations
 (simple/slow vs fast/sophisticated)

Slide 53

Slide 53 text

res, err := decode(data) if err != nil { return 0 } b, err := encode(res) if err != nil { panic(err) } if !reflect.DeepEqual(b, res) { panic(b) } return 1

Slide 54

Slide 54 text

Fuzz: custom types

Slide 55

Slide 55 text

Fuzz: custom types • When fuzzer gives you lemons bytes,
 make your types

Slide 56

Slide 56 text

Fuzz: custom types • When fuzzer gives you lemons bytes,
 make your types • Custom decode function is often better that generic

Slide 57

Slide 57 text

Fuzz: custom types • When fuzzer gives you lemons bytes,
 make your types • Custom decode function is often better that generic • If you have checksums, append them yourself

Slide 58

Slide 58 text

if len(data) != 8 { return 0 } i1, err1 := strconv.Atoi(string(data[:4])) i2, err2 := strconv.Atoi(string(data[4:])) if err1 != nil || err2 != nil { return 0 } // work with i1 and i2

Slide 59

Slide 59 text

A lot of fancy stuff • Versifier – reverse-engineers text protocol and learns its structure, makes structural mutations • Sonar – could help with stuff like checksums, but now public API yet

Slide 60

Slide 60 text

but also…

Slide 61

Slide 61 text

MORE POWER!!1 • Reuses single process • Forks it for faster startup • A lot of concurrency • Distributed across machines © Randall Munroe https://what-if.xkcd.com/13/

Slide 62

Slide 62 text

Fuzz something today!

Slide 63

Slide 63 text

Fuzz something today! … oh, and fuzzing may be coming to go tool

Slide 64

Slide 64 text

Not only Go american fuzzy lop
 http://lcamtuf.coredump.cx/afl/
 (that’s one awesome logo)

Slide 65

Slide 65 text

Not only Go american fuzzy lop
 http://lcamtuf.coredump.cx/afl/
 (that’s one awesome logo)

Slide 66

Slide 66 text

?

Slide 67

Slide 67 text

• golang-ru.googlegroups.com • meetup.com/Golang-Moscow • 4gophers.ru/slack • GolangShow.com