Types of testing
• Unit
• Integration
• System
• …
Slide 7
Slide 7 text
Types of testing
• Unit
• Integration
• System
• …
• Destructive
Slide 8
Slide 8 text
Types of testing
• Unit
• Integration
• System
• …
• Destructive
• Sanity "
Slide 9
Slide 9 text
Types of testing data
Slide 10
Slide 10 text
Types of testing data
• Well-known normal data
Slide 11
Slide 11 text
Types of testing data
• Well-known normal data
• Well-known corner cases
Slide 12
Slide 12 text
Types of testing data
• Well-known normal data
• Well-known corner cases
• Well-known fixed bugs
Slide 13
Slide 13 text
Types of testing data
• Well-known normal data
• Well-known corner cases
• Well-known fixed bugs
• Unknown random data
Slide 14
Slide 14 text
Testing with random data
Slide 15
Slide 15 text
Testing with random data
• Cheap
• Fast
• Not biased
Slide 16
Slide 16 text
• May not be deterministic
• Requires good spec
(«it’s a feature, not a bug»)
• Requires a lot of brute force
Testing with random data
• Cheap
• Fast
• Not biased
Slide 17
Slide 17 text
Testing with random data
• testing/quick
• github.com/google/gofuzz
• github.com/leanovate/gopter
(Go Property Tester)
Slide 18
Slide 18 text
No examples yet
We can do better
Slide 19
Slide 19 text
Quiz question!
How «go test -cover» works?
Slide 20
Slide 20 text
How can we do better?
Slide 21
Slide 21 text
• We can rewrite our code just like cover tool does
How can we do better?
Slide 22
Slide 22 text
• We can rewrite our code just like cover tool does
• We can use code coverage information to reduce
input while keeping the same coverage, then
mutate it until we get new coverage
How can we do better?
In a nutshell
Instrument program
Collect initial corpus, reduce it
for {
Apply random mutation to input
Check coverage
If gives new coverage, try to reduce input, then add to corpus
}
Slide 25
Slide 25 text
github.com/dvyukov/go-fuzz
Slide 26
Slide 26 text
Get it
Slide 27
Slide 27 text
Get it
• $ go get github.com/dvyukov/go-fuzz/go-fuzz
Slide 28
Slide 28 text
Get it
• $ go get github.com/dvyukov/go-fuzz/go-fuzz
•
☕
Slide 29
Slide 29 text
Get it
• $ go get github.com/dvyukov/go-fuzz/go-fuzz
•
☕
• $ go get github.com/dvyukov/go-fuzz/go-fuzz-build
Fuzz: basic rules
• Return 1 if data happens to be correct
Slide 33
Slide 33 text
Fuzz: basic rules
• Return 1 if data happens to be correct
• Panic in case of unexpected error
Slide 34
Slide 34 text
Fuzz: basic rules
• Return 1 if data happens to be correct
• Panic in case of unexpected error
• Return 0 otherwise
Slide 35
Slide 35 text
Fuzz: basic rules
• Return 1 if data happens to be correct
• Panic in case of unexpected error
• Return 0 otherwise
• No external state!
Slide 36
Slide 36 text
Fuzz: basic rules
• Return 1 if data happens to be correct
• Panic in case of unexpected error
• Return 0 otherwise
• No external state!
• Log only before panic
Slide 37
Slide 37 text
Fuzz: basic rules
• Return 1 if data happens to be correct
• Panic in case of unexpected error
• Return 0 otherwise
• No external state!
• Log only before panic
• That’s ok for function to hang or eat a lot of memory
Slide 38
Slide 38 text
Fuzz: basic rules
• Return 1 if data happens to be correct
• Panic in case of unexpected error
• Return 0 otherwise
• No external state!
• Log only before panic
• That’s ok for function to hang or eat a lot of memory
• Use build tag
Slide 39
Slide 39 text
Initial corpus collection
Slide 40
Slide 40 text
Initial corpus collection
• Small and diverse
Slide 41
Slide 41 text
Initial corpus collection
• Small and diverse
• Diversity is more important
Slide 42
Slide 42 text
Initial corpus collection
• Small and diverse
• Diversity is more important
• Use data from unit tests!
Slide 43
Slide 43 text
Run it
Slide 44
Slide 44 text
Run it
• $ go-fuzz-build package
Slide 45
Slide 45 text
Run it
• $ go-fuzz-build package
• Fill workdir/corpus with files
Slide 46
Slide 46 text
Run it
• $ go-fuzz-build package
• Fill workdir/corpus with files
• $ go-fuzz -bin=package-fuzz.zip -workdir=workdir
Slide 47
Slide 47 text
Run it
• $ go-fuzz-build package
• Fill workdir/corpus with files
• $ go-fuzz -bin=package-fuzz.zip -workdir=workdir
• See results in workdir/crashes
Slide 48
Slide 48 text
Run it
• $ go-fuzz-build package
• Fill workdir/corpus with files
• $ go-fuzz -bin=package-fuzz.zip -workdir=workdir
• See results in workdir/crashes
• Consider adding workdir/corpus to VCS
Fuzz: custom types
• When fuzzer gives you lemons bytes,
make your types
Slide 56
Slide 56 text
Fuzz: custom types
• When fuzzer gives you lemons bytes,
make your types
• Custom decode function is often better that generic
Slide 57
Slide 57 text
Fuzz: custom types
• When fuzzer gives you lemons bytes,
make your types
• Custom decode function is often better that generic
• If you have checksums, append them yourself
Slide 58
Slide 58 text
if len(data) != 8 {
return 0
}
i1, err1 := strconv.Atoi(string(data[:4]))
i2, err2 := strconv.Atoi(string(data[4:]))
if err1 != nil || err2 != nil {
return 0
}
// work with i1 and i2
Slide 59
Slide 59 text
A lot of fancy stuff
• Versifier – reverse-engineers text protocol and
learns its structure, makes structural mutations
• Sonar – could help with stuff like checksums, but
now public API yet