What the fuzz?

Fuzz testing of Go programs.

Transcript

1. What the fuzz? Alexey Palazhchenko
2. Types of testing
3. Types of testing • Unit
4. Types of testing • Unit • Integration
5. Types of testing • Unit • Integration • System
6. Types of testing • Unit • Integration • System • …
7. Types of testing • Unit • Integration • System • … • Destructive
8. Types of testing • Unit • Integration • System • … • Destructive • Sanity "
9. Types of testing data
10. Types of testing data • Well-known normal data
11. Types of testing data • Well-known normal data • Well-known corner cases
12. Types of testing data • Well-known normal data • Well-known corner cases • Well-known fixed bugs
13. Types of testing data • Well-known normal data • Well-known corner cases • Well-known fixed bugs • Unknown random data
14. Testing with random data
15. Testing with random data • Cheap • Fast • Not biased
16. • May not be deterministic • Requires good spec
 («it’s a feature, not a bug») • Requires a lot of brute force Testing with random data • Cheap • Fast • Not biased
17. Testing with random data • testing/quick • github.com/google/gofuzz • github.com/leanovate/gopter
 (Go Property Tester)
18. No examples yet We can do better
19. Quiz question! How «go test -cover» works?
20. How can we do better?
21. • We can rewrite our code just like cover tool does How can we do better?
22. • We can rewrite our code just like cover tool does • We can use code coverage information to reduce input while keeping the same coverage, then mutate it until we get new coverage How can we do better?
23. • We can rewrite our code just like cover tool does • We can use code coverage information to reduce input while keeping the same coverage, then mutate it until we get new coverage • We can use MORE POWER!!1 How can we do better? © Randall Munroe https://what-if.xkcd.com/13/
24. In a nutshell Instrument program Collect initial corpus, reduce it for { Apply random mutation to input Check coverage If gives new coverage, try to reduce input, then add to corpus }
25. github.com/dvyukov/go-fuzz
26. Get it
27. Get it • $ go get github.com/dvyukov/go-fuzz/go-fuzz
28. Get it • $ go get github.com/dvyukov/go-fuzz/go-fuzz • ☕
29. Get it • $ go get github.com/dvyukov/go-fuzz/go-fuzz • ☕ • $ go get github.com/dvyukov/go-fuzz/go-fuzz-build
30. // +build gofuzz func Fuzz(data []byte) int { _, err := handle(data) if err != nil { return 0 } return 1 }
31. Fuzz: basic rules
32. Fuzz: basic rules • Return 1 if data happens to be correct
33. Fuzz: basic rules • Return 1 if data happens to be correct • Panic in case of unexpected error
34. Fuzz: basic rules • Return 1 if data happens to be correct • Panic in case of unexpected error • Return 0 otherwise
35. Fuzz: basic rules • Return 1 if data happens to be correct • Panic in case of unexpected error • Return 0 otherwise • No external state!
36. Fuzz: basic rules • Return 1 if data happens to be correct • Panic in case of unexpected error • Return 0 otherwise • No external state! • Log only before panic
37. Fuzz: basic rules • Return 1 if data happens to be correct • Panic in case of unexpected error • Return 0 otherwise • No external state! • Log only before panic • That’s ok for function to hang or eat a lot of memory
38. Fuzz: basic rules • Return 1 if data happens to be correct • Panic in case of unexpected error • Return 0 otherwise • No external state! • Log only before panic • That’s ok for function to hang or eat a lot of memory • Use build tag
39. Initial corpus collection
40. Initial corpus collection • Small and diverse
41. Initial corpus collection • Small and diverse • Diversity is more important
42. Initial corpus collection • Small and diverse • Diversity is more important • Use data from unit tests!
43. Run it
44. Run it • $ go-fuzz-build package
45. Run it • $ go-fuzz-build package • Fill workdir/corpus with files
46. Run it • $ go-fuzz-build package • Fill workdir/corpus with files • $ go-fuzz -bin=package-fuzz.zip -workdir=workdir
47. Run it • $ go-fuzz-build package • Fill workdir/corpus with files • $ go-fuzz -bin=package-fuzz.zip -workdir=workdir • See results in workdir/crashes
48. Run it • $ go-fuzz-build package • Fill workdir/corpus with files • $ go-fuzz -bin=package-fuzz.zip -workdir=workdir • See results in workdir/crashes • Consider adding workdir/corpus to VCS
49. Fuzz: logic checks
50. Fuzz: logic checks • Decode, encode, compare bytes
51. Fuzz: logic checks • Decode, encode, compare bytes • Decode, encode, decode, compare objects
52. Fuzz: logic checks • Decode, encode, compare bytes • Decode, encode, decode, compare objects • Cross-check with different implementations
 (simple/slow vs fast/sophisticated)
53. res, err := decode(data) if err != nil { return 0 } b, err := encode(res) if err != nil { panic(err) } if !reflect.DeepEqual(b, res) { panic(b) } return 1
54. Fuzz: custom types
55. Fuzz: custom types • When fuzzer gives you lemons bytes,
 make your types
56. Fuzz: custom types • When fuzzer gives you lemons bytes,
 make your types • Custom decode function is often better that generic
57. Fuzz: custom types • When fuzzer gives you lemons bytes,
 make your types • Custom decode function is often better that generic • If you have checksums, append them yourself
58. if len(data) != 8 { return 0 } i1, err1 := strconv.Atoi(string(data[:4])) i2, err2 := strconv.Atoi(string(data[4:])) if err1 != nil || err2 != nil { return 0 } // work with i1 and i2
59. A lot of fancy stuff • Versifier – reverse-engineers text protocol and learns its structure, makes structural mutations • Sonar – could help with stuff like checksums, but now public API yet
60. but also…
61. MORE POWER!!1 • Reuses single process • Forks it for faster startup • A lot of concurrency • Distributed across machines © Randall Munroe https://what-if.xkcd.com/13/
62. Fuzz something today!
63. Fuzz something today! … oh, and fuzzing may be coming to go tool
64. Not only Go american fuzzy lop
 http://lcamtuf.coredump.cx/afl/
 (that’s one awesome logo)
65. Not only Go american fuzzy lop
 http://lcamtuf.coredump.cx/afl/
 (that’s one awesome logo)
66. ?
67. • golang-ru.googlegroups.com • meetup.com/Golang-Moscow • 4gophers.ru/slack • GolangShow.com