What the fuzz?

What the fuzz?

Fuzz testing of Go programs.

4618c5e97c59abd315cc2d7dc809f8c8?s=128

Alexey Palazhchenko

May 25, 2017
Tweet

Transcript

  1. What the fuzz? Alexey Palazhchenko

  2. Types of testing

  3. Types of testing • Unit

  4. Types of testing • Unit • Integration

  5. Types of testing • Unit • Integration • System

  6. Types of testing • Unit • Integration • System •

  7. Types of testing • Unit • Integration • System •

    … • Destructive
  8. Types of testing • Unit • Integration • System •

    … • Destructive • Sanity "
  9. Types of testing data

  10. Types of testing data • Well-known normal data

  11. Types of testing data • Well-known normal data • Well-known

    corner cases
  12. Types of testing data • Well-known normal data • Well-known

    corner cases • Well-known fixed bugs
  13. Types of testing data • Well-known normal data • Well-known

    corner cases • Well-known fixed bugs • Unknown random data
  14. Testing with random data

  15. Testing with random data • Cheap • Fast • Not

    biased
  16. • May not be deterministic • Requires good spec
 («it’s

    a feature, not a bug») • Requires a lot of brute force Testing with random data • Cheap • Fast • Not biased
  17. Testing with random data • testing/quick • github.com/google/gofuzz • github.com/leanovate/gopter


    (Go Property Tester)
  18. No examples yet We can do better

  19. Quiz question! How «go test -cover» works?

  20. How can we do better?

  21. • We can rewrite our code just like cover tool

    does How can we do better?
  22. • We can rewrite our code just like cover tool

    does • We can use code coverage information to reduce input while keeping the same coverage, then mutate it until we get new coverage How can we do better?
  23. • We can rewrite our code just like cover tool

    does • We can use code coverage information to reduce input while keeping the same coverage, then mutate it until we get new coverage • We can use MORE POWER!!1 How can we do better? © Randall Munroe https://what-if.xkcd.com/13/
  24. In a nutshell Instrument program Collect initial corpus, reduce it

    for { Apply random mutation to input Check coverage If gives new coverage, try to reduce input, then add to corpus }
  25. github.com/dvyukov/go-fuzz

  26. Get it

  27. Get it • $ go get github.com/dvyukov/go-fuzz/go-fuzz

  28. Get it • $ go get github.com/dvyukov/go-fuzz/go-fuzz • ☕

  29. Get it • $ go get github.com/dvyukov/go-fuzz/go-fuzz • ☕ •

    $ go get github.com/dvyukov/go-fuzz/go-fuzz-build
  30. // +build gofuzz func Fuzz(data []byte) int { _, err

    := handle(data) if err != nil { return 0 } return 1 }
  31. Fuzz: basic rules

  32. Fuzz: basic rules • Return 1 if data happens to

    be correct
  33. Fuzz: basic rules • Return 1 if data happens to

    be correct • Panic in case of unexpected error
  34. Fuzz: basic rules • Return 1 if data happens to

    be correct • Panic in case of unexpected error • Return 0 otherwise
  35. Fuzz: basic rules • Return 1 if data happens to

    be correct • Panic in case of unexpected error • Return 0 otherwise • No external state!
  36. Fuzz: basic rules • Return 1 if data happens to

    be correct • Panic in case of unexpected error • Return 0 otherwise • No external state! • Log only before panic
  37. Fuzz: basic rules • Return 1 if data happens to

    be correct • Panic in case of unexpected error • Return 0 otherwise • No external state! • Log only before panic • That’s ok for function to hang or eat a lot of memory
  38. Fuzz: basic rules • Return 1 if data happens to

    be correct • Panic in case of unexpected error • Return 0 otherwise • No external state! • Log only before panic • That’s ok for function to hang or eat a lot of memory • Use build tag
  39. Initial corpus collection

  40. Initial corpus collection • Small and diverse

  41. Initial corpus collection • Small and diverse • Diversity is

    more important
  42. Initial corpus collection • Small and diverse • Diversity is

    more important • Use data from unit tests!
  43. Run it

  44. Run it • $ go-fuzz-build package

  45. Run it • $ go-fuzz-build package • Fill workdir/corpus with

    files
  46. Run it • $ go-fuzz-build package • Fill workdir/corpus with

    files • $ go-fuzz -bin=package-fuzz.zip -workdir=workdir
  47. Run it • $ go-fuzz-build package • Fill workdir/corpus with

    files • $ go-fuzz -bin=package-fuzz.zip -workdir=workdir • See results in workdir/crashes
  48. Run it • $ go-fuzz-build package • Fill workdir/corpus with

    files • $ go-fuzz -bin=package-fuzz.zip -workdir=workdir • See results in workdir/crashes • Consider adding workdir/corpus to VCS
  49. Fuzz: logic checks

  50. Fuzz: logic checks • Decode, encode, compare bytes

  51. Fuzz: logic checks • Decode, encode, compare bytes • Decode,

    encode, decode, compare objects
  52. Fuzz: logic checks • Decode, encode, compare bytes • Decode,

    encode, decode, compare objects • Cross-check with different implementations
 (simple/slow vs fast/sophisticated)
  53. res, err := decode(data) if err != nil { return

    0 } b, err := encode(res) if err != nil { panic(err) } if !reflect.DeepEqual(b, res) { panic(b) } return 1
  54. Fuzz: custom types

  55. Fuzz: custom types • When fuzzer gives you lemons bytes,


    make your types
  56. Fuzz: custom types • When fuzzer gives you lemons bytes,


    make your types • Custom decode function is often better that generic
  57. Fuzz: custom types • When fuzzer gives you lemons bytes,


    make your types • Custom decode function is often better that generic • If you have checksums, append them yourself
  58. if len(data) != 8 { return 0 } i1, err1

    := strconv.Atoi(string(data[:4])) i2, err2 := strconv.Atoi(string(data[4:])) if err1 != nil || err2 != nil { return 0 } // work with i1 and i2
  59. A lot of fancy stuff • Versifier – reverse-engineers text

    protocol and learns its structure, makes structural mutations • Sonar – could help with stuff like checksums, but now public API yet
  60. but also…

  61. MORE POWER!!1 • Reuses single process • Forks it for

    faster startup • A lot of concurrency • Distributed across machines © Randall Munroe https://what-if.xkcd.com/13/
  62. Fuzz something today!

  63. Fuzz something today! … oh, and fuzzing may be coming

    to go tool
  64. Not only Go american fuzzy lop
 http://lcamtuf.coredump.cx/afl/
 (that’s one awesome

    logo)
  65. Not only Go american fuzzy lop
 http://lcamtuf.coredump.cx/afl/
 (that’s one awesome

    logo)
  66. ?

  67. • golang-ru.googlegroups.com • meetup.com/Golang-Moscow • 4gophers.ru/slack • GolangShow.com