Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How Go calculates code coverage

How Go calculates code coverage

B3a7c0d5c590642dbce68bce8929a2df?s=128

David Chou

August 31, 2021
Tweet

Transcript

  1. How Go calculates code coverage David Chou @ Golang Taiwan

    CC-BY-SA-3.0-TW
  2. @ Umbo Computer Vision 回家吃⾃⼰🏠 @ Golang Taiwan Co-organizer Software

    engineer, DevOps, and Gopher david74.chou @ gmail david74.chou @ facebook david74chou @ telegram Blog: https://blog.david74.dev
  3. Go's coverage-based fuzzing

  4. “ Fuzzing is the process of sending intentionally invalid data

    to a product in the hopes of triggering an error. - H.D. Moore What is fuzzing test?
  5. None
  6. Fuzzing test Start from a set of initial inputs Continuously

    manipulate inputs Semi-random input from various mutation Discover new code coverage based on instrumentation
  7. Go's official fuzzing solution Official proposal [ ] Coverage-based fuzzing

    Write fuzz function just like test function func FuzzFoo(f *testing.F) Integrate with go command go test -fuzz Plan to land in 1.18 link
  8. Code coverage in Go go test -cover go test -fuzz

  9. A simple example func CountAverage(num []byte) int { sum :=

    byte(0) for _, v := range num { sum += v } return int(sum) / len(num) } 1 2 3 4 5 6 7
  10. func TestCountAverage(t *testing.T) { tests := []struct { name string

    num []byte want int }{ { num: []byte{1, 2, 3, 4, 5}, want: 3, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { got := CountAverage(tt.num) assert.EqualValues(t, tt.want, got) }) } } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 $ go test -cover PASS coverage: 100.0% of statements Go test coverage
  11. Coverage visualization $ go test -coverprofile=coverage.out $ go tool cover

    -html=coverage.out
  12. Go tool cover $ go tool cover Usage of 'go

    tool cover': Given a coverage profile produced by 'go test': go test -coverprofile=c.out Open a web browser displaying annotated source code: go tool cover -html=c.out Write out an HTML file instead of launching a web browser: go tool cover -html=c.out -o coverage.html Display coverage percentages to stdout for each function: go tool cover -func=c.out Finally, to generate modified source code with coverage annotations (what go test -cover does): go tool cover -mode=set -var=CoverageVariableName program.go
  13. package go_fuzzing_playground func CountAverage(num []byte) int {GoCover.Count[0] = 1; sum

    := byte(0) for _, v := range num {GoCover.Count[2] = 1; sum += v } GoCover.Count[1] = 1;return int(sum) / len(num) } var GoCover = struct { Count [3]uint32 Pos [3 * 3]uint32 NumStmt [3]uint16 } { Pos: [3 * 3]uint32{ 3, 5, 0x180023, // [0] 8, 8, 0x1c0002, // [1] 5, 7, 0x30018, // [2] }, NumStmt: [3]uint16{ 2, // 0 1, // 1 1, // 2 }, } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 $ go tool cover -mode=set ./count_average.go {StartLine, EndLine, ColumnInfo} [ ] link Number of statements
  14. package go_fuzzing_playground func CountAverage(num []byte) int {GoCover.Count[0] = 1; sum

    := byte(0) for _, v := range num {GoCover.Count[2] = 1; sum += v } GoCover.Count[1] = 1;return int(sum) / len(num) } var GoCover = struct { Count [3]uint32 Pos [3 * 3]uint32 NumStmt [3]uint16 } { Pos: [3 * 3]uint32{ 3, 5, 0x180023, // [0] 8, 8, 0x1c0002, // [1] 5, 7, 0x30018, // [2] }, NumStmt: [3]uint16{ 2, // 0 1, // 1 1, // 2 }, } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 block0 block1 block2
  15. $ go test ./ -coverprofile=coverage.out && cat coverage.out ok 0.003s

    coverage: 100.0% of statements mode: set count_average.go:3.35,5.24 2 1 count_average.go:8.2, 8.28 1 1 count_average.go:5.24,7.3 1 1 1 2 3 4 5 6 $ go test ./ -coverprofile=coverage.out && cat coverage.out Coverage format: name.go: Line.Column, Line.Column NumStmt Count [ ] 100% = (2x1 + 1x1 + 1x1) / (2 + 1 + 1) link
  16. Go test coverage Source to source transform Add instrument code

    before compiling Basic block coverage
  17. Basic block v.s. Branch coverage block0 block1 block2 block0 block1

    block2 Basic block: 100% Branch coverage: 67%
  18. package go_fuzzing_playground func CountAverage(num []byte) int {GoCover.Count[0] = 1; sum

    := byte(0) for _, v := range num {GoCover.Count[2] = 1; sum += v } GoCover.Count[1] = 1;return int(sum) / len(num) } 1 2 3 4 5 6 7 8 9 block0 block1 block2
  19. How "go test -fuzz" works

  20. Go fuzz coverage

  21. Go fuzz coverage Compiler instrumentation Add instrument code during compiling

    [ ] and [ ] by mdempsky 1 2
  22. Compiler instrumentation // edge inserts coverage instrumentation for libfuzzer. func

    (o *orderState) edge() { // Create a new uint8 counter to be allocated in section // __libfuzzer_extra_counters. counter := staticinit.StaticName(types.Types[types.TUINT8]) counter.SetLibfuzzerExtraCounter(true) // counter += 1 incr := ir.NewAssignOpStmt(base.Pos, ir.OADD, counter, ir.NewInt(1)) o.append(incr) } 1 2 3 4 5 6 7 8 9 10 11 edge() inserts coverage instrumentation
  23. func (o *orderState) stmt(n ir.Node) { switch n.Op() { ...

    case ir.OFOR: edge() case ir.OIF: edge() case ir.ORANGE: edge() case ir.OSELECT: edge() case ir.OSWITCH: edge() case OANDAND, OOROR: edge() ... } } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 compiler adds edge() into each edge
  24. // _counters and _ecounters mark the start and end, respectively,

    of where // the 8-bit coverage counters reside in memory. They're known to cmd/link, // which specially assigns their addresses for this purpose. var _counters, _ecounters [0]byte func coverage() []byte { addr := unsafe.Pointer(&_counters) size := uintptr(unsafe.Pointer(&_ecounters)) - uintptr(addr) var res []byte *(*unsafeheader.Slice)(unsafe.Pointer(&res)) = unsafeheader.Slice{ Data: addr, Len: int(size), Cap: int(size), } return res } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 coverage() returns the coverage counters
  25. Go fuzz coverage Still using basic block, but this could

    be improved in the future Compared to source-to-source, it's much easier to implement branch coverage with compiler instrumentation go-fuzz uses s2s and has lots of corner cases: miscompile, crash, invalid codes. E.g., [ ], [ ] 1 2
  26. $ objdump go-fuzzing.test -h go-fuzzing.test: file format elf64-x86-64 Sections: Idx

    Name Size VMA LMA File off Algn 0 .text 0027890e 0000000000401000 0000000000401000 00001000 2**5 CONTENTS, ALLOC, LOAD, READONLY, CODE ... 19 .data 0000a550 0000000000883da0 0000000000883da0 00483da0 2**5 CONTENTS, ALLOC, LOAD, DATA 20 .bss 00031708 000000000088e300 000000000088e300 0048e300 2**5 ALLOC 21 .noptrbss 00006fc0 00000000008bfa20 00000000008bfa20 004bfa20 2**5 ALLOC 22 __libfuzzer_extra_counters 000052a4 00000000008c69e0 00000000008c69e0 004c69e0 2**0 ALLOC 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Bonus: libfuzzer_extra_counters
  27. libfuzzer is a well-known LLVM fuzzy engine It could take

    coverage number from libfuzzer_extra_counters variable It means Go could use external fuzzy engine [ ] link $ go build -gcflags=all=-d=fuzzing -buildmode=c-archive -o pngfuzz.a . $ clang -o png.fuzzer pngfuzz.a -fsanitize=fuzzer 1 2 Bonus: libfuzzer_extra_counters
  28. Compiler instrumentation Source to source transform Go test coverage Add

    instrument code before compiling Basic block coverage Difficult to adopt branch coverage Go fuzz coverage Add instrument code during compiling Basic block coverage Easier to adopt branch coverage Could expose code coverage during execution
  29. Amazon CTO Dr. Werner Vogels