How Go calculates code coverage

Transcript

1. How Go calculates code coverage David Chou @ Golang Taiwan CC-BY-SA-3.0-TW
2. @ Umbo Computer Vision 回家吃⾃⼰🏠 @ Golang Taiwan Co-organizer Software engineer, DevOps, and Gopher david74.chou @ gmail david74.chou @ facebook david74chou @ telegram Blog: https://blog.david74.dev
3. Go's coverage-based fuzzing
4. “ Fuzzing is the process of sending intentionally invalid data to a product in the hopes of triggering an error. - H.D. Moore What is fuzzing test?
6. Fuzzing test Start from a set of initial inputs Continuously manipulate inputs Semi-random input from various mutation Discover new code coverage based on instrumentation
7. Go's official fuzzing solution Official proposal [ ] Coverage-based fuzzing Write fuzz function just like test function func FuzzFoo(f *testing.F) Integrate with go command go test -fuzz Plan to land in 1.18 link
8. Code coverage in Go go test -cover go test -fuzz
9. A simple example func CountAverage(num []byte) int { sum := byte(0) for _, v := range num { sum += v } return int(sum) / len(num) } 1 2 3 4 5 6 7
10. func TestCountAverage(t *testing.T) { tests := []struct { name string num []byte want int }{ { num: []byte{1, 2, 3, 4, 5}, want: 3, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { got := CountAverage(tt.num) assert.EqualValues(t, tt.want, got) }) } } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 $ go test -cover PASS coverage: 100.0% of statements Go test coverage
11. Coverage visualization $ go test -coverprofile=coverage.out $ go tool cover -html=coverage.out
12. Go tool cover $ go tool cover Usage of 'go tool cover': Given a coverage profile produced by 'go test': go test -coverprofile=c.out Open a web browser displaying annotated source code: go tool cover -html=c.out Write out an HTML file instead of launching a web browser: go tool cover -html=c.out -o coverage.html Display coverage percentages to stdout for each function: go tool cover -func=c.out Finally, to generate modified source code with coverage annotations (what go test -cover does): go tool cover -mode=set -var=CoverageVariableName program.go
13. package go_fuzzing_playground func CountAverage(num []byte) int {GoCover.Count[0] = 1; sum := byte(0) for _, v := range num {GoCover.Count[2] = 1; sum += v } GoCover.Count[1] = 1;return int(sum) / len(num) } var GoCover = struct { Count [3]uint32 Pos [3 * 3]uint32 NumStmt [3]uint16 } { Pos: [3 * 3]uint32{ 3, 5, 0x180023, // [0] 8, 8, 0x1c0002, // [1] 5, 7, 0x30018, // [2] }, NumStmt: [3]uint16{ 2, // 0 1, // 1 1, // 2 }, } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 $ go tool cover -mode=set ./count_average.go {StartLine, EndLine, ColumnInfo} [ ] link Number of statements
14. package go_fuzzing_playground func CountAverage(num []byte) int {GoCover.Count[0] = 1; sum := byte(0) for _, v := range num {GoCover.Count[2] = 1; sum += v } GoCover.Count[1] = 1;return int(sum) / len(num) } var GoCover = struct { Count [3]uint32 Pos [3 * 3]uint32 NumStmt [3]uint16 } { Pos: [3 * 3]uint32{ 3, 5, 0x180023, // [0] 8, 8, 0x1c0002, // [1] 5, 7, 0x30018, // [2] }, NumStmt: [3]uint16{ 2, // 0 1, // 1 1, // 2 }, } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 block0 block1 block2
15. $ go test ./ -coverprofile=coverage.out && cat coverage.out ok 0.003s coverage: 100.0% of statements mode: set count_average.go:3.35,5.24 2 1 count_average.go:8.2, 8.28 1 1 count_average.go:5.24,7.3 1 1 1 2 3 4 5 6 $ go test ./ -coverprofile=coverage.out && cat coverage.out Coverage format: name.go: Line.Column, Line.Column NumStmt Count [ ] 100% = (2x1 + 1x1 + 1x1) / (2 + 1 + 1) link
16. Go test coverage Source to source transform Add instrument code before compiling Basic block coverage
17. Basic block v.s. Branch coverage block0 block1 block2 block0 block1 block2 Basic block: 100% Branch coverage: 67%
18. package go_fuzzing_playground func CountAverage(num []byte) int {GoCover.Count[0] = 1; sum := byte(0) for _, v := range num {GoCover.Count[2] = 1; sum += v } GoCover.Count[1] = 1;return int(sum) / len(num) } 1 2 3 4 5 6 7 8 9 block0 block1 block2
19. How "go test -fuzz" works
20. Go fuzz coverage
21. Go fuzz coverage Compiler instrumentation Add instrument code during compiling [ ] and [ ] by mdempsky 1 2
22. Compiler instrumentation // edge inserts coverage instrumentation for libfuzzer. func (o *orderState) edge() { // Create a new uint8 counter to be allocated in section // __libfuzzer_extra_counters. counter := staticinit.StaticName(types.Types[types.TUINT8]) counter.SetLibfuzzerExtraCounter(true) // counter += 1 incr := ir.NewAssignOpStmt(base.Pos, ir.OADD, counter, ir.NewInt(1)) o.append(incr) } 1 2 3 4 5 6 7 8 9 10 11 edge() inserts coverage instrumentation
23. func (o *orderState) stmt(n ir.Node) { switch n.Op() { ... case ir.OFOR: edge() case ir.OIF: edge() case ir.ORANGE: edge() case ir.OSELECT: edge() case ir.OSWITCH: edge() case OANDAND, OOROR: edge() ... } } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 compiler adds edge() into each edge
24. // _counters and _ecounters mark the start and end, respectively, of where // the 8-bit coverage counters reside in memory. They're known to cmd/link, // which specially assigns their addresses for this purpose. var _counters, _ecounters [0]byte func coverage() []byte { addr := unsafe.Pointer(&_counters) size := uintptr(unsafe.Pointer(&_ecounters)) - uintptr(addr) var res []byte *(*unsafeheader.Slice)(unsafe.Pointer(&res)) = unsafeheader.Slice{ Data: addr, Len: int(size), Cap: int(size), } return res } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 coverage() returns the coverage counters
25. Go fuzz coverage Still using basic block, but this could be improved in the future Compared to source-to-source, it's much easier to implement branch coverage with compiler instrumentation go-fuzz uses s2s and has lots of corner cases: miscompile, crash, invalid codes. E.g., [ ], [ ] 1 2
26. $ objdump go-fuzzing.test -h go-fuzzing.test: file format elf64-x86-64 Sections: Idx Name Size VMA LMA File off Algn 0 .text 0027890e 0000000000401000 0000000000401000 00001000 2**5 CONTENTS, ALLOC, LOAD, READONLY, CODE ... 19 .data 0000a550 0000000000883da0 0000000000883da0 00483da0 2**5 CONTENTS, ALLOC, LOAD, DATA 20 .bss 00031708 000000000088e300 000000000088e300 0048e300 2**5 ALLOC 21 .noptrbss 00006fc0 00000000008bfa20 00000000008bfa20 004bfa20 2**5 ALLOC 22 __libfuzzer_extra_counters 000052a4 00000000008c69e0 00000000008c69e0 004c69e0 2**0 ALLOC 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Bonus: libfuzzer_extra_counters
27. libfuzzer is a well-known LLVM fuzzy engine It could take coverage number from libfuzzer_extra_counters variable It means Go could use external fuzzy engine [ ] link $ go build -gcflags=all=-d=fuzzing -buildmode=c-archive -o pngfuzz.a . $ clang -o png.fuzzer pngfuzz.a -fsanitize=fuzzer 1 2 Bonus: libfuzzer_extra_counters
28. Compiler instrumentation Source to source transform Go test coverage Add instrument code before compiling Basic block coverage Difficult to adopt branch coverage Go fuzz coverage Add instrument code during compiling Basic block coverage Easier to adopt branch coverage Could expose code coverage during execution
29. Amazon CTO Dr. Werner Vogels