Slide 1

Slide 1 text

How Go calculates code coverage David Chou @ Golang Taiwan CC-BY-SA-3.0-TW

Slide 2

Slide 2 text

@ Umbo Computer Vision 回家吃⾃⼰🏠 @ Golang Taiwan Co-organizer Software engineer, DevOps, and Gopher david74.chou @ gmail david74.chou @ facebook david74chou @ telegram Blog: https://blog.david74.dev

Slide 3

Slide 3 text

Go's coverage-based fuzzing

Slide 4

Slide 4 text

“ Fuzzing is the process of sending intentionally invalid data to a product in the hopes of triggering an error. - H.D. Moore What is fuzzing test?

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

Fuzzing test Start from a set of initial inputs Continuously manipulate inputs Semi-random input from various mutation Discover new code coverage based on instrumentation

Slide 7

Slide 7 text

Go's official fuzzing solution Official proposal [ ] Coverage-based fuzzing Write fuzz function just like test function func FuzzFoo(f *testing.F) Integrate with go command go test -fuzz Plan to land in 1.18 link

Slide 8

Slide 8 text

Code coverage in Go go test -cover go test -fuzz

Slide 9

Slide 9 text

A simple example func CountAverage(num []byte) int { sum := byte(0) for _, v := range num { sum += v } return int(sum) / len(num) } 1 2 3 4 5 6 7

Slide 10

Slide 10 text

func TestCountAverage(t *testing.T) { tests := []struct { name string num []byte want int }{ { num: []byte{1, 2, 3, 4, 5}, want: 3, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { got := CountAverage(tt.num) assert.EqualValues(t, tt.want, got) }) } } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 $ go test -cover PASS coverage: 100.0% of statements Go test coverage

Slide 11

Slide 11 text

Coverage visualization $ go test -coverprofile=coverage.out $ go tool cover -html=coverage.out

Slide 12

Slide 12 text

Go tool cover $ go tool cover Usage of 'go tool cover': Given a coverage profile produced by 'go test': go test -coverprofile=c.out Open a web browser displaying annotated source code: go tool cover -html=c.out Write out an HTML file instead of launching a web browser: go tool cover -html=c.out -o coverage.html Display coverage percentages to stdout for each function: go tool cover -func=c.out Finally, to generate modified source code with coverage annotations (what go test -cover does): go tool cover -mode=set -var=CoverageVariableName program.go

Slide 13

Slide 13 text

package go_fuzzing_playground func CountAverage(num []byte) int {GoCover.Count[0] = 1; sum := byte(0) for _, v := range num {GoCover.Count[2] = 1; sum += v } GoCover.Count[1] = 1;return int(sum) / len(num) } var GoCover = struct { Count [3]uint32 Pos [3 * 3]uint32 NumStmt [3]uint16 } { Pos: [3 * 3]uint32{ 3, 5, 0x180023, // [0] 8, 8, 0x1c0002, // [1] 5, 7, 0x30018, // [2] }, NumStmt: [3]uint16{ 2, // 0 1, // 1 1, // 2 }, } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 $ go tool cover -mode=set ./count_average.go {StartLine, EndLine, ColumnInfo} [ ] link Number of statements

Slide 14

Slide 14 text

package go_fuzzing_playground func CountAverage(num []byte) int {GoCover.Count[0] = 1; sum := byte(0) for _, v := range num {GoCover.Count[2] = 1; sum += v } GoCover.Count[1] = 1;return int(sum) / len(num) } var GoCover = struct { Count [3]uint32 Pos [3 * 3]uint32 NumStmt [3]uint16 } { Pos: [3 * 3]uint32{ 3, 5, 0x180023, // [0] 8, 8, 0x1c0002, // [1] 5, 7, 0x30018, // [2] }, NumStmt: [3]uint16{ 2, // 0 1, // 1 1, // 2 }, } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 block0 block1 block2

Slide 15

Slide 15 text

$ go test ./ -coverprofile=coverage.out && cat coverage.out ok 0.003s coverage: 100.0% of statements mode: set count_average.go:3.35,5.24 2 1 count_average.go:8.2, 8.28 1 1 count_average.go:5.24,7.3 1 1 1 2 3 4 5 6 $ go test ./ -coverprofile=coverage.out && cat coverage.out Coverage format: name.go: Line.Column, Line.Column NumStmt Count [ ] 100% = (2x1 + 1x1 + 1x1) / (2 + 1 + 1) link

Slide 16

Slide 16 text

Go test coverage Source to source transform Add instrument code before compiling Basic block coverage

Slide 17

Slide 17 text

Basic block v.s. Branch coverage block0 block1 block2 block0 block1 block2 Basic block: 100% Branch coverage: 67%

Slide 18

Slide 18 text

package go_fuzzing_playground func CountAverage(num []byte) int {GoCover.Count[0] = 1; sum := byte(0) for _, v := range num {GoCover.Count[2] = 1; sum += v } GoCover.Count[1] = 1;return int(sum) / len(num) } 1 2 3 4 5 6 7 8 9 block0 block1 block2

Slide 19

Slide 19 text

How "go test -fuzz" works

Slide 20

Slide 20 text

Go fuzz coverage

Slide 21

Slide 21 text

Go fuzz coverage Compiler instrumentation Add instrument code during compiling [ ] and [ ] by mdempsky 1 2

Slide 22

Slide 22 text

Compiler instrumentation // edge inserts coverage instrumentation for libfuzzer. func (o *orderState) edge() { // Create a new uint8 counter to be allocated in section // __libfuzzer_extra_counters. counter := staticinit.StaticName(types.Types[types.TUINT8]) counter.SetLibfuzzerExtraCounter(true) // counter += 1 incr := ir.NewAssignOpStmt(base.Pos, ir.OADD, counter, ir.NewInt(1)) o.append(incr) } 1 2 3 4 5 6 7 8 9 10 11 edge() inserts coverage instrumentation

Slide 23

Slide 23 text

func (o *orderState) stmt(n ir.Node) { switch n.Op() { ... case ir.OFOR: edge() case ir.OIF: edge() case ir.ORANGE: edge() case ir.OSELECT: edge() case ir.OSWITCH: edge() case OANDAND, OOROR: edge() ... } } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 compiler adds edge() into each edge

Slide 24

Slide 24 text

// _counters and _ecounters mark the start and end, respectively, of where // the 8-bit coverage counters reside in memory. They're known to cmd/link, // which specially assigns their addresses for this purpose. var _counters, _ecounters [0]byte func coverage() []byte { addr := unsafe.Pointer(&_counters) size := uintptr(unsafe.Pointer(&_ecounters)) - uintptr(addr) var res []byte *(*unsafeheader.Slice)(unsafe.Pointer(&res)) = unsafeheader.Slice{ Data: addr, Len: int(size), Cap: int(size), } return res } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 coverage() returns the coverage counters

Slide 25

Slide 25 text

Go fuzz coverage Still using basic block, but this could be improved in the future Compared to source-to-source, it's much easier to implement branch coverage with compiler instrumentation go-fuzz uses s2s and has lots of corner cases: miscompile, crash, invalid codes. E.g., [ ], [ ] 1 2

Slide 26

Slide 26 text

$ objdump go-fuzzing.test -h go-fuzzing.test: file format elf64-x86-64 Sections: Idx Name Size VMA LMA File off Algn 0 .text 0027890e 0000000000401000 0000000000401000 00001000 2**5 CONTENTS, ALLOC, LOAD, READONLY, CODE ... 19 .data 0000a550 0000000000883da0 0000000000883da0 00483da0 2**5 CONTENTS, ALLOC, LOAD, DATA 20 .bss 00031708 000000000088e300 000000000088e300 0048e300 2**5 ALLOC 21 .noptrbss 00006fc0 00000000008bfa20 00000000008bfa20 004bfa20 2**5 ALLOC 22 __libfuzzer_extra_counters 000052a4 00000000008c69e0 00000000008c69e0 004c69e0 2**0 ALLOC 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Bonus: libfuzzer_extra_counters

Slide 27

Slide 27 text

libfuzzer is a well-known LLVM fuzzy engine It could take coverage number from libfuzzer_extra_counters variable It means Go could use external fuzzy engine [ ] link $ go build -gcflags=all=-d=fuzzing -buildmode=c-archive -o pngfuzz.a . $ clang -o png.fuzzer pngfuzz.a -fsanitize=fuzzer 1 2 Bonus: libfuzzer_extra_counters

Slide 28

Slide 28 text

Compiler instrumentation Source to source transform Go test coverage Add instrument code before compiling Basic block coverage Difficult to adopt branch coverage Go fuzz coverage Add instrument code during compiling Basic block coverage Easier to adopt branch coverage Could expose code coverage during execution

Slide 29

Slide 29 text

Amazon CTO Dr. Werner Vogels