Advanced debugging in Go 22-23 January 2020 Berlin, Germany

Andrii Soldatenko • Gopher, OSS contributor • father of • debuggers fan • Speaker at many conferences @a_soldatenko

Development is HARD:

Problem:

No content

Question to audience!

How can I debug Go program?

via prints> ? package main import ( "fmt" ) func main() { const name, age = "Andrii", 22 fmt.Println(name, "is", age, "years old.") }

No content

via Playground?

@a_soldatenko using duck debugging?

@a_soldatenko - delve - GDB using debuggers:

@a_soldatenko Interesting fact: Go version prior 1.0 included debugger OGLE

@a_soldatenko

Current versions of Go produce DWARF Debugging Standard http://dwarfstd.org/doc/dwarf-2.0.0.pdf

Disable DWARF generation go build -ldflags=-w but with no loss of functionality

delve: 101 - dlv debug - Compile and begin debugging main package in current directory, or the package specified. - dlv test - Compile test binary and begin debugging program.DWARF specification

Debug CLI util . ├── github.com/me/foo ├── cmd │ └── foo │ └── main.go ├── pkg │ └── baz │ ├── bar.go │ └── bar_test.go $ dlv debug github.com/me/foo/cmd/foo -- -arg1 valueType (dlv) break main.main Breakpoint 1 set at 0x49ecf3 for main.main() ./test.go:5 (dlv) continue

@a_soldatenko Debugging specific unit test go test -test.run TestFibonacciBig dlv test -- -test.run TestFibonacciBig

@a_soldatenko Debugging unit tests: example dlv test -- -test.run TestFibonacciBig (dlv) b main_test.go:6 Breakpoint 1 set at 0x115887f for github.com/andriisoldatenko/ debug_test.TestFibonacciBig() ./main_test.go:6 (dlv) c > github.com/andriisoldatenko/debug_test.TestFibonacciBig() ./ main_test.go:6 (hits goroutine(17):1 total:1) (PC: 0x115887f) 1: package main 2: 3: import "testing" 4: 5: func TestFibonacciBig(t *testing.T) { => 6: var want int64 = 55 7: got := FibonacciBig(10) 8: if got.Int64() != want { 9: t.Errorf("Invalid Fibonacci value for N: %d, got: %d, want: %d", 10, got.Int64(), want) 10: } 11: } (dlv)

Debugging unit tests: breakpoint dlv test github.com/andriisoldatenko/debugging-containerized-go- applications/ Type 'help' for list of commands. (dlv) b TestCloud1 Breakpoint 1 set at 0x113efaf for github.com/andriisoldatenko/ debugging-containerized-go-applications.TestCloud1() ./ hello_test.go:16 (dlv) c TestMy1 TestYour1 > github.com/andriisoldatenko/debugging-containerized-go- applications.TestCloud1() ./hello_test.go:16 (hits goroutine(33):1 total:1) (PC: 0x113efaf) 11: 12: func TestYour1(t *testing.T) { 13: fmt.Println("TestYour1") 14: } 15: => 16: func TestCloud1(t *testing.T) { 17: fmt.Println("TestCloud1") 18: }

@a_soldatenko Conditional breakpoint package main import "fmt" func main() { nums := []int{2, 3, 4} for i, num := range nums { if num == 3 { fmt.Println(“BUG!!! index:”, i) } } } t.me/golang_for_two

(dlv) b b2 hello.go:8 Breakpoint b2 set at 0x10b71e2 for main.main() ./hello.go:8 (dlv) cond b2 num == 3 (dlv) c > [b2] main.main() ./hello.go:8 (hits goroutine(1):1 total: 1) (PC: 0x10b71e2) 3: import "fmt" 4: 5: func main() { 6: nums := []int{2, 3, 4} 7: for i, num := range nums { => 8: if num == 3 { 9: fmt.Println("index:", i) 10: } 11: } 12: } (dlv) p num 3 (dlv) n Conditional breakpoint

@a_soldatenko How to set variable? 4: => 5: func main() { 6: a := 1 7: b := 2 8: fmt.Println(a, b) 9: } (dlv) set a = 10 t.me/golang_for_two

Call function (dlv) call myFunc(localVar, 4, "foo") - runtime: support for debugger function calls (https://golang.org/cl/ 109699) - Add ability to safely call functions (go-delve/delve/issues/119)

Call function (dlv) call t() > main.dummy() ./hello.go:18 (hits goroutine(1):1 total:1) (PC: 0x10b732d) 13: } 14: } 15: 16: 17: func dummy(){ => 18: fmt.Println("func call") 19: }

No content

Pretty print with go-spew // to illustrate the points. type ctfileParseState struct { header *ctfileHeader bar string } ctfh := ctfileHeader{1297035375, 82, 33060, 4, "test"} c := ctfileParseState{&ctfh, "bar"} fmt.Printf("%+v\n", c) {header:0xf84003f480 bar:bar} spew.Printf("%+v\n", c) {header:<*>(0xf84003f480){beacon:1297035375 numShas:82 mode: 33060 typ:4 filename:test} bar:bar} https://github.com/davecgh/go-spew

@a_soldatenko Debugging containerized go apps t.me/golang_for_two

Step 1: Dockerfile $ cat Dockerfile FROM golang:1.13 WORKDIR /go/src/app COPY . . RUN go get -u github.com/go-delve/delve/cmd/dlv CMD ["app"] $ docker build -t my-golang-app . $ docker run -it --rm my-golang-app bash

@a_soldatenko Step 2: operation not permitted $ docker run -it --rm my-golang-app bash $root@03c1977b1063:/go/src/app# dlv debug main.go could not launch process: fork/exec /go/src/app/ __debug_bin: operation not permitted

Step 3: $ docker run -it --rm \ —security-opt="apparmor=unconfined" \ —cap-add=SYS_PTRACE my-golang-app bash root@7dc3a7e8b3fc:/go/src/app# dlv debug main.go Type 'help' for list of commands. (dlv) https://docs.docker.com/engine/security/apparmor/

@a_soldatenko Remote debugging containerized go apps

# Final stage FROM alpine:3.7 # Port 8080 belongs to our application, 40000 belongs to Delve EXPOSE 8080 40000 # Allow delve to run on Alpine based containers. RUN apk add --no-cache libc6-compat WORKDIR / COPY --from=build-env /server / COPY --from=build-env /go/bin/dlv / # Run delve CMD ["/dlv", "--listen=0.0.0.0:40000", "-- headless=true", "--api-version=2", "debug", "go/ src/hello", "--log" Example of Dockerfile

#!/usr/bin/env bash docker run \ -p 8080:8080 \ -p 40000:40000 \ --security-opt="apparmor=unconfined" \ --cap-add=SYS_PTRACE hello-debug Docker run ⬆

$cat ~/.dlv/config.yml substitute-path: - {from: "/go/src/hello/", to: "/Users/ andrii/workspace/src/github.com/ andriisoldatenko/debugging-containerized- go-applications"} Let’s fix it to more readable

Configure Delve $HOME/.dlv/config.yml Or (dlv) config -list

dlv connect localhost:40000 (dlv) c > main.main() /go/src/hello/hello.go:5 (hits goroutine(1):1 total:1) (PC: 0x49b3d8) 1: package main 2: 3: import "fmt" 4: => 5: func main() { 6: a := 1 7: b := 2 8: fmt.Println(a, b) 9: } (dlv) Delve connect

@a_soldatenko LIVE DEMO B ./start.sh

@a_soldatenko GDB

@a_soldatenko GDB issues on OSX

@a_soldatenko Starting program: /x/y/foo Unable to find Mach task port for process-id 28885: (os/kern) failure (0x5). (please check gdb is codesigned - see taskgated(8)) https://sourceware.org/gdb/wiki/PermissionsDarwin check gdb is codesigned

@a_soldatenko [New Thread 0xe03 of process 45828] [New Thread 0x1003 of process 45828] warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/bsd.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/darwin_vers.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/dirstat.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/dirstat_collection.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/err.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/exception.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/init.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/mach.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/stdio.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/stdlib.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/string.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/variant.o': can't open to read symbols: No such file or directory. Thread 2 hit Breakpoint 1, 0x0000000100000fa4 in main () (gdb) x/i $rip => 0x100000fa4 : xor %eax,%eax (gdb) quit A debugging session is active. Inferior 1 [process 45828] will be killed. Quit anyway? (y or n) y can't open to read symbols:

@a_soldatenko GDB: hm….. (gdb) b main.main Breakpoint 1 at 0x10b70f0 (gdb) c The program is not being run. (gdb) run Starting program: /Users/andrii/workspace/src/ github.com/andriisoldatenko/debugging- containerized-go-applications/hello [New Thread 0xf03 of process 5994] ^C^Z [2] + 5911 suspended gdb hello

@a_soldatenko GDB: does’t understand comprised DWARF! go build -ldflags=-compressdwarf=false -gcflags=all="-N -l" -o hello hello.go

@a_soldatenko GDB: pretty print! go build -ldflags=-compressdwarf=false - gcflags=all="-N -l" -o hello hello.go ➜ go git:(master) find . -name 'runtime-gdb.py' ./src/runtime/runtime-gdb.py (gdb) source /Users/andrii/work/go/src/runtime/ runtime-gdb.py Loading Go Runtime support. ➜ debugging-containerized-go-applications git: (master) ✗ strings hello | grep gdb /usr/local/Cellar/go/1.12.7/libexec/src/runtime/ runtime-gdb.py

@a_soldatenko GDB: finally it works! (gdb) n 6 a := 1 (gdb) n 7 b := 2 (gdb) n 8 fmt.Println(a, b) (gdb) n 1 2 9 } (gdb)

@a_soldatenko Conclusion GDB: - GDB does not understand Go programs well. - The stack management, threading, and runtime contain aspects that differ enough from the execution model GDB expects - GDB can be useful in some situations (e.g., debugging Cgo code

@a_soldatenko Conclusion - Debugging is fun and always useful - console interactive debuggers is sometimes looks old, but helps a lot especially in cloud environments - UI clients for debuggers still slow…

@a_soldatenko Future Reading : - Internal Architecture of Delve - slides - DWARF specification DWARF - https://golang.org/doc/gdb - https://github.com/go-delve/delve/blob/ master/Documentation/ - (dlv) help - source code of go

No content

Thank You andrii.soldatenko@toptal.com @a_soldatenko

Questions ? @a_soldatenko