Advanced debugging in golang - Andrii Soldatenko - Toptal

In an ideal world, you would write Go code, compile it, and then it would work perfectly the first time. But unfortunately it doesn't work in this manner. There are many different books and articles about how to write good code in go, but not so many how to debug code efficiently. In my talk I'll try to cover such important topic.
Go is a new programming language with best tools for development. In my talk I'll cover how to efficiently using these tools to debug your code. I’ll start from history of debuggers, later I'll show you how to debug go itself, if you need to find bug in language. Than I can demonstrate how to effectively debug microservices using docker and k8s, what’s remote debugging and how to apply it to application which already has been deployed. Debugging unit tests and not only code. Some tricks of debugging command line applications.

Transcript

1. Advanced debugging in Go 22-23 January 2020 Berlin, Germany
2. Andrii Soldatenko • Gopher, OSS contributor • father of • debuggers fan • Speaker at many conferences @a_soldatenko
3. Development is HARD:
4. Problem:
6. Question to audience!
7. How can I debug Go program?
8. via prints> ? package main import ( "fmt" ) func main() { const name, age = "Andrii", 22 fmt.Println(name, "is", age, "years old.") }
10. via Playground?
11. @a_soldatenko using duck debugging?
12. @a_soldatenko - delve - GDB using debuggers:
13. @a_soldatenko Interesting fact: Go version prior 1.0 included debugger OGLE
14. @a_soldatenko
15. Current versions of Go produce DWARF Debugging Standard http://dwarfstd.org/doc/dwarf-2.0.0.pdf
16. Disable DWARF generation go build -ldflags=-w but with no loss of functionality
17. delve: 101 - dlv debug - Compile and begin debugging main package in current directory, or the package specified. - dlv test - Compile test binary and begin debugging program.DWARF specification
18. Debug CLI util . ├── github.com/me/foo ├── cmd │ └── foo │ └── main.go ├── pkg │ └── baz │ ├── bar.go │ └── bar_test.go $ dlv debug github.com/me/foo/cmd/foo -- -arg1 valueType (dlv) break main.main Breakpoint 1 set at 0x49ecf3 for main.main() ./test.go:5 (dlv) continue
19. @a_soldatenko Debugging specific unit test go test -test.run TestFibonacciBig dlv test -- -test.run TestFibonacciBig
20. @a_soldatenko Debugging unit tests: example dlv test -- -test.run TestFibonacciBig (dlv) b main_test.go:6 Breakpoint 1 set at 0x115887f for github.com/andriisoldatenko/ debug_test.TestFibonacciBig() ./main_test.go:6 (dlv) c > github.com/andriisoldatenko/debug_test.TestFibonacciBig() ./ main_test.go:6 (hits goroutine(17):1 total:1) (PC: 0x115887f) 1: package main 2: 3: import "testing" 4: 5: func TestFibonacciBig(t *testing.T) { => 6: var want int64 = 55 7: got := FibonacciBig(10) 8: if got.Int64() != want { 9: t.Errorf("Invalid Fibonacci value for N: %d, got: %d, want: %d", 10, got.Int64(), want) 10: } 11: } (dlv)
21. Debugging unit tests: breakpoint dlv test github.com/andriisoldatenko/debugging-containerized-go- applications/ Type 'help' for list of commands. (dlv) b TestCloud1 Breakpoint 1 set at 0x113efaf for github.com/andriisoldatenko/ debugging-containerized-go-applications.TestCloud1() ./ hello_test.go:16 (dlv) c TestMy1 TestYour1 > github.com/andriisoldatenko/debugging-containerized-go- applications.TestCloud1() ./hello_test.go:16 (hits goroutine(33):1 total:1) (PC: 0x113efaf) 11: 12: func TestYour1(t *testing.T) { 13: fmt.Println("TestYour1") 14: } 15: => 16: func TestCloud1(t *testing.T) { 17: fmt.Println("TestCloud1") 18: }
22. @a_soldatenko Conditional breakpoint package main import "fmt" func main() { nums := []int{2, 3, 4} for i, num := range nums { if num == 3 { fmt.Println(“BUG!!! index:”, i) } } } t.me/golang_for_two
23. (dlv) b b2 hello.go:8 Breakpoint b2 set at 0x10b71e2 for main.main() ./hello.go:8 (dlv) cond b2 num == 3 (dlv) c > [b2] main.main() ./hello.go:8 (hits goroutine(1):1 total: 1) (PC: 0x10b71e2) 3: import "fmt" 4: 5: func main() { 6: nums := []int{2, 3, 4} 7: for i, num := range nums { => 8: if num == 3 { 9: fmt.Println("index:", i) 10: } 11: } 12: } (dlv) p num 3 (dlv) n Conditional breakpoint
24. @a_soldatenko How to set variable? 4: => 5: func main() { 6: a := 1 7: b := 2 8: fmt.Println(a, b) 9: } (dlv) set a = 10 t.me/golang_for_two
25. Call function (dlv) call myFunc(localVar, 4, "foo") - runtime: support for debugger function calls (https://golang.org/cl/ 109699) - Add ability to safely call functions (go-delve/delve/issues/119)
26. Call function (dlv) call t() > main.dummy() ./hello.go:18 (hits goroutine(1):1 total:1) (PC: 0x10b732d) 13: } 14: } 15: 16: 17: func dummy(){ => 18: fmt.Println("func call") 19: }
28. Pretty print with go-spew // to illustrate the points. type ctfileParseState struct { header *ctfileHeader bar string } ctfh := ctfileHeader{1297035375, 82, 33060, 4, "test"} c := ctfileParseState{&ctfh, "bar"} fmt.Printf("%+v\n", c) {header:0xf84003f480 bar:bar} spew.Printf("%+v\n", c) {header:<*>(0xf84003f480){beacon:1297035375 numShas:82 mode: 33060 typ:4 filename:test} bar:bar} https://github.com/davecgh/go-spew
29. @a_soldatenko Debugging containerized go apps t.me/golang_for_two
30. Step 1: Dockerfile $ cat Dockerfile FROM golang:1.13 WORKDIR /go/src/app COPY . . RUN go get -u github.com/go-delve/delve/cmd/dlv CMD ["app"] $ docker build -t my-golang-app . $ docker run -it --rm my-golang-app bash
31. @a_soldatenko Step 2: operation not permitted $ docker run -it --rm my-golang-app bash $root@03c1977b1063:/go/src/app# dlv debug main.go could not launch process: fork/exec /go/src/app/ __debug_bin: operation not permitted
32. Step 3: $ docker run -it --rm \ —security-opt="apparmor=unconfined" \ —cap-add=SYS_PTRACE my-golang-app bash root@7dc3a7e8b3fc:/go/src/app# dlv debug main.go Type 'help' for list of commands. (dlv) https://docs.docker.com/engine/security/apparmor/
33. @a_soldatenko Remote debugging containerized go apps
34. # Final stage FROM alpine:3.7 # Port 8080 belongs to our application, 40000 belongs to Delve EXPOSE 8080 40000 # Allow delve to run on Alpine based containers. RUN apk add --no-cache libc6-compat WORKDIR / COPY --from=build-env /server / COPY --from=build-env /go/bin/dlv / # Run delve CMD ["/dlv", "--listen=0.0.0.0:40000", "-- headless=true", "--api-version=2", "debug", "go/ src/hello", "--log" Example of Dockerfile
35. #!/usr/bin/env bash docker run \ -p 8080:8080 \ -p 40000:40000 \ --security-opt="apparmor=unconfined" \ --cap-add=SYS_PTRACE hello-debug Docker run ⬆
36. $cat ~/.dlv/config.yml substitute-path: - {from: "/go/src/hello/", to: "/Users/ andrii/workspace/src/github.com/ andriisoldatenko/debugging-containerized- go-applications"} Let’s fix it to more readable
37. Configure Delve $HOME/.dlv/config.yml Or (dlv) config -list
38. dlv connect localhost:40000 (dlv) c > main.main() /go/src/hello/hello.go:5 (hits goroutine(1):1 total:1) (PC: 0x49b3d8) 1: package main 2: 3: import "fmt" 4: => 5: func main() { 6: a := 1 7: b := 2 8: fmt.Println(a, b) 9: } (dlv) Delve connect
39. @a_soldatenko LIVE DEMO B ./start.sh
40. @a_soldatenko GDB
41. @a_soldatenko GDB issues on OSX
42. @a_soldatenko Starting program: /x/y/foo Unable to find Mach task port for process-id 28885: (os/kern) failure (0x5). (please check gdb is codesigned - see taskgated(8)) https://sourceware.org/gdb/wiki/PermissionsDarwin check gdb is codesigned
43. @a_soldatenko [New Thread 0xe03 of process 45828] [New Thread 0x1003 of process 45828] warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/bsd.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/darwin_vers.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/dirstat.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/dirstat_collection.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/err.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/exception.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/init.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/mach.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/stdio.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/stdlib.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/string.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/variant.o': can't open to read symbols: No such file or directory. Thread 2 hit Breakpoint 1, 0x0000000100000fa4 in main () (gdb) x/i $rip => 0x100000fa4 <main+4>: xor %eax,%eax (gdb) quit A debugging session is active. Inferior 1 [process 45828] will be killed. Quit anyway? (y or n) y can't open to read symbols:
44. @a_soldatenko GDB: hm….. (gdb) b main.main Breakpoint 1 at 0x10b70f0 (gdb) c The program is not being run. (gdb) run Starting program: /Users/andrii/workspace/src/ github.com/andriisoldatenko/debugging- containerized-go-applications/hello [New Thread 0xf03 of process 5994] ^C^Z [2] + 5911 suspended gdb hello
45. @a_soldatenko GDB: does’t understand comprised DWARF! go build -ldflags=-compressdwarf=false -gcflags=all="-N -l" -o hello hello.go
46. @a_soldatenko GDB: pretty print! go build -ldflags=-compressdwarf=false - gcflags=all="-N -l" -o hello hello.go ➜ go git:(master) find . -name 'runtime-gdb.py' ./src/runtime/runtime-gdb.py (gdb) source /Users/andrii/work/go/src/runtime/ runtime-gdb.py Loading Go Runtime support. ➜ debugging-containerized-go-applications git: (master) ✗ strings hello | grep gdb /usr/local/Cellar/go/1.12.7/libexec/src/runtime/ runtime-gdb.py
47. @a_soldatenko GDB: finally it works! (gdb) n 6 a := 1 (gdb) n 7 b := 2 (gdb) n 8 fmt.Println(a, b) (gdb) n 1 2 9 } (gdb)
48. @a_soldatenko Conclusion GDB: - GDB does not understand Go programs well. - The stack management, threading, and runtime contain aspects that differ enough from the execution model GDB expects - GDB can be useful in some situations (e.g., debugging Cgo code
49. @a_soldatenko Conclusion - Debugging is fun and always useful - console interactive debuggers is sometimes looks old, but helps a lot especially in cloud environments - UI clients for debuggers still slow…
50. @a_soldatenko Future Reading : - Internal Architecture of Delve - slides - DWARF specification DWARF - https://golang.org/doc/gdb - https://github.com/go-delve/delve/blob/ master/Documentation/ - (dlv) help - source code of go
52. Thank You andrii.soldatenko@toptal.com @a_soldatenko
53. Questions ? @a_soldatenko