Debug a Go Application running on Kubernetes cluster / Alexei Ledenev

Slide 1

Slide 1 text

Debugging Go on Kubernetes GopherCon 2019, Israel Alexei Ledenev

Slide 2

Slide 2 text

Intro Alexei Ledenev @alexeiled alexei-led/pumba

Slide 3

Slide 3 text

Quick Survey 1. Who is using Docker today? 2. Who is using Kubernetes?

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

What is Kubernetes?

Slide 6

Slide 6 text

Kubernetes Objects wordpress service 210.14.128.31

Slide 7

Slide 7 text

Core Kubernetes

Slide 8

Slide 8 text

Debugging Go

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

K8s Development Flow

Slide 11

Slide 11 text

K8s Development Models 100% local 100% remote Realism Low High Feedback Fast Slow Resource Footprint Heavy Lightweight

Slide 12

Slide 12 text

100% Remote

Slide 13

Slide 13 text

Build image:x.x.x-debug Go App Static Binary (built with `go build --gcﬂag "-N -l" ...`) Delve Binary (COPY --from builder OR go get) EXPOSE 2345 (debugger port) Base Image (alpine/scratch OR golang) CMD ["dlv", "exec", "--headless", "--api-version=2", "--listen=:2345", "--log", "./todo-app"]

Slide 14

Slide 14 text

Deploy image:x.x.x-debug • Patch/Re-deploy K8s Deployment: image, SYS_PTRACE • Expose Delve server port (kubectl port-forward) • Rollback after debug session Every Commit

Slide 15

Slide 15 text

# build Docker image with delve debugger docker build -t alexeiled/todo-app-web:debug -f debug/Dockerfile . # push Docker image to registry docker push alexeiled/todo-app-web:debug # patch Kubernetes deployment and service kubectl patch deploy todo-app --type="json" -ntodo-app \ --patch "$(/bin/cat dbg/patch-debug.json)" # forward debugger port to the local machine kubectl port-forward pod/app-pod 2345 # DEBUG SESSION ....... # rollback the patch kubectl patch deploy todo-app --type="merge" \ --patch "$(/bin/cat k8s-deployment/todo-app-deployment.yaml)" -ntodo-app

Slide 16

Slide 16 text

Local + Remote

Slide 17

Slide 17 text

Telepresence (Datawire ) Debug Locally Instant Feedback Realistic Environment Minimal Resource Footprint Use your own tools

Slide 18

Slide 18 text

reverse tunnel (sshd) over a kubectl port-

Slide 19

Slide 19 text

Telepresence Demo

Slide 20

Slide 20 text

# open VS Code editor code . # [code.terminal.#1]: swap todo-app deployment with bash telepresence --namespace=todo-app --swap-deployment=todo-app \ --expose 3000 --run bash # [code.terminal.#1] run debugger (with .debug/runDelve.sh) dlv debug --listen=localhost:2345 \ --headless=true --api-version=2 --log=true ./ \ -- -config-file="$TELEPRESENCE_ROOT/etc/todo-app/redis.config" # [code.terminal.#2] invoke todo-app service in K8s cluster http http://a8d6ad3e6169111e98c1d02b9ca3b647-624533560.us- west-2.elb.amazonaws.com:9080/read/todo # DEBUG SESSION ....... stop debug (dlv); exit (swap back)