Debug a Go Application running on Kubernetes cluster / Alexei Ledenev

Debug a Go Application running on Kubernetes cluster / Alexei Ledenev

Kubernetes is a popular platform for running microservices. Debugging a microservice running on Kubernetes cluster can be a challenge. In this session, I will present two approaches to debug a running microservice, written in Go, on Kubernetes cluster.

The session will demonstrate how to setup and configure a development environment to debug Go microservice, both remotely and locally. The remote debugging demo will focus on building “debug-ready” Docker image with Go app and properly configured delve debugger and configuring IDE access. The local debugging demo will show how to use telepresence tool to swap already deployed Go microservice with local debug session, without a need to package and redeploy every code change.


GopherCon Israel

February 11, 2019


  1. Debugging Go on Kubernetes GopherCon 2019, Israel Alexei Ledenev

  2. Intro Alexei Ledenev @alexeiled alexei-led/pumba

  3. Quick Survey 1. Who is using Docker today? 2. Who

    is using Kubernetes?
  4. None
  5. What is Kubernetes?

  6. Kubernetes Objects wordpress service

  7. Core Kubernetes

  8. Debugging Go

  9. None
  10. K8s Development Flow

  11. K8s Development Models 100% local 100% remote Realism Low High

    Feedback Fast Slow Resource Footprint Heavy Lightweight
  12. 100% Remote

  13. Build image:x.x.x-debug Go App Static Binary (built with `go build

    --gcflag "-N -l" ...`) Delve Binary (COPY --from builder OR go get) EXPOSE 2345 (debugger port) Base Image (alpine/scratch OR golang) CMD ["dlv", "exec", "--headless", "--api-version=2", "--listen=:2345", "--log", "./todo-app"]
  14. Deploy image:x.x.x-debug • Patch/Re-deploy K8s Deployment: image, SYS_PTRACE • Expose

    Delve server port (kubectl port-forward) • Rollback after debug session Every Commit
  15. # build Docker image with delve debugger docker build -t

    alexeiled/todo-app-web:debug -f debug/Dockerfile . # push Docker image to registry docker push alexeiled/todo-app-web:debug # patch Kubernetes deployment and service kubectl patch deploy todo-app --type="json" -ntodo-app \ --patch "$(/bin/cat dbg/patch-debug.json)" # forward debugger port to the local machine kubectl port-forward pod/app-pod 2345 # DEBUG SESSION ....... # rollback the patch kubectl patch deploy todo-app --type="merge" \ --patch "$(/bin/cat k8s-deployment/todo-app-deployment.yaml)" -ntodo-app
  16. Local + Remote

  17. Telepresence (Datawire ) Debug Locally Instant Feedback Realistic Environment Minimal

    Resource Footprint Use your own tools
  18. reverse tunnel (sshd) over a kubectl port-

  19. Telepresence Demo

  20. # open VS Code editor code . # [code.terminal.#1]: swap

    todo-app deployment with bash telepresence --namespace=todo-app --swap-deployment=todo-app \ --expose 3000 --run bash # [code.terminal.#1] run debugger (with .debug/ dlv debug --listen=localhost:2345 \ --headless=true --api-version=2 --log=true ./ \ -- -config-file="$TELEPRESENCE_ROOT/etc/todo-app/redis.config" # [code.terminal.#2] invoke todo-app service in K8s cluster http # DEBUG SESSION ....... stop debug (dlv); exit (swap back)
  21. Debug Go in K8s

  22. @alexeiled