Building a Tool to Debug Minimal Container Images in Kubernetes, Docker and ContainerD

Minimal container images are getting more popular, but they can't go mainstream unless developers and DevOps engineers can debug them. You need to be able to debug them locally and in production across different container runtimes, but there was no tool to meet those needs until now.

This is a story of adding multi-runtime container debugging capability in MinToolkit (aka DockerSlim ). You will learn how this sidecar-based debugging capability is designed and implemented in addition to learning how to use it, so you'll be able to build your own tools to do the same.

You'll also learn about the technologies in different container runtimes leveraged to build this multi-platform debugging capability and how it fits in the existing tooling ecosystem as well as the gaps it addresses in existing tools like "kubectl debug". The talk will have a demo showing how to debug containerized apps on different runtimes using a set of predefined debug images contrasting the user experience compared to "kubectl debug".

Transcript

2. Saiyam Pathak Ivan Velichko Kyle Quest Building a Tool to Debug Minimal Container Images in Docker, containerd, and Kubernetes https://github.com/mintoolkit/mint (aka DockerSlim / SlimToolkit)
3. Ivan Velichko • The Iximiuz Labs Creator (the Learning Platform to Master Cloud Native Craft) • https://labs.iximiuz.com • https://iximiuz.com/newsletter • https://twitter.com/iximiuz • https://github.com/iximiuz
4. Saiyam Pathak • Field CTO, Civo • CNCF Ambassador • Founder Kubesimplify • https://xcom/saiyampathak • https://github.com/saiyam1814
5. Kyle Quest • Creator, DockerSlim (aka SlimToolkit/minToolkit) • Founder, AutonomousPlane • Founder/CTO, Slim.AI • https://twitter.com/kcqon • https://github.com/kcq
6. How It Started
8. Dan Čermák • Software Engineer, SUSE • Developer Tools • https://dancermak.name • https://twitter.com/DefolosDC • https://github.com/dcermak
9. The Problem
11. What do you typically do when a container… starts acting up? Docker • docker exec • nsenter • docker debug (beta/paid) containerd • nerdctl/ctr exec • nsenter Kubernetes • kubectl exec • kubectl debug • SSH + nsenter*
12. What if it’s a minimal container image… with no shell? • Slimmed/minified (minToolkit) • gcr.io/distroless • Chainguard Images • Slim and scratch bases
13. Docker • ❌ docker exec • ✅ nsenter • ✅ docker debug (beta/paid) containerd • ❌ nerdctl/ctr exec • ✅ nsenter Kubernetes • ❌ kubectl exec • ✅ kubectl debug • ✅ SSH + nsenter* What if it’s a minimal container image… with no shell?
14. 🤩 docker exec • Can run target binaries, see the target rootfs "as is", install extra tools • …but only works if there is a shell 🙂 kubectl debug / docker run (aka debugging sidecar) • Can see target processes, access target rootfs (but not “as is”), use custom debugger image • …but cannot see volumes, run as user, or fine-tune the ephemeral container spec, see target rootfs "as is" 🤯 nsenter • Can do anything • …given you know how and have sufficient access (on the node) Developer Experience During Debugging
15. From docker exec • Run target binaries • See the target rootfs "as is" From kubectl debug • Works even with shell-less targets • Can bring you own debugging toolkit Can We Combine the Best of Two Worlds?
16. Since we’re writing a new tool, we can try providing the same UX for all runtimes supported runtimes! Bonus: Uniform debugging UX across runtimes $ tool debug --runtime docker --target <container> [cmd] $ tool debug --runtime containerd --target <container> [cmd] $ tool debug --runtime k8s --pod <pod> --target <container> [cmd]
17. The Tech & Design
18. What actually happens when you docker exec
19. • https://github.com/opencontainers/runtime-spec/issues/345 • https://github.com/opencontainers/runtime-spec/pull/388 What actually happens when you docker exec 💡 Fun fact - The OCI Runtime Spec doesn't define the exec command. Check out the issues #345 and #388 for an interesting discussion of how the exec functionality is actually redundant and can be reproduced in runtimes implementing only create and start commands.
20. $ docker run \ --net container:app \ --ipc container:app \ --pid container:app \ busybox Reproducing docker exec with docker run
21. Reproducing docker exec with docker run
22. What actually happens when you kubectl debug
23. Unfortunately, mount namespace cannot be shared. Let’s see if we are clever enough to overcome that! Gotcha: But My rootfs Looks Different!
24. Accessing Target rootfs via /proc/<pid>/root Since pid namespace is shared, we can use the /proc/<pid>/root (e.g., /proc/1/root) trick to browse target rootfs!
25. Adding chroot, symlinks and $PATH to the mix!
26. The Implementation
27. High Level Algorithm (all runtimes) • Create the sidecar container (connected to the target container) • Start the sidecar container • Attach the terminal to its stdio streams • Wait for the sidecar container to exit Sounds too easy? Now repeat it for every runtime!
28. Gotcha: Will it Always Work? • The chroot trick requires the root user and sufficient privileges ◦ May need to run the sidecar with --privileged flag ◦ If the Pod spec mandates runAsNonRoot, don’t chroot! • The debugger toolkit image has to be “portable enough” (more later)
29. Docker API
30. Relevant Client Interfaces and Methods • ContainerAPIClient interface • ContainerCreate • ContainerStart • ContainerAttach • ContainerWait Notes • Configuring the namespace modes - key for connecting to target container!
31. API github.com/docker/docker/client
32. pkg/app/master/command/debug/handle_docker_runtime.go
33. ContainerD API
34. Relevant Client Interfaces and Methods • client.NewContainer • Container.NewTask • Task.Start • Task.HandleConsoleResize • Task.Wait Notes / Challenges • Much more lower-level (compared to Docker) • Doesn’t provide a facade client interface • Requires wrapping one’s head around it • Debugger Container Spec Namespaces - key for connecting to target container!
35. API github.com/containerd/containerd
36. pkg/app/master/command/debug/handle_containerd_runtime.go
37. Kubernetes API
38. Relevant Client Interfaces and Methods • core/v1.PodInterface.UpdateEphemeralContainers() or lower level core/v1.PodInterface.Patch/Put() to add the ephemeral container to the Pod’s spec • core/v1.PodInterface.Wait() • rest.Interface.Post() .Resource("pods").SubResource("attach") Notes • TargetContainerName in EphemeralContainer - key for connecting to target container! • Learning how to initialize and use K8s client-go library is a challenge on its own!
39. API: github.com/kubernetes/client-go
40. pkg/app/master/command/debug/handle_kubernetes_runtime.go
41. The Debugger Images
42. Gotchas and Considerations With the chroot trick (beware of portability issues) • busybox:musl - simple and portable • Nix and nixery.dev ...and we also can use the target image's binaries "as is" Without the chroot+PATH+symlink trick (simpler) • NetShoot, KoolKit or general ubuntu/debian/etc are good candidates ...but we cannot invoke binaries from the target image easily
43. pkg/app/master/command/debug/debug_images.go
44. The Challenges
45. Reverse Engineering APIs and Client Code • Not much documentation & examples …but you can find some inspiration in • github.com/containerd/nerdctl • github.com/docker/cli • github.com/kubernetes/kubectl • github.com/iximiuz/client-go-examples • https://github.com/iximiuz/cdebug Pro Tip: Use GitHub code search - in highly-specialized areas, it’s still more helpful than ChatGPT.
46. Other Gotchas Cases that are trickier to debug • Tightened security contexts • Non-root containers • Picking the portable debugging toolkit
47. The Demo
51. The Code
52. The Debug Command Code • https://github.com/mintoolkit/mint - main repo • https://github.com/mintoolkit/mint/tree/master/pkg/app/master/comman d/debug - the “debug” CLI command code • https://github.com/mintoolkit/mint/blob/master/pkg/app/master/comma nd/debug/handle_docker_runtime.go - Docker runtime support • https://github.com/mintoolkit/mint/blob/master/pkg/app/master/comma nd/debug/handle_kubernetes_runtime.go - K8s runtime support • https://github.com/mintoolkit/mint/blob/master/pkg/app/master/comma nd/debug/handle_containerd_runtime.go - ContainerD runtime support
53. Thank You!
54. Thank You for attending! You can connect with us: @kcqon @SaiyamPathak Share your feedback!