No content

Saiyam Pathak Ivan Velichko Kyle Quest Building a Tool to Debug Minimal Container Images in Docker, containerd, and Kubernetes https://github.com/mintoolkit/mint (aka DockerSlim / SlimToolkit) 

Ivan Velichko ● The Iximiuz Labs Creator (the Learning Platform to Master Cloud Native Craft) ● https://labs.iximiuz.com ● https://iximiuz.com/newsletter ● https://twitter.com/iximiuz ● https://github.com/iximiuz 

Saiyam Pathak ● Field CTO, Civo ● CNCF Ambassador ● Founder Kubesimplify ● https://xcom/saiyampathak ● https://github.com/saiyam1814 

Kyle Quest ● Creator, DockerSlim (aka SlimToolkit/minToolkit) ● Founder, AutonomousPlane ● Founder/CTO, Slim.AI ● https://twitter.com/kcqon ● https://github.com/kcq 

How It Started 

No content

Dan Čermák ● Software Engineer, SUSE ● Developer Tools ● https://dancermak.name ● https://twitter.com/DefolosDC ● https://github.com/dcermak 

The Problem 

No content

What do you typically do when a container… starts acting up? Docker ● docker exec ● nsenter ● docker debug (beta/paid) containerd ● nerdctl/ctr exec ● nsenter Kubernetes ● kubectl exec ● kubectl debug ● SSH + nsenter* 

What if it’s a minimal container image… with no shell? ● Slimmed/minified (minToolkit) ● gcr.io/distroless ● Chainguard Images ● Slim and scratch bases 

Docker ● ❌ docker exec ● ✅ nsenter ● ✅ docker debug (beta/paid) containerd ● ❌ nerdctl/ctr exec ● ✅ nsenter Kubernetes ● ❌ kubectl exec ● ✅ kubectl debug ● ✅ SSH + nsenter* What if it’s a minimal container image… with no shell? 

🤩 docker exec ● Can run target binaries, see the target rootfs "as is", install extra tools ● …but only works if there is a shell 🙂 kubectl debug / docker run (aka debugging sidecar) ● Can see target processes, access target rootfs (but not “as is”), use custom debugger image ● …but cannot see volumes, run as user, or fine-tune the ephemeral container spec, see target rootfs "as is" 🤯 nsenter ● Can do anything ● …given you know how and have sufficient access (on the node) Developer Experience During Debugging 

From docker exec ● Run target binaries ● See the target rootfs "as is" From kubectl debug ● Works even with shell-less targets ● Can bring you own debugging toolkit Can We Combine the Best of Two Worlds? 

Since we’re writing a new tool, we can try providing the same UX for all runtimes supported runtimes! Bonus: Uniform debugging UX across runtimes $ tool debug --runtime docker --target [cmd] $ tool debug --runtime containerd --target [cmd] $ tool debug --runtime k8s --pod --target [cmd] 

The Tech & Design 

What actually happens when you docker exec 

● https://github.com/opencontainers/runtime-spec/issues/345 ● https://github.com/opencontainers/runtime-spec/pull/388 What actually happens when you docker exec 💡 Fun fact - The OCI Runtime Spec doesn't define the exec command. Check out the issues #345 and #388 for an interesting discussion of how the exec functionality is actually redundant and can be reproduced in runtimes implementing only create and start commands. 

$ docker run \ --net container:app \ --ipc container:app \ --pid container:app \ busybox Reproducing docker exec with docker run 

Reproducing docker exec with docker run 

What actually happens when you kubectl debug 

Unfortunately, mount namespace cannot be shared. Let’s see if we are clever enough to overcome that! Gotcha: But My rootfs Looks Different! 

Accessing Target rootfs via /proc//root Since pid namespace is shared, we can use the /proc//root (e.g., /proc/1/root) trick to browse target rootfs! 

Adding chroot, symlinks and $PATH to the mix! 

The Implementation 

High Level Algorithm (all runtimes) ● Create the sidecar container (connected to the target container) ● Start the sidecar container ● Attach the terminal to its stdio streams ● Wait for the sidecar container to exit Sounds too easy? Now repeat it for every runtime! 

Gotcha: Will it Always Work? ● The chroot trick requires the root user and sufficient privileges ○ May need to run the sidecar with --privileged flag ○ If the Pod spec mandates runAsNonRoot, don’t chroot! ● The debugger toolkit image has to be “portable enough” (more later) 

Docker API 

Relevant Client Interfaces and Methods ● ContainerAPIClient interface ● ContainerCreate ● ContainerStart ● ContainerAttach ● ContainerWait Notes ● Configuring the namespace modes - key for connecting to target container! 

API github.com/docker/docker/client 

pkg/app/master/command/debug/handle_docker_runtime.go 

ContainerD API 

Relevant Client Interfaces and Methods ● client.NewContainer ● Container.NewTask ● Task.Start ● Task.HandleConsoleResize ● Task.Wait Notes / Challenges ● Much more lower-level (compared to Docker) ● Doesn’t provide a facade client interface ● Requires wrapping one’s head around it ● Debugger Container Spec Namespaces - key for connecting to target container! 

API github.com/containerd/containerd 

pkg/app/master/command/debug/handle_containerd_runtime.go 

Kubernetes API 

Relevant Client Interfaces and Methods ● core/v1.PodInterface.UpdateEphemeralContainers() or lower level core/v1.PodInterface.Patch/Put() to add the ephemeral container to the Pod’s spec ● core/v1.PodInterface.Wait() ● rest.Interface.Post() .Resource("pods").SubResource("attach") Notes ● TargetContainerName in EphemeralContainer - key for connecting to target container! ● Learning how to initialize and use K8s client-go library is a challenge on its own! 

API: github.com/kubernetes/client-go 

pkg/app/master/command/debug/handle_kubernetes_runtime.go 

The Debugger Images 

Gotchas and Considerations With the chroot trick (beware of portability issues) ● busybox:musl - simple and portable ● Nix and nixery.dev ...and we also can use the target image's binaries "as is" Without the chroot+PATH+symlink trick (simpler) ● NetShoot, KoolKit or general ubuntu/debian/etc are good candidates ...but we cannot invoke binaries from the target image easily 

pkg/app/master/command/debug/debug_images.go 

The Challenges 

Reverse Engineering APIs and Client Code ● Not much documentation & examples …but you can find some inspiration in ● github.com/containerd/nerdctl ● github.com/docker/cli ● github.com/kubernetes/kubectl ● github.com/iximiuz/client-go-examples ● https://github.com/iximiuz/cdebug Pro Tip: Use GitHub code search - in highly-specialized areas, it’s still more helpful than ChatGPT. 

Other Gotchas Cases that are trickier to debug ● Tightened security contexts ● Non-root containers ● Picking the portable debugging toolkit 

The Demo 

No content

No content

No content

The Code 

The Debug Command Code ● https://github.com/mintoolkit/mint - main repo ● https://github.com/mintoolkit/mint/tree/master/pkg/app/master/comman d/debug - the “debug” CLI command code ● https://github.com/mintoolkit/mint/blob/master/pkg/app/master/comma nd/debug/handle_docker_runtime.go - Docker runtime support ● https://github.com/mintoolkit/mint/blob/master/pkg/app/master/comma nd/debug/handle_kubernetes_runtime.go - K8s runtime support ● https://github.com/mintoolkit/mint/blob/master/pkg/app/master/comma nd/debug/handle_containerd_runtime.go - ContainerD runtime support 

Thank You! 

Thank You for attending! You can connect with us: @kcqon @SaiyamPathak Share your feedback!