The Path Less Traveled: Abusing Kubernetes Defaults

Kubernetes is a container orchestration framework that is increasingly widely used in enterprise and elsewhere. While the industry is starting to pay some attention to Kubernetes security, there are many attack paths that aren’t well-documented, and are rarely discussed. This lack of information can make your clusters vulnerable.

In this live demonstration-filled talk presented at Black Hat USA 2019, Ian Coldwater and Duffie Cooley walk through the Kubernetes control plane before using sigs.k8s.io/kind to show some of the attack surface exposed by a default configuration of Kubernetes. There will be multiple exploits, including cluster takeovers and host escapes. We’ll show you mitigations, and then show you how to get around those.

The audience will walk away from this talk with a better understanding of Kubernetes’ default attack surface, how it can be exploited, and how to keep their clusters safer.

Transcript

1. THE PATH LESS TRAVELED Abusing Kubernetes Defaults
2. • Ian Coldwater is a Lead Platform Security Engineer at Heroku, who specializes in hacking and hardening Kubernetes, containers and cloud infrastructure. • Duffie Cooley is a Staff Cloud Native Architect at VMWare. He likes to talk about Kubernetes and distributed systems. He is a presenter on tgik.io a weekly video blog on Kubernetes ecosystem. He can be found in most places as @mauilion Twitter @mauilion Twitter @IanColdwater #BHUSA
3. CHECK YOUR ASSUMPTIONS • Kubernetes is not secure by default. • Nor are the applications deployed upon it.
4. A WORD ON DEFAULTS • There is no singular set of Kubernetes defaults • Kubernetes requires config, and every cluster is likely to be configured differently • This talk uses a cluster with defaults defined by kubeadm
5. WHAT IS KUBERNETES? • Most widely used container orchestrator, with rapid rates of adoption and change • Kubernetes is both a distributed system and an API-based platform • These are both attack surfaces, but different ones
6. HOW DOES KUBERNETES WORK?
7. KUBERNETES COMPONENTS
8. THE KUBERNETES API • kubectl is a CLI tool bundled with Kubernetes. • kubectl explain lets you explore the api interactively • kubectl apply -f https://some/manifest.yaml applies config
9. HOW THE PARTS MOVE
10. DEMO direct schedule with hostPath and hostNetwork to grab cluster admin token from etcd Link: https://asciinema.org/a/261376 10
11. DEMO TAKEAWAYS • Users can create pods with wild permissions by default • Scheduling is not a security boundary • Namespace isolation is not always enough • Mitigations: encrypt etcd secrets at rest, and don’t run a kubelet on control plane nodes
12. HOSTPATH! Kubernetes documentation refers to hostPath as a “powerful escape hatch.” No kidding.
13. DOCKER IN DOCKER • Allows users to build containers inside of Kubernetes • Allows attackers to escalate privileges from an unprivileged pod • Docker socket lives at /var/run/docker.sock • Other container runtimes have sockets too
14. DEMO Docker in Docker Link: https://asciinema.org/a/261373 14
15. DEMO TAKEAWAYS • Docker in Docker gives a lot of access to the underlying host. • Docker is an unauthenticated api with full access to the kernel when used it in this way. • Mitigations: Restrict the use of hostPath with admission control.
16. CONTAINER ATTACKS • Kubernetes is a container orchestrator. Attacks on containers still work! • Understanding how containers work can be helpful for both attackers and defenders. • Containers are made of Linux primitives. Attacking containers is attacking Linux.
17. WHAT IS A CONTAINER? • Single process on a shared host, controlled by cgroups, isolated by namespaces • These primitives aren’t new tech. Containers seem simple on the outside, but all that old tech still lives in the background. That’s where the attack surface lives. • Capabilities and other ways to attack the Linux kernel still apply • Shared resources make for a shared attack surface!
18. WHAT ARE CONTAINERS MADE OF?
19. ONE TWEET TO ROOT
20. DEMO k8s root via nsenter link: https://asciinema.org/a/261377 20
21. DEMO TAKEAWAYS • Restrict the use of “privileged” Containers. • Restrict the use of hostPID. • nsenter is a very powerful tool that can be used to access any process on the host system
22. CAN WE FIX THIS? Yes we can! Mostly. 22
23. ADMISSION CONTROL • Admission control is your only line of defense! • Lets you limit what a user or controller can do in depth • Can be used to validate or mutate on admission
24. DEMO Pod Security Policy link: https://asciinema.org/a/261378 24
25. DEMO TAKEAWAYS • Pod Security Policies provide a granular way to define what a pod can do. • They are an admission controller that can mutate or validate pods. • PSP is hard to setup and adopt. • With constraint comes a loss of agility.
26. ...AND ONE MORE THING Can admission control stand up to a static pod? 26
27. COMING FULL CIRCLE • We’re really not trying to scare you here. • Kubernetes is powerful and complex, with a lot of moving parts and a few gotchas. • It is possible to make Kubernetes more secure! • We need your help to do that.
28. GET INVOLVED! • Kubernetes is an open source project that could use more security-minded contributors! • Vulnerability disclosure info: k8s.io/security • Kubernetes is getting more serious about the security of the project! • Third party code review and findings: git.io/k8s-audit • Bug Bounty coming soon.
29. BLACK HAT SOUND BYTES • Check your assumptions! Kubernetes is not secure by default. • Kubernetes is complex, with many moving parts and some unexpected behavior. Understanding how the system works can help you both as an operator and an attacker. • It is possible to make Kubernetes more secure, but you have to put in the work! Put admission control on your clusters, and get involved in the Kubernetes project. RESOURCES: git.io/bh-kubernetes