THE PATH LESS TRAVELED Abusing Kubernetes Defaults

• Ian Coldwater is a Lead Platform Security Engineer at Heroku, who specializes in hacking and hardening Kubernetes, containers and cloud infrastructure. • Duffie Cooley is a Staff Cloud Native Architect at VMWare. He likes to talk about Kubernetes and distributed systems. He is a presenter on tgik.io a weekly video blog on Kubernetes ecosystem. He can be found in most places as @mauilion Twitter @mauilion Twitter @IanColdwater #BHUSA

CHECK YOUR ASSUMPTIONS • Kubernetes is not secure by default. • Nor are the applications deployed upon it.

A WORD ON DEFAULTS • There is no singular set of Kubernetes defaults • Kubernetes requires config, and every cluster is likely to be configured differently • This talk uses a cluster with defaults defined by kubeadm

WHAT IS KUBERNETES? • Most widely used container orchestrator, with rapid rates of adoption and change • Kubernetes is both a distributed system and an API-based platform • These are both attack surfaces, but different ones

HOW DOES KUBERNETES WORK?

KUBERNETES COMPONENTS

THE KUBERNETES API • kubectl is a CLI tool bundled with Kubernetes. • kubectl explain lets you explore the api interactively • kubectl apply -f https://some/manifest.yaml applies config

HOW THE PARTS MOVE

DEMO direct schedule with hostPath and hostNetwork to grab cluster admin token from etcd Link: https://asciinema.org/a/261376 10

DEMO TAKEAWAYS ● Users can create pods with wild permissions by default ● Scheduling is not a security boundary ● Namespace isolation is not always enough ● Mitigations: encrypt etcd secrets at rest, and don’t run a kubelet on control plane nodes

HOSTPATH! Kubernetes documentation refers to hostPath as a “powerful escape hatch.” No kidding.

DOCKER IN DOCKER • Allows users to build containers inside of Kubernetes • Allows attackers to escalate privileges from an unprivileged pod • Docker socket lives at /var/run/docker.sock • Other container runtimes have sockets too

DEMO Docker in Docker Link: https://asciinema.org/a/261373 14

DEMO TAKEAWAYS ● Docker in Docker gives a lot of access to the underlying host. ● Docker is an unauthenticated api with full access to the kernel when used it in this way. ● Mitigations: Restrict the use of hostPath with admission control.

CONTAINER ATTACKS • Kubernetes is a container orchestrator. Attacks on containers still work! • Understanding how containers work can be helpful for both attackers and defenders. • Containers are made of Linux primitives. Attacking containers is attacking Linux.

WHAT IS A CONTAINER? • Single process on a shared host, controlled by cgroups, isolated by namespaces • These primitives aren’t new tech. Containers seem simple on the outside, but all that old tech still lives in the background. That’s where the attack surface lives. • Capabilities and other ways to attack the Linux kernel still apply • Shared resources make for a shared attack surface!

WHAT ARE CONTAINERS MADE OF?

ONE TWEET TO ROOT

DEMO k8s root via nsenter link: https://asciinema.org/a/261377 20

DEMO TAKEAWAYS ● Restrict the use of “privileged” Containers. ● Restrict the use of hostPID. ● nsenter is a very powerful tool that can be used to access any process on the host system

CAN WE FIX THIS? Yes we can! Mostly. 22

ADMISSION CONTROL ● Admission control is your only line of defense! ● Lets you limit what a user or controller can do in depth ● Can be used to validate or mutate on admission

DEMO Pod Security Policy link: https://asciinema.org/a/261378 24

DEMO TAKEAWAYS ● Pod Security Policies provide a granular way to define what a pod can do. ● They are an admission controller that can mutate or validate pods. ● PSP is hard to setup and adopt. ● With constraint comes a loss of agility.

...AND ONE MORE THING Can admission control stand up to a static pod? 26

COMING FULL CIRCLE • We’re really not trying to scare you here. • Kubernetes is powerful and complex, with a lot of moving parts and a few gotchas. • It is possible to make Kubernetes more secure! • We need your help to do that.

GET INVOLVED! • Kubernetes is an open source project that could use more security-minded contributors! • Vulnerability disclosure info: k8s.io/security • Kubernetes is getting more serious about the security of the project! • Third party code review and findings: git.io/k8s-audit • Bug Bounty coming soon.

BLACK HAT SOUND BYTES • Check your assumptions! Kubernetes is not secure by default. • Kubernetes is complex, with many moving parts and some unexpected behavior. Understanding how the system works can help you both as an operator and an attacker. • It is possible to make Kubernetes more secure, but you have to put in the work! Put admission control on your clusters, and get involved in the Kubernetes project. RESOURCES: git.io/bh-kubernetes