Kubernetes is a container orchestration framework that is increasingly widely used in enterprise and elsewhere. While the industry is starting to pay some attention to Kubernetes security, there are many attack paths that aren’t well-documented, and are rarely discussed. This lack of information can make your clusters vulnerable.
In this live demonstration-filled talk presented at Black Hat USA 2019, Ian Coldwater and Duffie Cooley walk through the Kubernetes control plane before using sigs.k8s.io/kind to show some of the attack surface exposed by a default configuration of Kubernetes. There will be multiple exploits, including cluster takeovers and host escapes. We’ll show you mitigations, and then show you how to get around those.
The audience will walk away from this talk with a better understanding of Kubernetes’ default attack surface, how it can be exploited, and how to keep their clusters safer.
THE PATH LESS TRAVELED
Abusing Kubernetes Defaults
• Ian Coldwater is a Lead Platform Security
Engineer at Heroku, who specializes in
hacking and hardening Kubernetes,
containers and cloud infrastructure.
• Dufﬁe Cooley is a Staff Cloud Native Architect
at VMWare. He likes to talk about Kubernetes
and distributed systems. He is a presenter on
tgik.io a weekly video blog on Kubernetes
ecosystem. He can be found in most places as
CHECK YOUR ASSUMPTIONS
• Kubernetes is not secure by default.
• Nor are the applications deployed upon it.
A WORD ON DEFAULTS
• There is no singular set of Kubernetes defaults
• Kubernetes requires conﬁg, and every cluster is likely to be conﬁgured differently
• This talk uses a cluster with defaults deﬁned by kubeadm
WHAT IS KUBERNETES?
• Most widely used container orchestrator, with rapid rates of adoption and change
• Kubernetes is both a distributed system and an API-based platform
• These are both attack surfaces, but different ones
THE KUBERNETES API
• kubectl is a CLI tool bundled with Kubernetes.
• kubectl explain lets you explore the api interactively
• kubectl apply -f https://some/manifest.yaml applies conﬁg
HOW THE PARTS MOVE
direct schedule with hostPath and hostNetwork to grab cluster admin token from etcd
● Users can create pods with wild permissions by default
● Scheduling is not a security boundary
● Namespace isolation is not always enough
● Mitigations: encrypt etcd secrets at rest, and don’t run a kubelet on control plane nodes
Kubernetes documentation refers to hostPath as a “powerful escape hatch.” No kidding.
DOCKER IN DOCKER
• Allows users to build containers inside of Kubernetes
• Allows attackers to escalate privileges from an unprivileged pod
• Docker socket lives at /var/run/docker.sock
• Other container runtimes have sockets too
Docker in Docker
● Docker in Docker gives a lot of access to the underlying host.
● Docker is an unauthenticated api with full access to the kernel when used it in this way.
● Mitigations: Restrict the use of hostPath with admission control.
• Kubernetes is a container orchestrator. Attacks on containers still work!
• Understanding how containers work can be helpful for both attackers and defenders.
• Containers are made of Linux primitives. Attacking containers is attacking Linux.
WHAT IS A CONTAINER?
• Single process on a shared host, controlled by cgroups, isolated by namespaces
• These primitives aren’t new tech. Containers seem simple on the outside, but all that old
tech still lives in the background. That’s where the attack surface lives.
• Capabilities and other ways to attack the Linux kernel still apply
• Shared resources make for a shared attack surface!
WHAT ARE CONTAINERS MADE OF?
ONE TWEET TO ROOT
k8s root via nsenter
● Restrict the use of “privileged” Containers.
● Restrict the use of hostPID.
● nsenter is a very powerful tool that can be used to access any process on the host system
CAN WE FIX THIS?
Yes we can! Mostly.
● Admission control is your only line of defense!
● Lets you limit what a user or controller can do in depth
● Can be used to validate or mutate on admission
Pod Security Policy
● Pod Security Policies provide a granular way to deﬁne what a pod can do.
● They are an admission controller that can mutate or validate pods.
● PSP is hard to setup and adopt.
● With constraint comes a loss of agility.
...AND ONE MORE THING
Can admission control stand up to a static pod?
COMING FULL CIRCLE
• We’re really not trying to scare you here.
• Kubernetes is powerful and complex, with a lot of moving parts and a few gotchas.
• It is possible to make Kubernetes more secure!
• We need your help to do that.
• Kubernetes is an open source project that could use more security-minded contributors!
• Vulnerability disclosure info: k8s.io/security
• Kubernetes is getting more serious about the security of the project!
• Third party code review and ﬁndings: git.io/k8s-audit
• Bug Bounty coming soon.
BLACK HAT SOUND BYTES
• Check your assumptions! Kubernetes is not secure by default.
• Kubernetes is complex, with many moving parts and some unexpected behavior.
Understanding how the system works can help you both as an operator and an attacker.
• It is possible to make Kubernetes more secure, but you have to put in the work!
Put admission control on your clusters, and get involved in the Kubernetes project.