The Path Less Traveled: Abusing Kubernetes Defaults

The Path Less Traveled: Abusing Kubernetes Defaults

Kubernetes is a container orchestration framework that is increasingly widely used in enterprise and elsewhere. While the industry is starting to pay some attention to Kubernetes security, there are many attack paths that aren’t well-documented, and are rarely discussed. This lack of information can make your clusters vulnerable.

In this live demonstration-filled talk presented at Black Hat USA 2019, Ian Coldwater and Duffie Cooley walk through the Kubernetes control plane before using sigs.k8s.io/kind to show some of the attack surface exposed by a default configuration of Kubernetes. There will be multiple exploits, including cluster takeovers and host escapes. We’ll show you mitigations, and then show you how to get around those.

The audience will walk away from this talk with a better understanding of Kubernetes’ default attack surface, how it can be exploited, and how to keep their clusters safer.

A6ff6fda1b44b5d02a970be24bd79f0b?s=128

Ian Coldwater

August 07, 2019
Tweet

Transcript

  1. THE PATH LESS TRAVELED Abusing Kubernetes Defaults

  2. • Ian Coldwater is a Lead Platform Security Engineer at

    Heroku, who specializes in hacking and hardening Kubernetes, containers and cloud infrastructure. • Duffie Cooley is a Staff Cloud Native Architect at VMWare. He likes to talk about Kubernetes and distributed systems. He is a presenter on tgik.io a weekly video blog on Kubernetes ecosystem. He can be found in most places as @mauilion Twitter @mauilion Twitter @IanColdwater #BHUSA
  3. CHECK YOUR ASSUMPTIONS • Kubernetes is not secure by default.

    • Nor are the applications deployed upon it.
  4. A WORD ON DEFAULTS • There is no singular set

    of Kubernetes defaults • Kubernetes requires config, and every cluster is likely to be configured differently • This talk uses a cluster with defaults defined by kubeadm
  5. WHAT IS KUBERNETES? • Most widely used container orchestrator, with

    rapid rates of adoption and change • Kubernetes is both a distributed system and an API-based platform • These are both attack surfaces, but different ones
  6. HOW DOES KUBERNETES WORK?

  7. KUBERNETES COMPONENTS

  8. THE KUBERNETES API • kubectl is a CLI tool bundled

    with Kubernetes. • kubectl explain lets you explore the api interactively • kubectl apply -f https://some/manifest.yaml applies config
  9. HOW THE PARTS MOVE

  10. DEMO direct schedule with hostPath and hostNetwork to grab cluster

    admin token from etcd Link: https://asciinema.org/a/261376 10
  11. DEMO TAKEAWAYS • Users can create pods with wild permissions

    by default • Scheduling is not a security boundary • Namespace isolation is not always enough • Mitigations: encrypt etcd secrets at rest, and don’t run a kubelet on control plane nodes
  12. HOSTPATH! Kubernetes documentation refers to hostPath as a “powerful escape

    hatch.” No kidding.
  13. DOCKER IN DOCKER • Allows users to build containers inside

    of Kubernetes • Allows attackers to escalate privileges from an unprivileged pod • Docker socket lives at /var/run/docker.sock • Other container runtimes have sockets too
  14. DEMO Docker in Docker Link: https://asciinema.org/a/261373 14

  15. DEMO TAKEAWAYS • Docker in Docker gives a lot of

    access to the underlying host. • Docker is an unauthenticated api with full access to the kernel when used it in this way. • Mitigations: Restrict the use of hostPath with admission control.
  16. CONTAINER ATTACKS • Kubernetes is a container orchestrator. Attacks on

    containers still work! • Understanding how containers work can be helpful for both attackers and defenders. • Containers are made of Linux primitives. Attacking containers is attacking Linux.
  17. WHAT IS A CONTAINER? • Single process on a shared

    host, controlled by cgroups, isolated by namespaces • These primitives aren’t new tech. Containers seem simple on the outside, but all that old tech still lives in the background. That’s where the attack surface lives. • Capabilities and other ways to attack the Linux kernel still apply • Shared resources make for a shared attack surface!
  18. WHAT ARE CONTAINERS MADE OF?

  19. ONE TWEET TO ROOT

  20. DEMO k8s root via nsenter link: https://asciinema.org/a/261377 20

  21. DEMO TAKEAWAYS • Restrict the use of “privileged” Containers. •

    Restrict the use of hostPID. • nsenter is a very powerful tool that can be used to access any process on the host system
  22. CAN WE FIX THIS? Yes we can! Mostly. 22

  23. ADMISSION CONTROL • Admission control is your only line of

    defense! • Lets you limit what a user or controller can do in depth • Can be used to validate or mutate on admission
  24. DEMO Pod Security Policy link: https://asciinema.org/a/261378 24

  25. DEMO TAKEAWAYS • Pod Security Policies provide a granular way

    to define what a pod can do. • They are an admission controller that can mutate or validate pods. • PSP is hard to setup and adopt. • With constraint comes a loss of agility.
  26. ...AND ONE MORE THING Can admission control stand up to

    a static pod? 26
  27. COMING FULL CIRCLE • We’re really not trying to scare

    you here. • Kubernetes is powerful and complex, with a lot of moving parts and a few gotchas. • It is possible to make Kubernetes more secure! • We need your help to do that.
  28. GET INVOLVED! • Kubernetes is an open source project that

    could use more security-minded contributors! • Vulnerability disclosure info: k8s.io/security • Kubernetes is getting more serious about the security of the project! • Third party code review and findings: git.io/k8s-audit • Bug Bounty coming soon.
  29. BLACK HAT SOUND BYTES • Check your assumptions! Kubernetes is

    not secure by default. • Kubernetes is complex, with many moving parts and some unexpected behavior. Understanding how the system works can help you both as an operator and an attacker. • It is possible to make Kubernetes more secure, but you have to put in the work! Put admission control on your clusters, and get involved in the Kubernetes project. RESOURCES: git.io/bh-kubernetes