$30 off During Our Annual Pro Sale. View Details »

The Path Less Traveled: Abusing Kubernetes Defaults

The Path Less Traveled: Abusing Kubernetes Defaults

Kubernetes is a container orchestration framework that is increasingly widely used in enterprise and elsewhere. While the industry is starting to pay some attention to Kubernetes security, there are many attack paths that aren’t well-documented, and are rarely discussed. This lack of information can make your clusters vulnerable.

In this live demonstration-filled talk presented at Black Hat USA 2019, Ian Coldwater and Duffie Cooley walk through the Kubernetes control plane before using sigs.k8s.io/kind to show some of the attack surface exposed by a default configuration of Kubernetes. There will be multiple exploits, including cluster takeovers and host escapes. We’ll show you mitigations, and then show you how to get around those.

The audience will walk away from this talk with a better understanding of Kubernetes’ default attack surface, how it can be exploited, and how to keep their clusters safer.

Ian Coldwater

August 07, 2019
Tweet

More Decks by Ian Coldwater

Other Decks in Technology

Transcript

  1. THE PATH LESS TRAVELED
    Abusing Kubernetes Defaults

    View Slide

  2. • Ian Coldwater is a Lead Platform Security
    Engineer at Heroku, who specializes in
    hacking and hardening Kubernetes,
    containers and cloud infrastructure.
    • Duffie Cooley is a Staff Cloud Native Architect
    at VMWare. He likes to talk about Kubernetes
    and distributed systems. He is a presenter on
    tgik.io a weekly video blog on Kubernetes
    ecosystem. He can be found in most places as
    @mauilion
    Twitter @mauilion
    Twitter @IanColdwater
    #BHUSA

    View Slide

  3. CHECK YOUR ASSUMPTIONS
    • Kubernetes is not secure by default.
    • Nor are the applications deployed upon it.

    View Slide

  4. A WORD ON DEFAULTS
    • There is no singular set of Kubernetes defaults
    • Kubernetes requires config, and every cluster is likely to be configured differently
    • This talk uses a cluster with defaults defined by kubeadm

    View Slide

  5. WHAT IS KUBERNETES?
    • Most widely used container orchestrator, with rapid rates of adoption and change
    • Kubernetes is both a distributed system and an API-based platform
    • These are both attack surfaces, but different ones

    View Slide

  6. HOW DOES
    KUBERNETES WORK?

    View Slide

  7. KUBERNETES COMPONENTS

    View Slide

  8. THE KUBERNETES API
    • kubectl is a CLI tool bundled with Kubernetes.
    • kubectl explain lets you explore the api interactively
    • kubectl apply -f https://some/manifest.yaml applies config

    View Slide

  9. HOW THE PARTS MOVE

    View Slide

  10. DEMO
    direct schedule with hostPath and hostNetwork to grab cluster admin token from etcd
    Link: https://asciinema.org/a/261376
    10

    View Slide

  11. DEMO TAKEAWAYS
    ● Users can create pods with wild permissions by default
    ● Scheduling is not a security boundary
    ● Namespace isolation is not always enough
    ● Mitigations: encrypt etcd secrets at rest, and don’t run a kubelet on control plane nodes

    View Slide

  12. HOSTPATH!
    Kubernetes documentation refers to hostPath as a “powerful escape hatch.” No kidding.

    View Slide

  13. DOCKER IN DOCKER
    • Allows users to build containers inside of Kubernetes
    • Allows attackers to escalate privileges from an unprivileged pod
    • Docker socket lives at /var/run/docker.sock
    • Other container runtimes have sockets too

    View Slide

  14. DEMO
    Docker in Docker
    Link: https://asciinema.org/a/261373
    14

    View Slide

  15. DEMO TAKEAWAYS
    ● Docker in Docker gives a lot of access to the underlying host.
    ● Docker is an unauthenticated api with full access to the kernel when used it in this way.
    ● Mitigations: Restrict the use of hostPath with admission control.

    View Slide

  16. CONTAINER ATTACKS
    • Kubernetes is a container orchestrator. Attacks on containers still work!
    • Understanding how containers work can be helpful for both attackers and defenders.
    • Containers are made of Linux primitives. Attacking containers is attacking Linux.

    View Slide

  17. WHAT IS A CONTAINER?
    • Single process on a shared host, controlled by cgroups, isolated by namespaces
    • These primitives aren’t new tech. Containers seem simple on the outside, but all that old
    tech still lives in the background. That’s where the attack surface lives.
    • Capabilities and other ways to attack the Linux kernel still apply
    • Shared resources make for a shared attack surface!

    View Slide

  18. WHAT ARE CONTAINERS MADE OF?

    View Slide

  19. ONE TWEET TO ROOT

    View Slide

  20. DEMO
    k8s root via nsenter
    link: https://asciinema.org/a/261377
    20

    View Slide

  21. DEMO TAKEAWAYS
    ● Restrict the use of “privileged” Containers.
    ● Restrict the use of hostPID.
    ● nsenter is a very powerful tool that can be used to access any process on the host system

    View Slide

  22. CAN WE FIX THIS?
    Yes we can! Mostly.
    22

    View Slide

  23. ADMISSION CONTROL
    ● Admission control is your only line of defense!
    ● Lets you limit what a user or controller can do in depth
    ● Can be used to validate or mutate on admission

    View Slide

  24. DEMO
    Pod Security Policy
    link: https://asciinema.org/a/261378
    24

    View Slide

  25. DEMO TAKEAWAYS
    ● Pod Security Policies provide a granular way to define what a pod can do.
    ● They are an admission controller that can mutate or validate pods.
    ● PSP is hard to setup and adopt.
    ● With constraint comes a loss of agility.

    View Slide

  26. ...AND ONE MORE THING
    Can admission control stand up to a static pod?
    26

    View Slide

  27. COMING FULL CIRCLE
    • We’re really not trying to scare you here.
    • Kubernetes is powerful and complex, with a lot of moving parts and a few gotchas.
    • It is possible to make Kubernetes more secure!
    • We need your help to do that.

    View Slide

  28. GET INVOLVED!
    • Kubernetes is an open source project that could use more security-minded contributors!
    • Vulnerability disclosure info: k8s.io/security
    • Kubernetes is getting more serious about the security of the project!
    • Third party code review and findings: git.io/k8s-audit
    • Bug Bounty coming soon.

    View Slide

  29. BLACK HAT SOUND BYTES
    • Check your assumptions! Kubernetes is not secure by default.
    • Kubernetes is complex, with many moving parts and some unexpected behavior.
    Understanding how the system works can help you both as an operator and an attacker.
    • It is possible to make Kubernetes more secure, but you have to put in the work!
    Put admission control on your clusters, and get involved in the Kubernetes project.
    RESOURCES: git.io/bh-kubernetes

    View Slide