The Path Less Traveled: Abusing Kubernetes Defaults

Kubernetes is a container orchestration framework that is increasingly widely used in enterprise and elsewhere. While the industry is starting to pay some attention to Kubernetes security, there are many attack paths that aren’t well-documented, and are rarely discussed. This lack of information can make your clusters vulnerable.

In this live demonstration-filled talk presented at Black Hat USA 2019, Ian Coldwater and Duffie Cooley walk through the Kubernetes control plane before using sigs.k8s.io/kind to show some of the attack surface exposed by a default configuration of Kubernetes. There will be multiple exploits, including cluster takeovers and host escapes. We’ll show you mitigations, and then show you how to get around those.

The audience will walk away from this talk with a better understanding of Kubernetes’ default attack surface, how it can be exploited, and how to keep their clusters safer.

Transcript

1. THE PATH LESS TRAVELED
   Abusing Kubernetes Defaults
   

   View Slide

2. • Ian Coldwater is a Lead Platform Security
   Engineer at Heroku, who specializes in
   hacking and hardening Kubernetes,
   containers and cloud infrastructure.
   • Duffie Cooley is a Staff Cloud Native Architect
   at VMWare. He likes to talk about Kubernetes
   and distributed systems. He is a presenter on
   tgik.io a weekly video blog on Kubernetes
   ecosystem. He can be found in most places as
   @mauilion
   Twitter @mauilion
   Twitter @IanColdwater
   #BHUSA
   

   View Slide

3. CHECK YOUR ASSUMPTIONS
   • Kubernetes is not secure by default.
   • Nor are the applications deployed upon it.
   

   View Slide

4. A WORD ON DEFAULTS
   • There is no singular set of Kubernetes defaults
   • Kubernetes requires config, and every cluster is likely to be configured differently
   • This talk uses a cluster with defaults defined by kubeadm
   

   View Slide

5. WHAT IS KUBERNETES?
   • Most widely used container orchestrator, with rapid rates of adoption and change
   • Kubernetes is both a distributed system and an API-based platform
   • These are both attack surfaces, but different ones
   

   View Slide

6. HOW DOES
   KUBERNETES WORK?
   

   View Slide

7. KUBERNETES COMPONENTS
   

   View Slide

8. THE KUBERNETES API
   • kubectl is a CLI tool bundled with Kubernetes.
   • kubectl explain lets you explore the api interactively
   • kubectl apply -f https://some/manifest.yaml applies config
   

   View Slide

9. HOW THE PARTS MOVE
   

   View Slide

10. DEMO
    direct schedule with hostPath and hostNetwork to grab cluster admin token from etcd
    Link: https://asciinema.org/a/261376
    10
    

    View Slide

11. DEMO TAKEAWAYS
    ● Users can create pods with wild permissions by default
    ● Scheduling is not a security boundary
    ● Namespace isolation is not always enough
    ● Mitigations: encrypt etcd secrets at rest, and don’t run a kubelet on control plane nodes
    

    View Slide

12. HOSTPATH!
    Kubernetes documentation refers to hostPath as a “powerful escape hatch.” No kidding.
    

    View Slide

13. DOCKER IN DOCKER
    • Allows users to build containers inside of Kubernetes
    • Allows attackers to escalate privileges from an unprivileged pod
    • Docker socket lives at /var/run/docker.sock
    • Other container runtimes have sockets too
    

    View Slide

14. DEMO
    Docker in Docker
    Link: https://asciinema.org/a/261373
    14
    

    View Slide

15. DEMO TAKEAWAYS
    ● Docker in Docker gives a lot of access to the underlying host.
    ● Docker is an unauthenticated api with full access to the kernel when used it in this way.
    ● Mitigations: Restrict the use of hostPath with admission control.
    

    View Slide

16. CONTAINER ATTACKS
    • Kubernetes is a container orchestrator. Attacks on containers still work!
    • Understanding how containers work can be helpful for both attackers and defenders.
    • Containers are made of Linux primitives. Attacking containers is attacking Linux.
    

    View Slide

17. WHAT IS A CONTAINER?
    • Single process on a shared host, controlled by cgroups, isolated by namespaces
    • These primitives aren’t new tech. Containers seem simple on the outside, but all that old
    tech still lives in the background. That’s where the attack surface lives.
    • Capabilities and other ways to attack the Linux kernel still apply
    • Shared resources make for a shared attack surface!
    

    View Slide

18. WHAT ARE CONTAINERS MADE OF?
    

    View Slide

19. ONE TWEET TO ROOT
    

    View Slide

20. DEMO
    k8s root via nsenter
    link: https://asciinema.org/a/261377
    20
    

    View Slide

21. DEMO TAKEAWAYS
    ● Restrict the use of “privileged” Containers.
    ● Restrict the use of hostPID.
    ● nsenter is a very powerful tool that can be used to access any process on the host system
    

    View Slide

22. CAN WE FIX THIS?
    Yes we can! Mostly.
    22
    

    View Slide

23. ADMISSION CONTROL
    ● Admission control is your only line of defense!
    ● Lets you limit what a user or controller can do in depth
    ● Can be used to validate or mutate on admission
    

    View Slide

24. DEMO
    Pod Security Policy
    link: https://asciinema.org/a/261378
    24
    

    View Slide

25. DEMO TAKEAWAYS
    ● Pod Security Policies provide a granular way to define what a pod can do.
    ● They are an admission controller that can mutate or validate pods.
    ● PSP is hard to setup and adopt.
    ● With constraint comes a loss of agility.
    

    View Slide

26. ...AND ONE MORE THING
    Can admission control stand up to a static pod?
    26
    

    View Slide

27. COMING FULL CIRCLE
    • We’re really not trying to scare you here.
    • Kubernetes is powerful and complex, with a lot of moving parts and a few gotchas.
    • It is possible to make Kubernetes more secure!
    • We need your help to do that.
    

    View Slide

28. GET INVOLVED!
    • Kubernetes is an open source project that could use more security-minded contributors!
    • Vulnerability disclosure info: k8s.io/security
    • Kubernetes is getting more serious about the security of the project!
    • Third party code review and findings: git.io/k8s-audit
    • Bug Bounty coming soon.
    

    View Slide

29. BLACK HAT SOUND BYTES
    • Check your assumptions! Kubernetes is not secure by default.
    • Kubernetes is complex, with many moving parts and some unexpected behavior.
    Understanding how the system works can help you both as an operator and an attacker.
    • It is possible to make Kubernetes more secure, but you have to put in the work!
    Put admission control on your clusters, and get involved in the Kubernetes project.
    RESOURCES: git.io/bh-kubernetes
    

    View Slide