Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Enemy Within: Running untrusted code in Kubernetes

01dc8e954957a10b428aa60b28c89d52?s=47 Ian Lewis
November 17, 2019

The Enemy Within: Running untrusted code in Kubernetes

Containers are a great way to deploy and isolate application resources but they can fall short when it comes to security isolation. How do you improve the security of a container while maintaining the flexible and dynamic resource usage of a container? There are many options for sandbox containers but which is right for you?

In this talk I will explore sandbox runtimes in depth with a focus on use-cases and challenges on their implementation and maintenance. I will dive into the container security model, the use cases for sandbox pods. I will discuss various approaches and their tradeoffs before diving into the architecture of gVisor, how it differs from virtual machine based sandboxes, and how we are working to make running untrusted code feel more like the containers you know and love. Finally, I will bring it all together with a demo of best practices for using gVisor to run untrusted user code in a Kubernetes cluster.


Ian Lewis

November 17, 2019


  1. Ian Lewis Developer Advocate, Google Cloud Platform The Enemy Within

    Running Untrusted Code in Kubernetes
  2. 2 gVisor Ian Lewis (@IanMLewis) Developer Advocate, Google

  3. 3 gVisor • Running untrusted code • Third-party/User uploaded code

    ◦ Vendor-built apps ◦ SaaS/Serverless • Complex code/Complex user input ◦ Video/Image transcoding ◦ Machine learning • Code you wrote but you don't trust yourself…. So you want to run some code...
  4. 4 gVisor Too much privileged code Application Host Kernel

  5. 5 gVisor Too much privileged code Application Host Kernel open("/path/to/file",

  6. 6 gVisor Too much privileged code Application Host Kernel

  7. 7 gVisor Too much privileged code Application Host Kernel file

  8. 8 gVisor Too much privileged code Application Host Kernel Application

  9. 9 gVisor Too much privileged code Host Kernel open("/../\x00...", O_RDWR)

    Application Application
  10. 10 gVisor Too much privileged code Host Kernel Application Application

  11. 11 gVisor Too much privileged code Host Kernel Application Application

  12. 12 gVisor

  13. 13 gVisor • Protects attackers from escaping the runtime environment

    • Code running in the sandbox is untrusted • Goal: Reduce the attack surface for untrusted code • Method: Reduce the amount of trusted or privileged code (e.g. kernel code) that gets run. ◦ Achieved through abstraction/virtualization of host. Container Sandboxes
  14. 14 gVisor Containers & Seccomp sandboxes Application Host Kernel Namespace

  15. 15 gVisor • Pros ◦ Same performance as normal processes

    ◦ Uses built-in OS features. • Cons ◦ They aren't good security isolation boundaries ◦ Any one bug in the host kernel could lead to a full host compromise ◦ Seccomp sandboxes must be tweaked per application Containers & Seccomp sandboxes
  16. 16 gVisor Unikernels Application Host Kernel Guest OS Hypervisor

  17. 17 gVisor • Pros ◦ Reasonably fast ◦ Low overhead/startup

    ◦ Good compatibility • Cons ◦ Can't bring your own container (must be specially crafted) ◦ Many limitations (fork, dynamic loading of libraries, etc.) Unikernels
  18. 18 gVisor Nabla Containers (nabla-containers.github.io)

  19. 19 gVisor Virtual Machines Application OS Virtualized Hardware

  20. 20 gVisor • Pros ◦ Hardware acceleration. Raw-performance is good

    ◦ Maintenance is well understood • Cons ◦ Security implications often not well understood ◦ Nested virtualization slow & source of security bugs ◦ Resource management difficult ◦ Often requires layers of indirection and proxies Virtual Machines
  21. 21 gVisor Kata Containers (katacontainers.io)

  22. 22 gVisor • Hybrid approach = Virtualization w/ Host Resources

    • Uses the same principle of virtualization as VMs ◦ Virtualization at the OS; Linux Syscall layer • Reduces the host attack surface ◦ Calls to the host OS are controlled by the Sentry ◦ Most syscall logic handled by Sentry ◦ No syscalls are "passed through". Applications cannot pass arbitrary arguments to the host kernel. gVisor (gvisor.dev)
  23. 23 gVisor Hardware Virtualization Application OS Virtualized Hardware

  24. 24 gVisor OS Virtualization Application Virtualized OS

  25. 25 gVisor gVisor (gvisor.dev) KVM/ptrace Gofer Gofer Gofers Containers Containers

    Host Linux Kernel Containers Sentry Sandbox User Kernel 9P runsc OCI seccomp + ns seccomp + ns
  26. 26 gVisor • Container-Native ◦ Built from the ground up

    for containers ◦ Host process - resource limits, secrets, volumes, etc. • Portable ◦ Can run inside of VMs ◦ Doesn't need hardware support but can use it if available • Resource Efficient ◦ Able to use the host OS for resource management ◦ Workloads can be tightly packed gVisor (gvisor.dev)
  27. 27 gVisor • Cons ◦ Compatibility (always getting better) ◦

    I/O Performance (also being improved) gVisor (gvisor.dev)
  28. 28 gVisor apiVersion: node.k8s.io/v1beta1 kind: RuntimeClass metadata: name: myclass #

    The name of the corresponding CRI configuration handler: myhandler Runtimes & Kubernetes
  29. 29 gVisor Runtimes & Kubernetes myhandler shim OCI Kubernetes myhandler

    runtime CRI handler: myhandler
  30. 30 gVisor • Be aware of defaults ◦ K8s is

    optimized for ease-of-use, not security ◦ CPU/Memory/Disk limits • Network/Disk isolation ◦ Network access: Use NetworkPolicy ◦ Arbitrary packet injection ◦ File writes/permissions: Use read-only filesystems ◦ No throttling mechanism: use cgroups What's not protected?
  31. 31 gVisor gVisor https://gvisor.dev/ Gitter: https://gitter.im/gvisor/community Mailing lists: gvisor-users, gvisor-dev

    Thanks! https://katacontainers.io/ Nabla Containers https://nabla-containers.github.io/