The Enemy Within: Running untrusted code in Kubernetes

Containers are a great way to deploy and isolate application resources but they can fall short when it comes to security isolation. How do you improve the security of a container while maintaining the flexible and dynamic resource usage of a container? There are many options for sandbox containers but which is right for you?

In this talk I will explore sandbox runtimes in depth with a focus on use-cases and challenges on their implementation and maintenance. I will dive into the container security model, the use cases for sandbox pods. I will discuss various approaches and their tradeoffs before diving into the architecture of gVisor, how it differs from virtual machine based sandboxes, and how we are working to make running untrusted code feel more like the containers you know and love. Finally, I will bring it all together with a demo of best practices for using gVisor to run untrusted user code in a Kubernetes cluster.

Transcript

1. Ian Lewis
   Developer Advocate, Google Cloud Platform
   The Enemy Within
   Running Untrusted Code
   in Kubernetes
   

   View Slide

2. 2
   gVisor
   Ian Lewis (@IanMLewis)
   Developer Advocate, Google
   

   View Slide

3. 3
   gVisor
   ● Running untrusted code
   ● Third-party/User uploaded code
   ○ Vendor-built apps
   ○ SaaS/Serverless
   ● Complex code/Complex user input
   ○ Video/Image transcoding
   ○ Machine learning
   ● Code you wrote but you don't trust yourself….
   So you want to run some code...
   

   View Slide

4. 4
   gVisor
   Too much privileged code
   Application
   Host Kernel
   

   View Slide

5. 5
   gVisor
   Too much privileged code
   Application
   Host Kernel
   open("/path/to/file", O_RDWR)
   

   View Slide

6. 6
   gVisor
   Too much privileged code
   Application
   Host Kernel
   

   View Slide

7. 7
   gVisor
   Too much privileged code
   Application
   Host Kernel
   file descriptor
   

   View Slide

8. 8
   gVisor
   Too much privileged code
   Application
   Host Kernel
   Application
   

   View Slide

9. 9
   gVisor
   Too much privileged code
   Host Kernel
   open("/../\x00...", O_RDWR)
   Application Application
   

   View Slide

10. 10
    gVisor
    Too much privileged code
    Host Kernel
    Application Application
    

    View Slide

11. 11
    gVisor
    Too much privileged code
    Host Kernel
    Application Application
    

    View Slide

12. 12
    gVisor
    

    View Slide

13. 13
    gVisor
    ● Protects attackers from escaping the runtime environment
    ● Code running in the sandbox is untrusted
    ● Goal: Reduce the attack surface for untrusted code
    ● Method: Reduce the amount of trusted or privileged code (e.g. kernel
    code) that gets run.
    ○ Achieved through abstraction/virtualization of host.
    Container Sandboxes
    

    View Slide

14. 14
    gVisor
    Containers & Seccomp sandboxes
    Application
    Host Kernel
    Namespace
    

    View Slide

15. 15
    gVisor
    ● Pros
    ○ Same performance as normal processes
    ○ Uses built-in OS features.
    ● Cons
    ○ They aren't good security isolation boundaries
    ○ Any one bug in the host kernel could lead to a full host compromise
    ○ Seccomp sandboxes must be tweaked per application
    Containers & Seccomp sandboxes
    

    View Slide

16. 16
    gVisor
    Unikernels
    Application
    Host Kernel
    Guest OS
    Hypervisor
    

    View Slide

17. 17
    gVisor
    ● Pros
    ○ Reasonably fast
    ○ Low overhead/startup
    ○ Good compatibility
    ● Cons
    ○ Can't bring your own container (must be specially crafted)
    ○ Many limitations (fork, dynamic loading of libraries, etc.)
    Unikernels
    

    View Slide

18. 18
    gVisor
    Nabla Containers (nabla-containers.github.io)
    

    View Slide

19. 19
    gVisor
    Virtual Machines
    Application
    OS
    Virtualized Hardware
    

    View Slide

20. 20
    gVisor
    ● Pros
    ○ Hardware acceleration. Raw-performance is good
    ○ Maintenance is well understood
    ● Cons
    ○ Security implications often not well understood
    ○ Nested virtualization slow & source of security bugs
    ○ Resource management difficult
    ○ Often requires layers of indirection and proxies
    Virtual Machines
    

    View Slide

21. 21
    gVisor
    Kata Containers (katacontainers.io)
    

    View Slide

22. 22
    gVisor
    ● Hybrid approach = Virtualization w/ Host Resources
    ● Uses the same principle of virtualization as VMs
    ○ Virtualization at the OS; Linux Syscall layer
    ● Reduces the host attack surface
    ○ Calls to the host OS are controlled by the Sentry
    ○ Most syscall logic handled by Sentry
    ○ No syscalls are "passed through". Applications cannot pass arbitrary
    arguments to the host kernel.
    gVisor (gvisor.dev)
    

    View Slide

23. 23
    gVisor
    Hardware Virtualization
    Application
    OS
    Virtualized Hardware
    

    View Slide

24. 24
    gVisor
    OS Virtualization
    Application
    Virtualized OS
    

    View Slide

25. 25
    gVisor
    gVisor (gvisor.dev)
    KVM/ptrace
    Gofer
    Gofer
    Gofers
    Containers
    Containers
    Host Linux Kernel
    Containers Sentry
    Sandbox
    User
    Kernel
    9P
    runsc
    OCI
    seccomp + ns
    seccomp + ns
    

    View Slide

26. 26
    gVisor
    ● Container-Native
    ○ Built from the ground up for containers
    ○ Host process - resource limits, secrets, volumes, etc.
    ● Portable
    ○ Can run inside of VMs
    ○ Doesn't need hardware support but can use it if available
    ● Resource Efficient
    ○ Able to use the host OS for resource management
    ○ Workloads can be tightly packed
    gVisor (gvisor.dev)
    

    View Slide

27. 27
    gVisor
    ● Cons
    ○ Compatibility (always getting better)
    ○ I/O Performance (also being improved)
    gVisor (gvisor.dev)
    

    View Slide

28. 28
    gVisor
    apiVersion: node.k8s.io/v1beta1
    kind: RuntimeClass
    metadata:
    name: myclass
    # The name of the corresponding CRI configuration
    handler: myhandler
    Runtimes & Kubernetes
    

    View Slide

29. 29
    gVisor
    Runtimes & Kubernetes
    myhandler
    shim
    OCI
    Kubernetes
    myhandler
    runtime
    CRI
    handler: myhandler
    

    View Slide

30. 30
    gVisor
    ● Be aware of defaults
    ○ K8s is optimized for ease-of-use, not security
    ○ CPU/Memory/Disk limits
    ● Network/Disk isolation
    ○ Network access: Use NetworkPolicy
    ○ Arbitrary packet injection
    ○ File writes/permissions: Use read-only filesystems
    ○ No throttling mechanism: use cgroups
    What's not protected?
    

    View Slide

31. 31
    gVisor
    gVisor
    https://gvisor.dev/
    Gitter: https://gitter.im/gvisor/community
    Mailing lists: gvisor-users, gvisor-dev
    Thanks!
    https://katacontainers.io/
    Nabla Containers
    https://nabla-containers.github.io/
    

    View Slide