Slide 1

Slide 1 text

Ian Lewis Developer Advocate, Google Cloud Platform The Enemy Within Running Untrusted Code in Kubernetes

Slide 2

Slide 2 text

2 gVisor Ian Lewis (@IanMLewis) Developer Advocate, Google

Slide 3

Slide 3 text

3 gVisor ● Running untrusted code ● Third-party/User uploaded code ○ Vendor-built apps ○ SaaS/Serverless ● Complex code/Complex user input ○ Video/Image transcoding ○ Machine learning ● Code you wrote but you don't trust yourself…. So you want to run some code...

Slide 4

Slide 4 text

4 gVisor Too much privileged code Application Host Kernel

Slide 5

Slide 5 text

5 gVisor Too much privileged code Application Host Kernel open("/path/to/file", O_RDWR)

Slide 6

Slide 6 text

6 gVisor Too much privileged code Application Host Kernel

Slide 7

Slide 7 text

7 gVisor Too much privileged code Application Host Kernel file descriptor

Slide 8

Slide 8 text

8 gVisor Too much privileged code Application Host Kernel Application

Slide 9

Slide 9 text

9 gVisor Too much privileged code Host Kernel open("/../\x00...", O_RDWR) Application Application

Slide 10

Slide 10 text

10 gVisor Too much privileged code Host Kernel Application Application

Slide 11

Slide 11 text

11 gVisor Too much privileged code Host Kernel Application Application

Slide 12

Slide 12 text

12 gVisor

Slide 13

Slide 13 text

13 gVisor ● Protects attackers from escaping the runtime environment ● Code running in the sandbox is untrusted ● Goal: Reduce the attack surface for untrusted code ● Method: Reduce the amount of trusted or privileged code (e.g. kernel code) that gets run. ○ Achieved through abstraction/virtualization of host. Container Sandboxes

Slide 14

Slide 14 text

14 gVisor Containers & Seccomp sandboxes Application Host Kernel Namespace

Slide 15

Slide 15 text

15 gVisor ● Pros ○ Same performance as normal processes ○ Uses built-in OS features. ● Cons ○ They aren't good security isolation boundaries ○ Any one bug in the host kernel could lead to a full host compromise ○ Seccomp sandboxes must be tweaked per application Containers & Seccomp sandboxes

Slide 16

Slide 16 text

16 gVisor Unikernels Application Host Kernel Guest OS Hypervisor

Slide 17

Slide 17 text

17 gVisor ● Pros ○ Reasonably fast ○ Low overhead/startup ○ Good compatibility ● Cons ○ Can't bring your own container (must be specially crafted) ○ Many limitations (fork, dynamic loading of libraries, etc.) Unikernels

Slide 18

Slide 18 text

18 gVisor Nabla Containers (nabla-containers.github.io)

Slide 19

Slide 19 text

19 gVisor Virtual Machines Application OS Virtualized Hardware

Slide 20

Slide 20 text

20 gVisor ● Pros ○ Hardware acceleration. Raw-performance is good ○ Maintenance is well understood ● Cons ○ Security implications often not well understood ○ Nested virtualization slow & source of security bugs ○ Resource management difficult ○ Often requires layers of indirection and proxies Virtual Machines

Slide 21

Slide 21 text

21 gVisor Kata Containers (katacontainers.io)

Slide 22

Slide 22 text

22 gVisor ● Hybrid approach = Virtualization w/ Host Resources ● Uses the same principle of virtualization as VMs ○ Virtualization at the OS; Linux Syscall layer ● Reduces the host attack surface ○ Calls to the host OS are controlled by the Sentry ○ Most syscall logic handled by Sentry ○ No syscalls are "passed through". Applications cannot pass arbitrary arguments to the host kernel. gVisor (gvisor.dev)

Slide 23

Slide 23 text

23 gVisor Hardware Virtualization Application OS Virtualized Hardware

Slide 24

Slide 24 text

24 gVisor OS Virtualization Application Virtualized OS

Slide 25

Slide 25 text

25 gVisor gVisor (gvisor.dev) KVM/ptrace Gofer Gofer Gofers Containers Containers Host Linux Kernel Containers Sentry Sandbox User Kernel 9P runsc OCI seccomp + ns seccomp + ns

Slide 26

Slide 26 text

26 gVisor ● Container-Native ○ Built from the ground up for containers ○ Host process - resource limits, secrets, volumes, etc. ● Portable ○ Can run inside of VMs ○ Doesn't need hardware support but can use it if available ● Resource Efficient ○ Able to use the host OS for resource management ○ Workloads can be tightly packed gVisor (gvisor.dev)

Slide 27

Slide 27 text

27 gVisor ● Cons ○ Compatibility (always getting better) ○ I/O Performance (also being improved) gVisor (gvisor.dev)

Slide 28

Slide 28 text

28 gVisor apiVersion: node.k8s.io/v1beta1 kind: RuntimeClass metadata: name: myclass # The name of the corresponding CRI configuration handler: myhandler Runtimes & Kubernetes

Slide 29

Slide 29 text

29 gVisor Runtimes & Kubernetes myhandler shim OCI Kubernetes myhandler runtime CRI handler: myhandler

Slide 30

Slide 30 text

30 gVisor ● Be aware of defaults ○ K8s is optimized for ease-of-use, not security ○ CPU/Memory/Disk limits ● Network/Disk isolation ○ Network access: Use NetworkPolicy ○ Arbitrary packet injection ○ File writes/permissions: Use read-only filesystems ○ No throttling mechanism: use cgroups What's not protected?

Slide 31

Slide 31 text

31 gVisor gVisor https://gvisor.dev/ Gitter: https://gitter.im/gvisor/community Mailing lists: gvisor-users, gvisor-dev Thanks! https://katacontainers.io/ Nabla Containers https://nabla-containers.github.io/