2018 Kubecon NA: How Symlinks Pwned Kubernetes

Slide 1

Slide 1 text

How Symlinks Pwned K8s Michelle Au, Software Engineer, Google Jan Šafránek, Software Engineer, Red Hat

Slide 2

Slide 2 text

Agenda Discovery Development Disclosure Secure Practices Future

Slide 3

Slide 3 text

Vulnerability Reporting

Slide 4

Slide 4 text

Reporting Github issue created 2017-11-30 PodSecurityPolicy can be sidestepped with innocent emptyDir and subpath Here is a pod which would be allowed by fairly strict security policies, yet gives full control over node host by gaining access to docker socket: …

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Reporting That’s not how it’s done! Follow https://kubernetes.io/docs/reference/issues-security/security/ ● Responsibly disclose to allow time to fix before public disclosure ● [email protected] (optionally GPG encrypted) ● Product Security Team handles the rest ○ Evaluate impact ○ Request CVE ○ Coordinate development of fix, release, disclosure

Slide 7

Slide 7 text

Vulnerability Details

Slide 8

Slide 8 text

Volumes Background Node kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {}

Slide 9

Slide 9 text

Volumes Background Node kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {}

Slide 10

Slide 10 text

Volumes Background Node kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Pod volume’s root path (host): /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume my- volume

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Volumes Background Node Final host’s path: /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} my- volume

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Vulnerability? Node Final host’s path: /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Docker Container my- volume’ Bind mount my- volume

Slide 16

Slide 16 text

Vulnerability Node Final host’s path: /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Docker Container my- volume’ Created by user! Bind mount my- volume

Slide 17

Slide 17 text

Vulnerability Node my- volume Final host’s path: /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Symlink

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Demo

Slide 21

Slide 21 text

Solutions

Slide 22

Slide 22 text

Naive Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 3. Give to CRI Subpath volumePath: /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume subPath: data1

Slide 23

Slide 23 text

Naive Solution before: /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume/data1 volume subpath 1. Resolve all symlinks 2. Validate resolved path is inside volume 3. Give to CRI Subpath

Slide 24

Slide 24 text

Naive Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 3. Give to CRI Subpath before: /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume/data1 after: /

Slide 25

Slide 25 text

Naive Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 3. Give to CRI Subpath before: /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume/data1 after: / ❌

Slide 26

Slide 26 text

Naive Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 3. Give to CRI Subpath before: /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume/data1 after: /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume/data1

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Naive Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 3. Give to CRI Subpath Race condition! User can still change anything to a symlink

Slide 31

Slide 31 text

Naive Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 3. Give to CRI Subpath data1: /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume/a/b/c ✅

Slide 32

Slide 32 text

Naive Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 3. Give to CRI Subpath data1: /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume/a/b/c ✅ $ rmdir /mnt/my-volume/a/b/c $ ln -s / /mnt/my-volume/a/b/c

Slide 33

Slide 33 text

Naive Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 3. Give to CRI Subpath data1: /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume/a/b/c data1: / ✅ $ rmdir /mnt/my-volume/a/b/c $ ln -s / /mnt/my-volume/a/b/c

Slide 34

Slide 34 text

What Now? Need to “lock” directory between validation and CRI ● Windows: lock using CreateFile ● Linux: bind mount

Slide 35

Slide 35 text

Bind mount ● Remount part of the file hierarchy somewhere else ● Independent on the original hierarchy ● Atomic $ mount --bind \ /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume/a/b/c \ /var/lib/kubelet/safe/place

Slide 36

Slide 36 text

Naive Bind-mount Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 4. Give bind mount to CRI Subpath 3. Bind mount to safe place

Slide 37

Slide 37 text

Naive Bind-mount Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 4. Give bind mount to CRI Subpath 3. Bind mount to safe place before: /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume/data1 after: /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume/a/b/c

Slide 38

Slide 38 text

Slide 39

Slide 39 text

Naive Bind-mount Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 4. Give bind mount to CRI Subpath 3. Bind mount to safe place $ mount --bind \ /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume/a/b/c \ /var/lib/kubelet/pods//volume-subpaths///0 safe place

Slide 40

Slide 40 text

Naive Bind-mount Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 4. Give bind mount to CRI Subpath 3. Bind mount to safe place $ docker -v /var/lib/kubelet/pods//volume-subpaths///0:/mnt/data

Slide 41

Slide 41 text

Slide 42

Slide 42 text

Naive Bind-mount Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 4. Give bind mount to CRI Subpath 3. Bind mount to safe place Safe

Slide 43

Slide 43 text

Naive Bind-mount Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 4. Give bind mount to CRI Subpath 3. Bind mount to safe place Race condition! User can still change anything to a symlink Race condition! User can still change anything to a symlink Safe

Slide 44

Slide 44 text

Final Solution 1. Resolve all symlinks 2. Safely open FD to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD

Slide 45

Slide 45 text

Final Solution 1. Resolve all symlinks 2. Safely open FD to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD before: /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume/data1 after: /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume/a/b/c

Slide 46

Slide 46 text

Slide 47

Slide 47 text

Final Solution 1. Resolve all symlinks 2. Safely open FD to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD Goal: safely open /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume/a/b/c open(“/var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume/”) = 10

Slide 48

Slide 48 text

Slide 49

Slide 49 text

Slide 50

Slide 50 text

Slide 51

Slide 51 text

Final Solution 1. Resolve all symlinks 2. Safely open FD to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD $ ls -la /proc//fd/13 13 -> /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume/a/b/c

Slide 52

Slide 52 text

$ ls -la /proc//fd/13 13 -> /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume/a/b/c $ mount --bind /proc//fd/13 \ /var/lib/kubelet/pods//volume-subpaths///0 Final Solution 1. Resolve all symlinks 2. Safely open FD to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD

Slide 53

Slide 53 text

Final Solution 1. Resolve all symlinks 2. Safely open FD to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD

Slide 54

Slide 54 text

Final Solution 1. Resolve all symlinks 2. Safely open FD to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD $ docker -v /var/lib/kubelet/pods//volume-subpaths///0:/mnt/data

Slide 55

Slide 55 text

Final Solution 1. Resolve all symlinks 2. Safely open FD to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD Safe (bind mount) Safe (no symlinks, inside container)

Slide 56

Slide 56 text

Release Process

Slide 57

Slide 57 text

Development github.com/kubernetes-security/kubernetes ● Private, very limited access ○ Access revoked once fix released ● Similar process as kubernetes/kubernetes ○ Same test jobs ○ Logs in private buckets ○ Be careful with git pushes ● Coordinated by Product Security Team

Slide 58

Slide 58 text

Pre-Disclosure ● Coordinated by Product Security Team ● Kubernetes release and branch managers ● 3rd party Kubernetes vendors (“Private Distributors List”) ○ Patch and test under embargo

Slide 59

Slide 59 text

Public Disclosure 2018-03-12: CVE-2017-1002101 announced Kubernetes 1.7, 1.8, 1.9 quickly patched and released Post-mortem document

Slide 60

Slide 60 text

Securing Volumes

Slide 61

Slide 61 text

Secure Practices Now Don’t run containers as root user ● Use PodSecurityPolicy to require pods run as non-root ● Caveat: containers still run as root gid ○ K8s 1.10: RunAsGroup alpha feature Use PodSecurityPolicy to restrict volume access ● Whitelist allowed volume types ● Restrict both allowed HostPath prefixes and readOnly (K8s 1.11)

Slide 62

Slide 62 text

Future Improvements

Slide 63

Slide 63 text

Future Core Principle Multiple security boundaries around untrusted code

Slide 64

Slide 64 text

Slide 65

Slide 65 text

Node Sandbox Future: Sandboxed Subpath? my- volume Final host’s path: /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Symlink Root FS

Slide 66

Slide 66 text

Node Sandbox Future: Sandboxed Subpath? Final host’s path: /var/lib/kubelet/pods//volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Symlink Root FS Subpath Handling my- volume

Slide 67

Slide 67 text

Slide 68

Slide 68 text

Summary Follow the Kubernetes security disclosure process Be extra cautious when handling untrusted paths ● symlink race ● time of check to time of use Set restrictive policies and use multiple security boundaries

Slide 69

Slide 69 text

Acknowledgments Thanks to Maxim Ivanov for reporting the vulnerability!

Slide 70

Slide 70 text

Get Involved! Kubernetes product security team Secure Container Isolation ● Overall: sig-node ● Volumes: sig-storage ● “Recent Advancements in Container Isolation” - today 1:45pm

Slide 71

Slide 71 text

No content