2018 Kubecon NA: How Symlinks Pwned Kubernetes

We go into technical details of the exploit and fix for CVE-2017-1002101

Transcript

1. How Symlinks Pwned K8s Michelle Au, Software Engineer, Google Jan Šafránek, Software Engineer, Red Hat
2. Agenda Discovery Development Disclosure Secure Practices Future
3. Vulnerability Reporting
4. Reporting Github issue created 2017-11-30 PodSecurityPolicy can be sidestepped with innocent emptyDir and subpath Here is a pod which would be allowed by fairly strict security policies, yet gives full control over node host by gaining access to docker socket: …
5. Reporting Github issue created 2017-11-30 PodSecurityPolicy can be sidestepped with innocent emptyDir and subpath Here is a pod which would be allowed by fairly strict security policies, yet gives full control over node host by gaining access to docker socket: …
6. Reporting That’s not how it’s done! Follow https://kubernetes.io/docs/reference/issues-security/security/ • Responsibly disclose to allow time to fix before public disclosure • security@kubernetes.io (optionally GPG encrypted) • Product Security Team handles the rest ◦ Evaluate impact ◦ Request CVE ◦ Coordinate development of fix, release, disclosure
7. Vulnerability Details
8. Volumes Background Node kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {}
9. Volumes Background Node kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {}
10. Volumes Background Node kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Pod volume’s root path (host): /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume my- volume
11. Volumes Background Node kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Final host’s path: <pod volume’s root path> + <subpath> my- volume
12. Volumes Background Node Final host’s path: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} my- volume
13. Volumes Background Node Final host’s path: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Docker my- volume
14. Volumes Background Node Final host’s path: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Docker Container my- volume’ Bind mount my- volume
15. Vulnerability? Node Final host’s path: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Docker Container my- volume’ Bind mount my- volume
16. Vulnerability Node Final host’s path: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Docker Container my- volume’ Created by user! Bind mount my- volume
17. Vulnerability Node my- volume Final host’s path: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Symlink
18. Vulnerability Node my- volume Final host’s path: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Symlink Root FS
19. Vulnerability Node Final host’s path: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Docker Container my- volume’ Symlink Root FS Bind mount my- volume
20. Demo
21. Solutions
22. Naive Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 3. Give to CRI Subpath volumePath: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume subPath: data1
23. Naive Solution before: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 volume subpath 1. Resolve all symlinks 2. Validate resolved path is inside volume 3. Give to CRI Subpath
24. Naive Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 3. Give to CRI Subpath before: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 after: /
25. Naive Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 3. Give to CRI Subpath before: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 after: / ❌
26. Naive Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 3. Give to CRI Subpath before: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 after: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1
27. Naive Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 3. Give to CRI Subpath before: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 after: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 ✅
28. Naive Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 3. Give to CRI Subpath before: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 after: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c
29. Naive Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 3. Give to CRI Subpath before: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 after: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c ✅
30. Naive Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 3. Give to CRI Subpath Race condition! User can still change anything to a symlink
31. Naive Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 3. Give to CRI Subpath data1: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c ✅
32. Naive Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 3. Give to CRI Subpath data1: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c ✅ $ rmdir /mnt/my-volume/a/b/c $ ln -s / /mnt/my-volume/a/b/c
33. Naive Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 3. Give to CRI Subpath data1: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c data1: / ✅ $ rmdir /mnt/my-volume/a/b/c $ ln -s / /mnt/my-volume/a/b/c
34. What Now? Need to “lock” directory between validation and CRI • Windows: lock using CreateFile • Linux: bind mount
35. Bind mount • Remount part of the file hierarchy somewhere else • Independent on the original hierarchy • Atomic $ mount --bind \ /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c \ /var/lib/kubelet/safe/place
36. Naive Bind-mount Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 4. Give bind mount to CRI Subpath 3. Bind mount to safe place
37. Naive Bind-mount Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 4. Give bind mount to CRI Subpath 3. Bind mount to safe place before: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 after: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c
38. Naive Bind-mount Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 4. Give bind mount to CRI Subpath 3. Bind mount to safe place before: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 after: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c ✅
39. Naive Bind-mount Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 4. Give bind mount to CRI Subpath 3. Bind mount to safe place $ mount --bind \ /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c \ /var/lib/kubelet/pods/<uid>/volume-subpaths/<container name>/<volume name>/0 safe place
40. Naive Bind-mount Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 4. Give bind mount to CRI Subpath 3. Bind mount to safe place $ docker -v /var/lib/kubelet/pods/<uid>/volume-subpaths/<container name>/<volume name>/0:/mnt/data
41. Naive Bind-mount Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 4. Give bind mount to CRI Subpath 3. Bind mount to safe place $ docker -v /var/lib/kubelet/pods/<uid>/volume-subpaths/<container name>/<volume name>/0:/mnt/data (was)$ -v /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1:/mnt/data
42. Naive Bind-mount Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 4. Give bind mount to CRI Subpath 3. Bind mount to safe place Safe
43. Naive Bind-mount Solution 1. Resolve all symlinks 2. Validate resolved path is inside volume 4. Give bind mount to CRI Subpath 3. Bind mount to safe place Race condition! User can still change anything to a symlink Race condition! User can still change anything to a symlink Safe
44. Final Solution 1. Resolve all symlinks 2. Safely open FD to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD
45. Final Solution 1. Resolve all symlinks 2. Safely open FD to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD before: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 after: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c
46. Final Solution 1. Resolve all symlinks 2. Safely open FD to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD Goal: safely open /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c
47. Final Solution 1. Resolve all symlinks 2. Safely open FD to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD Goal: safely open /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c open(“/var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/”) = 10
48. Final Solution 1. Resolve all symlinks 2. Safely open FD to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD Goal: safely open /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c open(“/var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/”) = 10 openat(10, “a”, O_NOFOLLOW) = 11
49. Final Solution 1. Resolve all symlinks 2. Safely open FD to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD Goal: safely open /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c open(“/var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/”) = 10 openat(10, “a”, O_NOFOLLOW) = 11 openat(11, “b”, O_NOFOLLOW) = 12
50. Final Solution 1. Resolve all symlinks 2. Safely open FD to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD Goal: safely open /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c open(“/var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/”) = 10 openat(10, “a”, O_NOFOLLOW) = 11 openat(11, “b”, O_NOFOLLOW) = 12 openat(12, “c”, O_NOFOLLOW) = 13
51. Final Solution 1. Resolve all symlinks 2. Safely open FD to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD $ ls -la /proc/<pidof kubelet>/fd/13 13 -> /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c
52. $ ls -la /proc/<pidof kubelet>/fd/13 13 -> /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c $ mount --bind /proc/<pidof kubelet>/fd/13 \ /var/lib/kubelet/pods/<uid>/volume-subpaths/<container name>/<volume name>/0 Final Solution 1. Resolve all symlinks 2. Safely open FD to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD
53. Final Solution 1. Resolve all symlinks 2. Safely open FD to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD
54. Final Solution 1. Resolve all symlinks 2. Safely open FD to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD $ docker -v /var/lib/kubelet/pods/<uid>/volume-subpaths/<container name>/<volume name>/0:/mnt/data
55. Final Solution 1. Resolve all symlinks 2. Safely open FD to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD Safe (bind mount) Safe (no symlinks, inside container)
56. Release Process
57. Development github.com/kubernetes-security/kubernetes • Private, very limited access ◦ Access revoked once fix released • Similar process as kubernetes/kubernetes ◦ Same test jobs ◦ Logs in private buckets ◦ Be careful with git pushes • Coordinated by Product Security Team
58. Pre-Disclosure • Coordinated by Product Security Team • Kubernetes release and branch managers • 3rd party Kubernetes vendors (“Private Distributors List”) ◦ Patch and test under embargo
59. Public Disclosure 2018-03-12: CVE-2017-1002101 announced Kubernetes 1.7, 1.8, 1.9 quickly patched and released Post-mortem document
60. Securing Volumes
61. Secure Practices Now Don’t run containers as root user • Use PodSecurityPolicy to require pods run as non-root • Caveat: containers still run as root gid ◦ K8s 1.10: RunAsGroup alpha feature Use PodSecurityPolicy to restrict volume access • Whitelist allowed volume types • Restrict both allowed HostPath prefixes and readOnly (K8s 1.11)
62. Future Improvements
63. Future Core Principle Multiple security boundaries around untrusted code
64. Vulnerability Node Final host’s path: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Docker Container my- volume’ Symlink Root FS Bind mount my- volume
65. Node Sandbox Future: Sandboxed Subpath? my- volume Final host’s path: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Symlink Root FS
66. Node Sandbox Future: Sandboxed Subpath? Final host’s path: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Symlink Root FS Subpath Handling my- volume
67. Node Sandbox Future: Sandboxed Subpath? Final host’s path: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Symlink Root FS Container my- volume’ Subpath Handling my- volume
68. Summary Follow the Kubernetes security disclosure process Be extra cautious when handling untrusted paths • symlink race • time of check to time of use Set restrictive policies and use multiple security boundaries
69. Acknowledgments Thanks to Maxim Ivanov for reporting the vulnerability!
70. Get Involved! Kubernetes product security team Secure Container Isolation • Overall: sig-node • Volumes: sig-storage • “Recent Advancements in Container Isolation” - today 1:45pm