Kubernetes Security Best Practices

Containers give developers the ability to isolate applications from one another, but that’s not enough. Resource isolation is much different that security isolation. How do we make applications deployed in containers more secure? How do we apply existing tools like SELinux and AppArmor, and seccomp to our containers running in Kubernetes? How can we apply policy to our network and services to make sure applications only have access to what they need and nothing more?

Kubernetes provides the ability to secure containers, and secure access to the API. But it also has a flexible enough architecture to allow for applying network and service policy to various pods and services.

In this talk we will learn about the risks and attack surfaces and how to use tools like SELinux, AppArmor and seccomp to improve the security of containers deployed in Kubernetes. We’ll then go up the stack and learn how to apply network policy to containers to further improve security. Finally we will look at the Istio service mesh and how we can add authentication, mutual TLS, and access policies to whole services greatly reducing application attack surfaces.

Transcript

1. Kubernetes Security Best Practices ianmlewis@
2. Ian Lewis • @IanMLewis • • Tokyo, Japan • #kubernetes, #go, #python
3. Kubernetes • Container Orchestrator • An operations framework
4. Topics • Security 101 • Runtime Security • Host Security • Network Security • Threat detection • Build Hygiene • Image Hygiene • SecOps
5. Topics • Security 101 • Runtime Security • Host Security • Network Security • Threat detection • Build Hygiene • Image Hygiene • SecOps ✔ ✔ ✔ ✔ ✗ ✗ ✗ ✗
6. Security 101 • Security is a spectrum • Attackers can pick their targets • Provide as many hurdles between the threat and asset • Attackers can shift focus. "It doesn't matter how many locks are on your door if your window is open"
7. Security 101 • Layered Security/Defence in Depth • Good security is redundant (not DRY) Network Node/Host Runtime
8. Security 101 • Limit the attack surface Network Node/Host Runtime
9. Security 101 • Least Privilege • Only give as much permission/privilege as is absolutely necessary
10. Guestbook app • Frontend ◦ Serves web traffic • Message ◦ Stores/lists messages • User ◦ Authentication Kubernetes Cluster Web Frontend Redis user message
11. Kubernetes API Server 1. Get token from frontend Pod 2. Use token to attack cluster API server 3. Get secrets etc. to further attack Kubernetes Cluster Web Frontend Redis user message ① ② ③
12. Mitigate 1 & 2: RBAC • Role Based Access Control • Roles given to users • Each role has permission to perform some operation ◦ get secrets ◦ update configmap ◦ etc. • RBAC settings apply to namespace
13. Mitigate 1 & 2: RBAC ClusterRole ClusterRoleBinding 1:many many:1 Verb + Type
14. Mitigate 2: API Server Firewall • Restrict access to API server to certain IP addresses. • GKE: ◦ gcloud container clusters create --enable-master-authorized-networks --master-authorized-networks=....
15. Mitigate 3: Network Policy • Restrict access to database to only the Pods that require it • Specify access via labels • Implemented by Network solution: Calico, Weave, etc.
16. NetworkPolicy kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: name: redis spec: podSelector: matchLabels: name: redis ingress: - from: - podSelector: matchLabels: name: message
17. Get access to cluster components 1. Manipulate cluster data in etcd Host Web Frontend etcd
18. Mitigate 1: Secure etcd • Use authentication and firewalls to restrict access to etcd • Encrypt data in etcd (encryption at rest)
19. Get access to host 1. Break out of the container using container or kernel exploits 2. Attack the Kubelet 3. Attack other containers running on the same host Host Web Frontend
20. Mitigate 1: Run as non-root apiVersion: v1 kind: Pod metadata: name: security-context-demo spec: securityContext: runAsUser: 1000
21. Mitigate 1: Read only root filesystem apiVersion: v1 kind: Pod metadata: name: security-context-demo spec: securityContext: readOnlyRootFilesystem: true
22. Mitigate 1: no_new_privs apiVersion: v1 kind: Pod metadata: name: security-context-demo spec: securityContext: allowPrivilegeEscalation: false
23. Mitigate 1: Do them all apiVersion: v1 kind: Pod metadata: name: security-context-demo spec: securityContext: runAsUser: 1000 readOnlyRootFilesystem: true allowPrivilegeEscalation: false
24. Mitigate 1: Sandboxed Pods 1. Pods are sandboxed from other Pods on the same host 2. Sandbox provides 2 layers of isolation: Sandbox + Container (Linux Kernel) 3. Examples: kata containers, gVisor Node Host Sandbox Pod Container Kernel Container Kubelet Runtime
25. gVisor 1. User space kernel 2. Intercepts and implements syscalls in userspace 3. Sandbox has low capabilities and runs with restricted seccomp filters Sandbox Process gVisor Sentry Application Gofer Kernel Ptrace/KVM Host Kernel 9P seccomp + namespaces
26. Your App Mitigate 1: seccomp/ AppArmor/ SELinux
27. Container Mitigate 1: seccomp/ AppArmor/ SELinux
28. seccomp Mitigate 1: seccomp/ AppArmor/ SELinux
29. Mitigate 1: seccomp/ AppArmor/ SELinux AppArmor/ SELinux
30. seccomp apiVersion: v1 kind: Pod metadata: name: mypod annotations: seccomp.security.alpha.kubernetes.io/pod: runtime/default ...
31. AppArmor apiVersion: v1 kind: Pod metadata: name: mypod annotations: container.apparmor.security.beta.kubernetes.io/hello: runtime/default spec: containers: - name: hello ...
32. SELinux apiVersion: v1 kind: Pod metadata: name: mypod spec: securityContext: seLinuxOptions: level: "s0:c123,c456" containers: - name: hello ...
33. Mitigate 2 & 3: Restrict Kubelet permissions • RBAC for Kubelet: ◦ --authorization-mode=RBAC,Node --admission-control=...,NodeRestriction • Rotate Kubelet certs: ◦ kubelet … --rotate-certificates
34. Unsecured Pods • You follow the rules but others don't Kubernetes Cluster Web Frontend Redis user message
35. Mitigate: Open Policy Agent
36. Listening to Traffic 1. Sniffing or intercepting traffic on the network 2. Request forgery Kubernetes Cluster Web Frontend Redis user message ① ②
37. istio • Service mesh • Includes Envoy proxy
38. istio 1. Proxy data between services 2. End-to-end encryption 3. Rolling certificates 4. Policy managed by central server Kubernetes Cluster Web Frontend Redis user message
39. istio 1. Proxy data between services 2. End-to-end encryption 3. Rolling certificates 4. Policy managed by central server Kubernetes Cluster Web Frontend Redis user message
40. Tips 1. Update Kubernetes early & often 2. Don't use admin for day-to-day work 3. Try benchmarking tools like kube-bench 4. Use managed services like GKE
41. Thanks!