Slide 1

Slide 1 text

Kubernetes Security Best Practices ianmlewis@

Slide 2

Slide 2 text

Ian Lewis ● @IanMLewis ● ● Tokyo, Japan ● #kubernetes, #go, #python

Slide 3

Slide 3 text

Kubernetes ● Container Orchestrator ● An operations framework

Slide 4

Slide 4 text

Topics ● Security 101 ● Runtime Security ● Host Security ● Network Security ● Threat detection ● Build Hygiene ● Image Hygiene ● SecOps

Slide 5

Slide 5 text

Topics ● Security 101 ● Runtime Security ● Host Security ● Network Security ● Threat detection ● Build Hygiene ● Image Hygiene ● SecOps ✔ ✔ ✔ ✔ ✗ ✗ ✗ ✗

Slide 6

Slide 6 text

Security 101 ● Security is a spectrum ● Attackers can pick their targets ● Provide as many hurdles between the threat and asset ● Attackers can shift focus. "It doesn't matter how many locks are on your door if your window is open"

Slide 7

Slide 7 text

Security 101 ● Layered Security/Defence in Depth ● Good security is redundant (not DRY) Network Node/Host Runtime

Slide 8

Slide 8 text

Security 101 ● Limit the attack surface Network Node/Host Runtime

Slide 9

Slide 9 text

Security 101 ● Least Privilege ● Only give as much permission/privilege as is absolutely necessary

Slide 10

Slide 10 text

Guestbook app ● Frontend ○ Serves web traffic ● Message ○ Stores/lists messages ● User ○ Authentication Kubernetes Cluster Web Frontend Redis user message

Slide 11

Slide 11 text

Kubernetes API Server 1. Get token from frontend Pod 2. Use token to attack cluster API server 3. Get secrets etc. to further attack Kubernetes Cluster Web Frontend Redis user message ① ② ③

Slide 12

Slide 12 text

Mitigate 1 & 2: RBAC ● Role Based Access Control ● Roles given to users ● Each role has permission to perform some operation ○ get secrets ○ update configmap ○ etc. ● RBAC settings apply to namespace

Slide 13

Slide 13 text

Mitigate 1 & 2: RBAC ClusterRole ClusterRoleBinding 1:many many:1 Verb + Type

Slide 14

Slide 14 text

Mitigate 2: API Server Firewall ● Restrict access to API server to certain IP addresses. ● GKE: ○ gcloud container clusters create --enable-master-authorized-networks --master-authorized-networks=....

Slide 15

Slide 15 text

Mitigate 3: Network Policy ● Restrict access to database to only the Pods that require it ● Specify access via labels ● Implemented by Network solution: Calico, Weave, etc.

Slide 16

Slide 16 text

NetworkPolicy kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: name: redis spec: podSelector: matchLabels: name: redis ingress: - from: - podSelector: matchLabels: name: message

Slide 17

Slide 17 text

Get access to cluster components 1. Manipulate cluster data in etcd Host Web Frontend etcd

Slide 18

Slide 18 text

Mitigate 1: Secure etcd ● Use authentication and firewalls to restrict access to etcd ● Encrypt data in etcd (encryption at rest)

Slide 19

Slide 19 text

Get access to host 1. Break out of the container using container or kernel exploits 2. Attack the Kubelet 3. Attack other containers running on the same host Host Web Frontend

Slide 20

Slide 20 text

Mitigate 1: Run as non-root apiVersion: v1 kind: Pod metadata: name: security-context-demo spec: securityContext: runAsUser: 1000

Slide 21

Slide 21 text

Mitigate 1: Read only root filesystem apiVersion: v1 kind: Pod metadata: name: security-context-demo spec: securityContext: readOnlyRootFilesystem: true

Slide 22

Slide 22 text

Mitigate 1: no_new_privs apiVersion: v1 kind: Pod metadata: name: security-context-demo spec: securityContext: allowPrivilegeEscalation: false

Slide 23

Slide 23 text

Mitigate 1: Do them all apiVersion: v1 kind: Pod metadata: name: security-context-demo spec: securityContext: runAsUser: 1000 readOnlyRootFilesystem: true allowPrivilegeEscalation: false

Slide 24

Slide 24 text

Mitigate 1: Sandboxed Pods 1. Pods are sandboxed from other Pods on the same host 2. Sandbox provides 2 layers of isolation: Sandbox + Container (Linux Kernel) 3. Examples: kata containers, gVisor Node Host Sandbox Pod Container Kernel Container Kubelet Runtime

Slide 25

Slide 25 text

gVisor 1. User space kernel 2. Intercepts and implements syscalls in userspace 3. Sandbox has low capabilities and runs with restricted seccomp filters Sandbox Process gVisor Sentry Application Gofer Kernel Ptrace/KVM Host Kernel 9P seccomp + namespaces

Slide 26

Slide 26 text

Your App Mitigate 1: seccomp/ AppArmor/ SELinux

Slide 27

Slide 27 text

Container Mitigate 1: seccomp/ AppArmor/ SELinux

Slide 28

Slide 28 text

seccomp Mitigate 1: seccomp/ AppArmor/ SELinux

Slide 29

Slide 29 text

Mitigate 1: seccomp/ AppArmor/ SELinux AppArmor/ SELinux

Slide 30

Slide 30 text

seccomp apiVersion: v1 kind: Pod metadata: name: mypod annotations: seccomp.security.alpha.kubernetes.io/pod: runtime/default ...

Slide 31

Slide 31 text

AppArmor apiVersion: v1 kind: Pod metadata: name: mypod annotations: container.apparmor.security.beta.kubernetes.io/hello: runtime/default spec: containers: - name: hello ...

Slide 32

Slide 32 text

SELinux apiVersion: v1 kind: Pod metadata: name: mypod spec: securityContext: seLinuxOptions: level: "s0:c123,c456" containers: - name: hello ...

Slide 33

Slide 33 text

Mitigate 2 & 3: Restrict Kubelet permissions ● RBAC for Kubelet: ○ --authorization-mode=RBAC,Node --admission-control=...,NodeRestriction ● Rotate Kubelet certs: ○ kubelet … --rotate-certificates

Slide 34

Slide 34 text

Unsecured Pods ● You follow the rules but others don't Kubernetes Cluster Web Frontend Redis user message

Slide 35

Slide 35 text

Mitigate: Open Policy Agent

Slide 36

Slide 36 text

Listening to Traffic 1. Sniffing or intercepting traffic on the network 2. Request forgery Kubernetes Cluster Web Frontend Redis user message ① ②

Slide 37

Slide 37 text

istio ● Service mesh ● Includes Envoy proxy

Slide 38

Slide 38 text

istio 1. Proxy data between services 2. End-to-end encryption 3. Rolling certificates 4. Policy managed by central server Kubernetes Cluster Web Frontend Redis user message

Slide 39

Slide 39 text

istio 1. Proxy data between services 2. End-to-end encryption 3. Rolling certificates 4. Policy managed by central server Kubernetes Cluster Web Frontend Redis user message

Slide 40

Slide 40 text

Tips 1. Update Kubernetes early & often 2. Don't use admin for day-to-day work 3. Try benchmarking tools like kube-bench 4. Use managed services like GKE

Slide 41

Slide 41 text

Thanks!