Kubernetes Security Best Practices

01dc8e954957a10b428aa60b28c89d52?s=47 Ian Lewis
December 12, 2019

Kubernetes Security Best Practices

Containers give developers the ability to isolate applications from one another, but that’s not enough. Resource isolation is much different that security isolation. How do we make applications deployed in containers more secure? How do we apply existing tools like SELinux and AppArmor, and seccomp to our containers running in Kubernetes? How can we apply policy to our network and services to make sure applications only have access to what they need and nothing more?

Kubernetes provides the ability to secure containers, and secure access to the API. But it also has a flexible enough architecture to allow for applying network and service policy to various pods and services.

In this talk we will learn about the risks and attack surfaces and how to use tools like SELinux, AppArmor and seccomp to improve the security of containers deployed in Kubernetes. We’ll then go up the stack and learn how to apply network policy to containers to further improve security. Finally we will look at the Istio service mesh and how we can add authentication, mutual TLS, and access policies to whole services greatly reducing application attack surfaces.

01dc8e954957a10b428aa60b28c89d52?s=128

Ian Lewis

December 12, 2019
Tweet

Transcript

  1. 4.

    Topics • Security 101 • Runtime Security • Host Security

    • Network Security • Threat detection • Build Hygiene • Image Hygiene • SecOps
  2. 5.

    Topics • Security 101 • Runtime Security • Host Security

    • Network Security • Threat detection • Build Hygiene • Image Hygiene • SecOps ✔ ✔ ✔ ✔ ✗ ✗ ✗ ✗
  3. 6.

    Security 101 • Security is a spectrum • Attackers can

    pick their targets • Provide as many hurdles between the threat and asset • Attackers can shift focus. "It doesn't matter how many locks are on your door if your window is open"
  4. 7.

    Security 101 • Layered Security/Defence in Depth • Good security

    is redundant (not DRY) Network Node/Host Runtime
  5. 9.

    Security 101 • Least Privilege • Only give as much

    permission/privilege as is absolutely necessary
  6. 10.

    Guestbook app • Frontend ◦ Serves web traffic • Message

    ◦ Stores/lists messages • User ◦ Authentication Kubernetes Cluster Web Frontend Redis user message
  7. 11.

    Kubernetes API Server 1. Get token from frontend Pod 2.

    Use token to attack cluster API server 3. Get secrets etc. to further attack Kubernetes Cluster Web Frontend Redis user message ① ② ③
  8. 12.

    Mitigate 1 & 2: RBAC • Role Based Access Control

    • Roles given to users • Each role has permission to perform some operation ◦ get secrets ◦ update configmap ◦ etc. • RBAC settings apply to namespace
  9. 14.

    Mitigate 2: API Server Firewall • Restrict access to API

    server to certain IP addresses. • GKE: ◦ gcloud container clusters create --enable-master-authorized-networks --master-authorized-networks=....
  10. 15.

    Mitigate 3: Network Policy • Restrict access to database to

    only the Pods that require it • Specify access via labels • Implemented by Network solution: Calico, Weave, etc.
  11. 16.

    NetworkPolicy kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: name: redis spec: podSelector:

    matchLabels: name: redis ingress: - from: - podSelector: matchLabels: name: message
  12. 18.

    Mitigate 1: Secure etcd • Use authentication and firewalls to

    restrict access to etcd • Encrypt data in etcd (encryption at rest)
  13. 19.

    Get access to host 1. Break out of the container

    using container or kernel exploits 2. Attack the Kubelet 3. Attack other containers running on the same host Host Web Frontend
  14. 20.

    Mitigate 1: Run as non-root apiVersion: v1 kind: Pod metadata:

    name: security-context-demo spec: securityContext: runAsUser: 1000
  15. 21.

    Mitigate 1: Read only root filesystem apiVersion: v1 kind: Pod

    metadata: name: security-context-demo spec: securityContext: readOnlyRootFilesystem: true
  16. 22.
  17. 23.

    Mitigate 1: Do them all apiVersion: v1 kind: Pod metadata:

    name: security-context-demo spec: securityContext: runAsUser: 1000 readOnlyRootFilesystem: true allowPrivilegeEscalation: false
  18. 24.

    Mitigate 1: Sandboxed Pods 1. Pods are sandboxed from other

    Pods on the same host 2. Sandbox provides 2 layers of isolation: Sandbox + Container (Linux Kernel) 3. Examples: kata containers, gVisor Node Host Sandbox Pod Container Kernel Container Kubelet Runtime
  19. 25.

    gVisor 1. User space kernel 2. Intercepts and implements syscalls

    in userspace 3. Sandbox has low capabilities and runs with restricted seccomp filters Sandbox Process gVisor Sentry Application Gofer Kernel Ptrace/KVM Host Kernel 9P seccomp + namespaces
  20. 32.

    SELinux apiVersion: v1 kind: Pod metadata: name: mypod spec: securityContext:

    seLinuxOptions: level: "s0:c123,c456" containers: - name: hello ...
  21. 33.

    Mitigate 2 & 3: Restrict Kubelet permissions • RBAC for

    Kubelet: ◦ --authorization-mode=RBAC,Node --admission-control=...,NodeRestriction • Rotate Kubelet certs: ◦ kubelet … --rotate-certificates
  22. 34.

    Unsecured Pods • You follow the rules but others don't

    Kubernetes Cluster Web Frontend Redis user message
  23. 36.

    Listening to Traffic 1. Sniffing or intercepting traffic on the

    network 2. Request forgery Kubernetes Cluster Web Frontend Redis user message ① ②
  24. 38.

    istio 1. Proxy data between services 2. End-to-end encryption 3.

    Rolling certificates 4. Policy managed by central server Kubernetes Cluster Web Frontend Redis user message
  25. 39.

    istio 1. Proxy data between services 2. End-to-end encryption 3.

    Rolling certificates 4. Policy managed by central server Kubernetes Cluster Web Frontend Redis user message
  26. 40.

    Tips 1. Update Kubernetes early & often 2. Don't use

    admin for day-to-day work 3. Try benchmarking tools like kube-bench 4. Use managed services like GKE
  27. 41.