Kubernetes Security Best Practices

01dc8e954957a10b428aa60b28c89d52?s=47 Ian Lewis
December 12, 2019

Kubernetes Security Best Practices

Containers give developers the ability to isolate applications from one another, but that’s not enough. Resource isolation is much different that security isolation. How do we make applications deployed in containers more secure? How do we apply existing tools like SELinux and AppArmor, and seccomp to our containers running in Kubernetes? How can we apply policy to our network and services to make sure applications only have access to what they need and nothing more?

Kubernetes provides the ability to secure containers, and secure access to the API. But it also has a flexible enough architecture to allow for applying network and service policy to various pods and services.

In this talk we will learn about the risks and attack surfaces and how to use tools like SELinux, AppArmor and seccomp to improve the security of containers deployed in Kubernetes. We’ll then go up the stack and learn how to apply network policy to containers to further improve security. Finally we will look at the Istio service mesh and how we can add authentication, mutual TLS, and access policies to whole services greatly reducing application attack surfaces.

01dc8e954957a10b428aa60b28c89d52?s=128

Ian Lewis

December 12, 2019
Tweet

Transcript

  1. Kubernetes Security Best Practices ianmlewis@

  2. Ian Lewis • @IanMLewis • • Tokyo, Japan • #kubernetes,

    #go, #python
  3. Kubernetes • Container Orchestrator • An operations framework

  4. Topics • Security 101 • Runtime Security • Host Security

    • Network Security • Threat detection • Build Hygiene • Image Hygiene • SecOps
  5. Topics • Security 101 • Runtime Security • Host Security

    • Network Security • Threat detection • Build Hygiene • Image Hygiene • SecOps ✔ ✔ ✔ ✔ ✗ ✗ ✗ ✗
  6. Security 101 • Security is a spectrum • Attackers can

    pick their targets • Provide as many hurdles between the threat and asset • Attackers can shift focus. "It doesn't matter how many locks are on your door if your window is open"
  7. Security 101 • Layered Security/Defence in Depth • Good security

    is redundant (not DRY) Network Node/Host Runtime
  8. Security 101 • Limit the attack surface Network Node/Host Runtime

  9. Security 101 • Least Privilege • Only give as much

    permission/privilege as is absolutely necessary
  10. Guestbook app • Frontend ◦ Serves web traffic • Message

    ◦ Stores/lists messages • User ◦ Authentication Kubernetes Cluster Web Frontend Redis user message
  11. Kubernetes API Server 1. Get token from frontend Pod 2.

    Use token to attack cluster API server 3. Get secrets etc. to further attack Kubernetes Cluster Web Frontend Redis user message ① ② ③
  12. Mitigate 1 & 2: RBAC • Role Based Access Control

    • Roles given to users • Each role has permission to perform some operation ◦ get secrets ◦ update configmap ◦ etc. • RBAC settings apply to namespace
  13. Mitigate 1 & 2: RBAC ClusterRole ClusterRoleBinding 1:many many:1 Verb

    + Type
  14. Mitigate 2: API Server Firewall • Restrict access to API

    server to certain IP addresses. • GKE: ◦ gcloud container clusters create --enable-master-authorized-networks --master-authorized-networks=....
  15. Mitigate 3: Network Policy • Restrict access to database to

    only the Pods that require it • Specify access via labels • Implemented by Network solution: Calico, Weave, etc.
  16. NetworkPolicy kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: name: redis spec: podSelector:

    matchLabels: name: redis ingress: - from: - podSelector: matchLabels: name: message
  17. Get access to cluster components 1. Manipulate cluster data in

    etcd Host Web Frontend etcd
  18. Mitigate 1: Secure etcd • Use authentication and firewalls to

    restrict access to etcd • Encrypt data in etcd (encryption at rest)
  19. Get access to host 1. Break out of the container

    using container or kernel exploits 2. Attack the Kubelet 3. Attack other containers running on the same host Host Web Frontend
  20. Mitigate 1: Run as non-root apiVersion: v1 kind: Pod metadata:

    name: security-context-demo spec: securityContext: runAsUser: 1000
  21. Mitigate 1: Read only root filesystem apiVersion: v1 kind: Pod

    metadata: name: security-context-demo spec: securityContext: readOnlyRootFilesystem: true
  22. Mitigate 1: no_new_privs apiVersion: v1 kind: Pod metadata: name: security-context-demo

    spec: securityContext: allowPrivilegeEscalation: false
  23. Mitigate 1: Do them all apiVersion: v1 kind: Pod metadata:

    name: security-context-demo spec: securityContext: runAsUser: 1000 readOnlyRootFilesystem: true allowPrivilegeEscalation: false
  24. Mitigate 1: Sandboxed Pods 1. Pods are sandboxed from other

    Pods on the same host 2. Sandbox provides 2 layers of isolation: Sandbox + Container (Linux Kernel) 3. Examples: kata containers, gVisor Node Host Sandbox Pod Container Kernel Container Kubelet Runtime
  25. gVisor 1. User space kernel 2. Intercepts and implements syscalls

    in userspace 3. Sandbox has low capabilities and runs with restricted seccomp filters Sandbox Process gVisor Sentry Application Gofer Kernel Ptrace/KVM Host Kernel 9P seccomp + namespaces
  26. Your App Mitigate 1: seccomp/ AppArmor/ SELinux

  27. Container Mitigate 1: seccomp/ AppArmor/ SELinux

  28. seccomp Mitigate 1: seccomp/ AppArmor/ SELinux

  29. Mitigate 1: seccomp/ AppArmor/ SELinux AppArmor/ SELinux

  30. seccomp apiVersion: v1 kind: Pod metadata: name: mypod annotations: seccomp.security.alpha.kubernetes.io/pod:

    runtime/default ...
  31. AppArmor apiVersion: v1 kind: Pod metadata: name: mypod annotations: container.apparmor.security.beta.kubernetes.io/hello:

    runtime/default spec: containers: - name: hello ...
  32. SELinux apiVersion: v1 kind: Pod metadata: name: mypod spec: securityContext:

    seLinuxOptions: level: "s0:c123,c456" containers: - name: hello ...
  33. Mitigate 2 & 3: Restrict Kubelet permissions • RBAC for

    Kubelet: ◦ --authorization-mode=RBAC,Node --admission-control=...,NodeRestriction • Rotate Kubelet certs: ◦ kubelet … --rotate-certificates
  34. Unsecured Pods • You follow the rules but others don't

    Kubernetes Cluster Web Frontend Redis user message
  35. Mitigate: Open Policy Agent

  36. Listening to Traffic 1. Sniffing or intercepting traffic on the

    network 2. Request forgery Kubernetes Cluster Web Frontend Redis user message ① ②
  37. istio • Service mesh • Includes Envoy proxy

  38. istio 1. Proxy data between services 2. End-to-end encryption 3.

    Rolling certificates 4. Policy managed by central server Kubernetes Cluster Web Frontend Redis user message
  39. istio 1. Proxy data between services 2. End-to-end encryption 3.

    Rolling certificates 4. Policy managed by central server Kubernetes Cluster Web Frontend Redis user message
  40. Tips 1. Update Kubernetes early & often 2. Don't use

    admin for day-to-day work 3. Try benchmarking tools like kube-bench 4. Use managed services like GKE
  41. Thanks!