Kubernetes Security Best Practices

Containers give developers the ability to isolate applications from one another, but that’s not enough. Resource isolation is much different that security isolation. How do we make applications deployed in containers more secure? How do we apply existing tools like SELinux and AppArmor, and seccomp to our containers running in Kubernetes? How can we apply policy to our network and services to make sure applications only have access to what they need and nothing more?

Kubernetes provides the ability to secure containers, and secure access to the API. But it also has a flexible enough architecture to allow for applying network and service policy to various pods and services.

In this talk we will learn about the risks and attack surfaces and how to use tools like SELinux, AppArmor and seccomp to improve the security of containers deployed in Kubernetes. We’ll then go up the stack and learn how to apply network policy to containers to further improve security. Finally we will look at the Istio service mesh and how we can add authentication, mutual TLS, and access policies to whole services greatly reducing application attack surfaces.

Transcript

1. Kubernetes Security
   Best Practices
   ianmlewis@
   

   View Slide

2. Ian Lewis
   ●
   @IanMLewis
   ●
   ●
   Tokyo, Japan
   ●
   #kubernetes, #go,
   #python
   

   View Slide

3. Kubernetes
   ●
   Container
   Orchestrator
   ●
   An operations
   framework
   

   View Slide

4. Topics
   ●
   Security 101
   ●
   Runtime Security
   ●
   Host Security
   ●
   Network Security
   ●
   Threat detection
   ●
   Build Hygiene
   ●
   Image Hygiene
   ●
   SecOps
   

   View Slide

5. Topics
   ●
   Security 101
   ●
   Runtime Security
   ●
   Host Security
   ●
   Network Security
   ●
   Threat detection
   ●
   Build Hygiene
   ●
   Image Hygiene
   ●
   SecOps
   ✔
   ✔
   ✔
   ✔
   ✗
   ✗
   ✗
   ✗
   

   View Slide

6. Security 101
   ●
   Security is a spectrum
   ●
   Attackers can pick their
   targets
   ●
   Provide as many hurdles
   between the threat and
   asset
   ●
   Attackers can shift focus.
   "It doesn't matter how
   many locks are on your
   door if your window is
   open"
   

   View Slide

7. Security 101
   ●
   Layered Security/Defence
   in Depth
   ●
   Good security is redundant
   (not DRY)
   Network
   Node/Host
   Runtime
   

   View Slide

8. Security 101
   ●
   Limit the attack surface
   Network
   Node/Host
   Runtime
   

   View Slide

9. Security 101
   ●
   Least Privilege
   ●
   Only give as much
   permission/privilege as is
   absolutely necessary
   

   View Slide

10. Guestbook app
    ●
    Frontend
    ○
    Serves web traffic
    ●
    Message
    ○
    Stores/lists
    messages
    ●
    User
    ○
    Authentication
    Kubernetes Cluster
    Web
    Frontend
    Redis
    user
    message
    

    View Slide

11. Kubernetes API Server
    1. Get token from
    frontend Pod
    2. Use token to attack
    cluster API server
    3. Get secrets etc. to
    further attack
    Kubernetes Cluster
    Web
    Frontend
    Redis
    user
    message
    ①
    ②
    ③
    

    View Slide

12. Mitigate 1 & 2: RBAC
    ●
    Role Based Access Control
    ●
    Roles given to users
    ●
    Each role has permission to perform some operation
    ○
    get secrets
    ○
    update configmap
    ○
    etc.
    ●
    RBAC settings apply to namespace
    

    View Slide

13. Mitigate 1 & 2: RBAC
    ClusterRole
    ClusterRoleBinding
    1:many many:1
    Verb +
    Type
    

    View Slide

14. Mitigate 2: API Server Firewall
    ●
    Restrict access to API server to certain IP addresses.
    ●
    GKE:
    ○ gcloud container clusters create
    --enable-master-authorized-networks
    --master-authorized-networks=....
    

    View Slide

15. Mitigate 3: Network Policy
    ●
    Restrict access to database to only the Pods that require it
    ●
    Specify access via labels
    ●
    Implemented by Network solution: Calico, Weave, etc.
    

    View Slide

16. NetworkPolicy
    kind: NetworkPolicy
    apiVersion: networking.k8s.io/v1
    metadata:
    name: redis
    spec:
    podSelector:
    matchLabels:
    name: redis
    ingress:
    - from:
    - podSelector:
    matchLabels:
    name: message
    

    View Slide

17. Get access to cluster
    components
    1. Manipulate cluster
    data in etcd
    Host
    Web
    Frontend
    etcd
    

    View Slide

18. Mitigate 1: Secure etcd
    ●
    Use authentication and firewalls to restrict access to etcd
    ●
    Encrypt data in etcd (encryption at rest)
    

    View Slide

19. Get access to host
    1. Break out of the
    container using
    container or kernel
    exploits
    2. Attack the Kubelet
    3. Attack other
    containers running on
    the same host
    Host
    Web
    Frontend
    

    View Slide

20. Mitigate 1: Run as non-root
    apiVersion: v1
    kind: Pod
    metadata:
    name: security-context-demo
    spec:
    securityContext:
    runAsUser: 1000
    

    View Slide

21. Mitigate 1: Read only root filesystem
    apiVersion: v1
    kind: Pod
    metadata:
    name: security-context-demo
    spec:
    securityContext:
    readOnlyRootFilesystem: true
    

    View Slide

22. Mitigate 1: no_new_privs
    apiVersion: v1
    kind: Pod
    metadata:
    name: security-context-demo
    spec:
    securityContext:
    allowPrivilegeEscalation: false
    

    View Slide

23. Mitigate 1: Do them all
    apiVersion: v1
    kind: Pod
    metadata:
    name: security-context-demo
    spec:
    securityContext:
    runAsUser: 1000
    readOnlyRootFilesystem: true
    allowPrivilegeEscalation: false
    

    View Slide

24. Mitigate 1: Sandboxed Pods
    1. Pods are sandboxed
    from other Pods on
    the same host
    2. Sandbox provides 2
    layers of isolation:
    Sandbox + Container
    (Linux Kernel)
    3. Examples: kata
    containers, gVisor
    Node Host
    Sandbox Pod
    Container
    Kernel
    Container
    Kubelet
    Runtime
    

    View Slide

25. gVisor
    1. User space kernel
    2. Intercepts and
    implements syscalls in
    userspace
    3. Sandbox has low
    capabilities and runs
    with restricted
    seccomp filters
    Sandbox Process
    gVisor
    Sentry
    Application
    Gofer
    Kernel
    Ptrace/KVM
    Host Kernel
    9P
    seccomp +
    namespaces
    

    View Slide

26. Your App
    Mitigate 1:
    seccomp/
    AppArmor/
    SELinux
    

    View Slide

27. Container
    Mitigate 1:
    seccomp/
    AppArmor/
    SELinux
    

    View Slide

28. seccomp
    Mitigate 1:
    seccomp/
    AppArmor/
    SELinux
    

    View Slide

29. Mitigate 1:
    seccomp/
    AppArmor/
    SELinux
    AppArmor/
    SELinux
    

    View Slide

30. seccomp
    apiVersion: v1
    kind: Pod
    metadata:
    name: mypod
    annotations:
    seccomp.security.alpha.kubernetes.io/pod: runtime/default
    ...
    

    View Slide

31. AppArmor
    apiVersion: v1
    kind: Pod
    metadata:
    name: mypod
    annotations:
    container.apparmor.security.beta.kubernetes.io/hello: runtime/default
    spec:
    containers:
    - name: hello
    ...
    

    View Slide

32. SELinux
    apiVersion: v1
    kind: Pod
    metadata:
    name: mypod
    spec:
    securityContext:
    seLinuxOptions:
    level: "s0:c123,c456"
    containers:
    - name: hello
    ...
    

    View Slide

33. Mitigate 2 & 3: Restrict Kubelet permissions
    ● RBAC for Kubelet:
    ○ --authorization-mode=RBAC,Node
    --admission-control=...,NodeRestriction
    ● Rotate Kubelet certs:
    ○ kubelet … --rotate-certificates
    

    View Slide

34. Unsecured Pods
    ●
    You follow the rules
    but others don't
    Kubernetes Cluster
    Web
    Frontend
    Redis
    user
    message
    

    View Slide

35. Mitigate: Open Policy Agent
    

    View Slide

36. Listening to Traffic
    1. Sniffing or
    intercepting traffic on
    the network
    2. Request forgery
    Kubernetes Cluster
    Web
    Frontend
    Redis
    user
    message
    ①
    ②
    

    View Slide

37. istio
    ●
    Service mesh
    ●
    Includes Envoy proxy
    

    View Slide

38. istio
    1. Proxy data between
    services
    2. End-to-end
    encryption
    3. Rolling certificates
    4. Policy managed by
    central server
    Kubernetes Cluster
    Web
    Frontend
    Redis
    user
    message
    

    View Slide

39. istio
    1. Proxy data between
    services
    2. End-to-end
    encryption
    3. Rolling certificates
    4. Policy managed by
    central server
    Kubernetes Cluster
    Web
    Frontend
    Redis
    user
    message
    

    View Slide

40. Tips
    1. Update Kubernetes early & often
    2. Don't use admin for day-to-day work
    3. Try benchmarking tools like kube-bench
    4. Use managed services like GKE
    

    View Slide

41. Thanks!
    

    View Slide