$30 off During Our Annual Pro Sale. View Details »

Kubernetes Security Best Practices

Ian Lewis
December 12, 2019

Kubernetes Security Best Practices

Containers give developers the ability to isolate applications from one another, but that’s not enough. Resource isolation is much different that security isolation. How do we make applications deployed in containers more secure? How do we apply existing tools like SELinux and AppArmor, and seccomp to our containers running in Kubernetes? How can we apply policy to our network and services to make sure applications only have access to what they need and nothing more?

Kubernetes provides the ability to secure containers, and secure access to the API. But it also has a flexible enough architecture to allow for applying network and service policy to various pods and services.

In this talk we will learn about the risks and attack surfaces and how to use tools like SELinux, AppArmor and seccomp to improve the security of containers deployed in Kubernetes. We’ll then go up the stack and learn how to apply network policy to containers to further improve security. Finally we will look at the Istio service mesh and how we can add authentication, mutual TLS, and access policies to whole services greatly reducing application attack surfaces.

Ian Lewis

December 12, 2019
Tweet

More Decks by Ian Lewis

Other Decks in Technology

Transcript

  1. Kubernetes Security
    Best Practices
    ianmlewis@

    View Slide

  2. Ian Lewis

    @IanMLewis


    Tokyo, Japan

    #kubernetes, #go,
    #python

    View Slide

  3. Kubernetes

    Container
    Orchestrator

    An operations
    framework

    View Slide

  4. Topics

    Security 101

    Runtime Security

    Host Security

    Network Security

    Threat detection

    Build Hygiene

    Image Hygiene

    SecOps

    View Slide

  5. Topics

    Security 101

    Runtime Security

    Host Security

    Network Security

    Threat detection

    Build Hygiene

    Image Hygiene

    SecOps








    View Slide

  6. Security 101

    Security is a spectrum

    Attackers can pick their
    targets

    Provide as many hurdles
    between the threat and
    asset

    Attackers can shift focus.
    "It doesn't matter how
    many locks are on your
    door if your window is
    open"

    View Slide

  7. Security 101

    Layered Security/Defence
    in Depth

    Good security is redundant
    (not DRY)
    Network
    Node/Host
    Runtime

    View Slide

  8. Security 101

    Limit the attack surface
    Network
    Node/Host
    Runtime

    View Slide

  9. Security 101

    Least Privilege

    Only give as much
    permission/privilege as is
    absolutely necessary

    View Slide

  10. Guestbook app

    Frontend

    Serves web traffic

    Message

    Stores/lists
    messages

    User

    Authentication
    Kubernetes Cluster
    Web
    Frontend
    Redis
    user
    message

    View Slide

  11. Kubernetes API Server
    1. Get token from
    frontend Pod
    2. Use token to attack
    cluster API server
    3. Get secrets etc. to
    further attack
    Kubernetes Cluster
    Web
    Frontend
    Redis
    user
    message



    View Slide

  12. Mitigate 1 & 2: RBAC

    Role Based Access Control

    Roles given to users

    Each role has permission to perform some operation

    get secrets

    update configmap

    etc.

    RBAC settings apply to namespace

    View Slide

  13. Mitigate 1 & 2: RBAC
    ClusterRole
    ClusterRoleBinding
    1:many many:1
    Verb +
    Type

    View Slide

  14. Mitigate 2: API Server Firewall

    Restrict access to API server to certain IP addresses.

    GKE:
    ○ gcloud container clusters create
    --enable-master-authorized-networks
    --master-authorized-networks=....

    View Slide

  15. Mitigate 3: Network Policy

    Restrict access to database to only the Pods that require it

    Specify access via labels

    Implemented by Network solution: Calico, Weave, etc.

    View Slide

  16. NetworkPolicy
    kind: NetworkPolicy
    apiVersion: networking.k8s.io/v1
    metadata:
    name: redis
    spec:
    podSelector:
    matchLabels:
    name: redis
    ingress:
    - from:
    - podSelector:
    matchLabels:
    name: message

    View Slide

  17. Get access to cluster
    components
    1. Manipulate cluster
    data in etcd
    Host
    Web
    Frontend
    etcd

    View Slide

  18. Mitigate 1: Secure etcd

    Use authentication and firewalls to restrict access to etcd

    Encrypt data in etcd (encryption at rest)

    View Slide

  19. Get access to host
    1. Break out of the
    container using
    container or kernel
    exploits
    2. Attack the Kubelet
    3. Attack other
    containers running on
    the same host
    Host
    Web
    Frontend

    View Slide

  20. Mitigate 1: Run as non-root
    apiVersion: v1
    kind: Pod
    metadata:
    name: security-context-demo
    spec:
    securityContext:
    runAsUser: 1000

    View Slide

  21. Mitigate 1: Read only root filesystem
    apiVersion: v1
    kind: Pod
    metadata:
    name: security-context-demo
    spec:
    securityContext:
    readOnlyRootFilesystem: true

    View Slide

  22. Mitigate 1: no_new_privs
    apiVersion: v1
    kind: Pod
    metadata:
    name: security-context-demo
    spec:
    securityContext:
    allowPrivilegeEscalation: false

    View Slide

  23. Mitigate 1: Do them all
    apiVersion: v1
    kind: Pod
    metadata:
    name: security-context-demo
    spec:
    securityContext:
    runAsUser: 1000
    readOnlyRootFilesystem: true
    allowPrivilegeEscalation: false

    View Slide

  24. Mitigate 1: Sandboxed Pods
    1. Pods are sandboxed
    from other Pods on
    the same host
    2. Sandbox provides 2
    layers of isolation:
    Sandbox + Container
    (Linux Kernel)
    3. Examples: kata
    containers, gVisor
    Node Host
    Sandbox Pod
    Container
    Kernel
    Container
    Kubelet
    Runtime

    View Slide

  25. gVisor
    1. User space kernel
    2. Intercepts and
    implements syscalls in
    userspace
    3. Sandbox has low
    capabilities and runs
    with restricted
    seccomp filters
    Sandbox Process
    gVisor
    Sentry
    Application
    Gofer
    Kernel
    Ptrace/KVM
    Host Kernel
    9P
    seccomp +
    namespaces

    View Slide

  26. Your App
    Mitigate 1:
    seccomp/
    AppArmor/
    SELinux

    View Slide

  27. Container
    Mitigate 1:
    seccomp/
    AppArmor/
    SELinux

    View Slide

  28. seccomp
    Mitigate 1:
    seccomp/
    AppArmor/
    SELinux

    View Slide

  29. Mitigate 1:
    seccomp/
    AppArmor/
    SELinux
    AppArmor/
    SELinux

    View Slide

  30. seccomp
    apiVersion: v1
    kind: Pod
    metadata:
    name: mypod
    annotations:
    seccomp.security.alpha.kubernetes.io/pod: runtime/default
    ...

    View Slide

  31. AppArmor
    apiVersion: v1
    kind: Pod
    metadata:
    name: mypod
    annotations:
    container.apparmor.security.beta.kubernetes.io/hello: runtime/default
    spec:
    containers:
    - name: hello
    ...

    View Slide

  32. SELinux
    apiVersion: v1
    kind: Pod
    metadata:
    name: mypod
    spec:
    securityContext:
    seLinuxOptions:
    level: "s0:c123,c456"
    containers:
    - name: hello
    ...

    View Slide

  33. Mitigate 2 & 3: Restrict Kubelet permissions
    ● RBAC for Kubelet:
    ○ --authorization-mode=RBAC,Node
    --admission-control=...,NodeRestriction
    ● Rotate Kubelet certs:
    ○ kubelet … --rotate-certificates

    View Slide

  34. Unsecured Pods

    You follow the rules
    but others don't
    Kubernetes Cluster
    Web
    Frontend
    Redis
    user
    message

    View Slide

  35. Mitigate: Open Policy Agent

    View Slide

  36. Listening to Traffic
    1. Sniffing or
    intercepting traffic on
    the network
    2. Request forgery
    Kubernetes Cluster
    Web
    Frontend
    Redis
    user
    message


    View Slide

  37. istio

    Service mesh

    Includes Envoy proxy

    View Slide

  38. istio
    1. Proxy data between
    services
    2. End-to-end
    encryption
    3. Rolling certificates
    4. Policy managed by
    central server
    Kubernetes Cluster
    Web
    Frontend
    Redis
    user
    message

    View Slide

  39. istio
    1. Proxy data between
    services
    2. End-to-end
    encryption
    3. Rolling certificates
    4. Policy managed by
    central server
    Kubernetes Cluster
    Web
    Frontend
    Redis
    user
    message

    View Slide

  40. Tips
    1. Update Kubernetes early & often
    2. Don't use admin for day-to-day work
    3. Try benchmarking tools like kube-bench
    4. Use managed services like GKE

    View Slide

  41. Thanks!

    View Slide