$30 off During Our Annual Pro Sale. View Details »

Drei Dinge, die jeder Entwickler über Kubernetes Security wissen sollte - heise devSec 2019

schnatterer
September 26, 2019

Drei Dinge, die jeder Entwickler über Kubernetes Security wissen sollte - heise devSec 2019

Viele Entwickler deployen ihre Apps auf Kubernetes-Clustern, die von anderen betrieben werden, wie Amazon oder "dem Betrieb". Das bedeutet, dass auch andere für die Cluster-Security verantwortlich sind, richtig? Nein, so einfach ist es nicht!
In Kubernetes (K8s) gibt es vielfältige Security-Optionen: RBAC, securityContexts, Network Policies, PodSecurityPolicies, etc. Doch welches sind die relevanten für Entwickler?
In diesem Talk werde ich meine persönliche K8s Security Best Practice vorstellen, die sich in den letzten Jahren als DevOps mit K8s-Clustern destilliert hat. Dabei wird praktisch gezeigt, wie mit überschaubarem Aufwand in der täglichen Arbeit die Anwendungssicherheit erhöht werden kann.

Vorkenntnisse
Grundlagen der Benutzung von K8s-Clustern

Lernziele
Mit überschaubarem Aufwand wichtige Angriffsvektoren auf Container in K8s abschwächen

schnatterer

September 26, 2019
Tweet

More Decks by schnatterer

Other Decks in Programming

Transcript

  1. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 1/43
    //
    // 3 THINGS EVERY DEVELOPER SHOULD
    3 THINGS EVERY DEVELOPER SHOULD
    KNOW ABOUT K8S SECURITY
    KNOW ABOUT K8S SECURITY
    JOHANNES SCHNATTERER CLOUDOGU GMBH
    VERSION: 201909252020-645D5FD
    1

    View Slide

  2. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 2/43
    https://haveibeenpwned.com/PwnedWebsites
    @haveibeenpwned
    2

    View Slide

  3. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 3/43
    https://haveibeenpwned.com/PwnedWebsites
    @haveibeenpwned
    2

    View Slide

  4. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 4/43
    https://de.wikipedia.org/wiki/Elfenbeinturm#/media/Datei:Altenmarkt_Kapelle_-_Elfenbeinerner_Turm.jpg
    3 . 1

    View Slide

  5. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 5/43
    3 . 2

    View Slide

  6. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 6/43
    3 . 3

    View Slide

  7. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 7/43
    What about Security?
    What about Security?
    3 . 4

    View Slide

  8. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 8/43
    3 . 5

    View Slide

  9. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 9/43
    Plenty of security options
    Plenty of security options
    3 . 6

    View Slide

  10. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 10/43
    securityContext
    runAsNonRoot runAsUser privileged
    procMount
    allowPrivilegeEscalation readOnlyRootFilesystem PodSecurityPolicy
    RBAC seccomp
    Linux Capabilities AppArmor SELinux
    NetworkPolicy Falco Open Policy Agent gVisor Kata Containers Nabla Containers
    Service Mesh mTLS
    KubeSec KubeBench
    4

    View Slide

  11. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 11/43
    3 things every developer should know about K8s security
    3 things every developer should know about K8s security
    • a very opinionated list of actions that make a huge difference with manageable effort
    • distilled from the experience of the last years developing and operating apps on k8s
    5

    View Slide

  12. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 12/43
    1. RBAC
    1. RBAC
    6 . 1

    View Slide

  13. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 13/43
    https://memegenerator.net/instance/83566913/homer-simpson-boring
    6 . 2

    View Slide

  14. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 14/43
    • RBAC active by default since K8s 1.6
    • ... but not if you migrated!
    6 . 3

    View Slide

  15. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 15/43
    • Every Container is mounted the token of its service account at
    /var/run/secrets/kubernetes.io/serviceaccount/token
    • With RBAC the default service account is only authorized to read publicly accessible API info
    • With legacy authz the default service account is cluster admin
    • You can test if your pod is authorized by executing the following in it:
    • If a pod does not need access to K8s API, mounting the token can be disabled in the pod spec:
    automountServiceAccountToken: false
    curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
    -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    https://${KUBERNETES_SERVICE_HOST}/api/v1/secrets
    6 . 4

    View Slide

  16. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 16/43
    Demo
    Demo
    Cluster legacy authz
    Namespace 'default'
    Cluster RBAC
    Namespace 'default'
    Web Console Web Console
    HTTPS


    legacy-authz
    RBAC
    6 . 5

    View Slide

  17. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 17/43
    2. Network Policies (
    2. Network Policies (netpol
    netpol)
    )
    7 . 1

    View Slide

  18. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 18/43
    A kind of firewall for communication between pods.
    • Apply to pods (podSelector)
    • within a namespace
    • via labels
    • Ingress or egress
    • to/from pods (in namespaces) or CIDRs (egress only)
    • for specific ports (optional)
    • Are enforced by the CNI Plugin (e.g. Calico)
    • No Network Policies: All traffic allowed
    7 . 2

    View Slide

  19. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 19/43
    Helpful to get started:
    Helpful to get started:

    • Securing Cluster Networking with Network Policies - Ahmet Balkan
    • Interactively describes what a netpol does:
    https://github.com/ahmetb/kubernetes-network-policy-recipes
    https://www.youtube.com/watch?v=3gGpMmYeEO8
    kubectl describe netpol
    7 . 3

    View Slide

  20. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 20/43
    Recommendation: Whitelist ingress traffic
    Recommendation: Whitelist ingress traffic
    In every namespace except kube-system:
    • Deny all ingress traffic between pods ...
    • ... and then whitelist all allowed routes.
    7 . 4

    View Slide

  21. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 21/43
    Advanced: ingress to
    Advanced: ingress to kube-system
    kube-system namespace
    namespace
    You might stop the apps in your cluster from working
    For example, don't forget to:
    • Allow external access to ingress controller
    (otherwise no more external access on any cluster resource)
    • Allow access to kube-dns/core-dns to every namespace
    (otherwise no more service discovery by name)
    7 . 5

    View Slide

  22. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 22/43
    Advanced: egress
    Advanced: egress
    • Verbose solution:
    • Deny all egress traffic between pods ...
    • ... and then whitelist all allowed routes...
    • ... repeating all ingress rules.
    • More pragmatic solution:
    • Allow only egress traffic within the cluster...
    • ... and then whitelist pods that need access to the internet.
    7 . 6

    View Slide

  23. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 23/43
    Net pol pitfalls
    Net pol pitfalls
    • Don't forget to whitelist your monitoring tools (e.g. Prometheus)
    • A restart of the pods might be necessary for the netpol to become effective
    (e.g. Prometheus)
    • In order to match namespaces, labels need to be added to the namespaces, e.g.
    • Matching both pods and namespace is only possible from k8s 1.11+
    • Restricting kube-system might be more of a challenge (DNS, ingress controller)
    • egress rules are more recent feature than ingress rules and seem less sophisticated
    • Policies might not be supported by CNI Plugin.
    Make sure to test them!
    • On GKE: "at least 2 nodes of type n1-standard-1" are required
    kubectl label namespace/kube-system namespace=kube-system
    https://www.inovex.de/blog/test-kubernetes-network-policies/
    7 . 7

    View Slide

  24. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 24/43
    Limitations
    Limitations
    • no option for cluster-wide policies
    • whitelisting egress for domain names instead of CIDRs
    • filtering on L7 (e.g. HTTP or gRPC)
    • netpols will not work in multi-cloud / cluster-federation scenarios
    Possible solutions:
    • Proprietary extensions of CNI Plugin (e.g. cilium or calico)
    • Service Meshes provides similar features and work also work with multiple clusters.
    Service Meshes operate on L7, NetPol on L3/4
    ➜ different strengths, support each other
    https://istio.io/blog/2017/0.1-using-network-policy/
    7 . 8

    View Slide

  25. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 25/43
    Demo
    Demo
    Cluster Network Policies
    Namespace 'kube-system'
    Namespace 'default' Namespace 'production'
    Traefik
    Web Console nosqlclient
    mongodb
    HTTP
    web-console:80 nosqlclient:3000
    mongodb:/mongodb:27017


    nosqlclient
    web-console
    7 . 9

    View Slide

  26. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 26/43
    Wrap-Up: Network Policies
    Wrap-Up: Network Policies
    My recommendations:
    • Definitely use DENY all ingress rule in non-kube-system namespaces
    • Use with care
    • rules in kube-system
    • egress rules
    7 . 10

    View Slide

  27. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 27/43
    3. Security Context
    3. Security Context
    8 . 1

    View Slide

  28. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 28/43
    Defines privilege and access control settings for a Pod or Container
    See also: Secure Pods - Tim Allclair
    https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
    https://www.youtube.com/watch?v=GLwmJh-j3rs
    8 . 2

    View Slide

  29. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 29/43
    Recommendation per Container
    Recommendation per Container
    There is also a securityContext on pod level, but not all of those settings cannot be applied there.
    apiVersion: v1
    kind: Pod
    # ...
    metadata:
    annotations:
    seccomp.security.alpha.kubernetes.io/pod: runtime/default
    spec:
    containers:
    - name: restricted
    securityContext:
    runAsNonRoot: true
    runAsUser: 100000
    runAsGroup: 100000
    readOnlyRootFilesystem: true
    allowPrivilegeEscalation: false
    capabilities:
    drop:
    - ALL
    8 . 3

    View Slide

  30. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 30/43
    Recommendation per Container in Detail (1)
    Recommendation per Container in Detail (1)
    • allowPrivilegeEscalation: false
    • mitigates a process within the container from gaining higher privileges than its parent (the container
    process)
    • E.g. sudo, setuid, Kernel vulnerabilities
    • seccomp.security.alpha.kubernetes.io/pod: runtime/default
    • Enables e.g. docker's seccomp default profile that block 44/~300 Syscalls
    • Has mitigated some Kernel vulns in the past and might in the future :
    • no seccomp profile is also one of the findings of the k8s security audit:
    • "capabilities": { "drop": [ "ALL" ] }
    • Reduces attack surface
    • Drops even the default caps:
    https://docs.docker.com/engine/security/non-events/
    https://www.cncf.io/blog/2019/08/06/open-sourcing-the-kubernetes-security-audit/
    https://github.com/moby/moby/blob/master/oci/defaults.go#L14-L30
    8 . 4

    View Slide

  31. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 31/43
    Recommendation per Container in Detail (2)
    Recommendation per Container in Detail (2)
    • runAsNonRoot: true - Container is not started when the user is root
    • runAsUser and runAsGroup > 10000
    • Reduces risk to run as user existing on host
    • In case of container escape UID/GID does not have privileges on host/filesystem
    • readOnlyRootFilesystem: true
    • Mounts the whole file system in the container read-only. Writing only allowed in volumes.
    • Makes sure that config or code within the container cannot be manipulated.
    • It's also more efficient (no CoW).
    8 . 5

    View Slide

  32. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 32/43
    Security context pitfalls
    Security context pitfalls
    • readOnlyRootFilesystem - most applications need temp folders to write to
    • Run image locally using docker, access app ( run automated e2e/integration tests)
    • Then use docker diff to see a diff between container layer and image
    • and mount all folders listed there as emptyDir volumes in your pod
    • capabilities - some images require capabilities
    • Start container locally with docker and --cap-drop ALL, then check logs for errors
    • Start again add caps as needed with e.g. --cap-add CAP_CHOWN, check logs for errors
    • Start again with additional caps and so forth.
    • Add all necessary caps to k8s resource
    • Alternative: Find an image of same app that does not require caps, e.g. nginxinc/nginx-
    unprivileged
    • runAsGroup - beta from K8s 1.14. Before that defaults to GID 0
    https://github.com/kubernetes/enhancements/issues/213
    8 . 6

    View Slide

  33. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 33/43
    Security context pitfalls -
    Security context pitfalls - runAsNonRoot
    runAsNonRoot
    • Non-root verification only supports numeric user.
    • runAsUser: 100000 in securityContext of pod or
    • USER 100000 in Dockerfile of image.
    • Some official images run as root by default.
    • Find a trusted image that does not run as root
    e.g. for nginx, or postgres:
    • Derive from the original image and create your own non-root image
    e.g. nginx:
    • UID 100000 might not have permissions to read/write. Possible solutions:
    • Init Container sets permissions for PVCs
    • Wrong permissions in container ➜ chmod/chown in Dockerfile
    • Some applications require a user for UID in /etc/passwd
    • New image that contains a user for UID e.g. 100000 or
    • Create /etc/passwd with user in init container and mount into application container
    https://hub.docker.com/r/bitnami/
    https://github.com/schnatterer/nginx-unpriv
    8 . 7

    View Slide

  34. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 34/43
    Tools
    Tools
    Find out if your cluster adheres to these and other good security practices:
    • - managable amount of checks

    • a whole lot of checks,
    • even deny all ingress and egress NetPols and AppArmor Annotations
    ➜ Be prepared for a lot of findings
    ➜ Create your own good practices
    controlplaneio/kubesec
    Shopify/kubeaudit
    8 . 8

    View Slide

  35. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 35/43
    Demo
    Demo
    Security Context & PSP
    Namespace 'wild-west'
    nginx nginxinc/nginx-unprivileged
    docker-sudo
    8 . 9

    View Slide

  36. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 36/43
    Wrap-Up: Security Context
    Wrap-Up: Security Context
    My recommendations
    • Security Context
    • Start with least privilege
    • Only differ if there's absolutely no other way
    • BTW - you can enforce Security Context Settings by using Pod Security Policies.
    However, those cause a lot more effort to maintain.
    8 . 10

    View Slide

  37. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 37/43
    4 things every developer should know about K8s security?
    4 things every developer should know about K8s security?
    9 . 1

    View Slide

  38. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 38/43
    4 things every developer should know about K8s security?
    4 things every developer should know about K8s security?
    Pod Security Policies (PSP)
    Pod Security Policies (PSP)
    9 . 1

    View Slide

  39. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 39/43
    a cluster-level resource [..] that define a set of conditions that a pod must run with in order to be
    accepted into the system
    https://kubernetes.io/docs/concepts/policy/pod-security-policy/
    9 . 2

    View Slide

  40. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 40/43
    • can be used to enforce security context cluster-wide
    • has additional options such as block pods that try to
    • enter node's Linux namespaces (net, PID, etc.)
    • mounting the docker socket,
    • binding ports to nodes,
    • starting privileged containers
    • etc.
    • more effort than security context and different syntax as in securityContext
    ➜ Still highly recommended!
    9 . 3

    View Slide

  41. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 41/43
    Too much ground to cover for 45 min!
    See Demo Repo on last slide
    9 . 4

    View Slide

  42. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 42/43
    Summary
    Summary
    IMHO ever person working with k8s should at least adhere to the following:
    • Enable RBAC!
    • Don't allow arbitrary connection between pods.
    (e.g. use Network Policies to whitelist ingresses)
    • Start with least privilege for your containers:
    • Block privilege escalation via the security context of each container
    • Enable the seccomp default module via annotation of each pod
    • Try to run your containers
    • as non-root user, with UID & GID >= 10000,
    • with a read-only file system and
    • without capabilities.
    • Least privilege rules can either be set per container (securityContext) or cluster-wide (PodSecurityPolicy)
    10 . 1

    View Slide

  43. 9/25/2019 3 things every developer should know about K8s security
    http://172.17.0.2:8000/?print-pdf#/ 43/43
    Johannes Schnatterer
    Johannes Schnatterer
    Cloudogu GmbH
    K8s Security series on JavaSPEKTRUM starting 05/2019
    See also
    Demo Source:
    https://cloudogu.com/schulungen
    https://cloudogu.com/blog
    @jschnatterer
    @cloudogu
    https://github.com/cloudogu/k8s-security-demos
    10 . 2

    View Slide