Upgrade to Pro — share decks privately, control downloads, hide ads and more …

3 things every developer should know about K8s security - KubeCologne

schnatterer
February 08, 2019

3 things every developer should know about K8s security - KubeCologne

A lot of developers deploy their apps on a Kubernetes Cluster that is „managed by someone else“™ like Google, Amazon, Microsoft or the ops department. This also means someone else is responsible for securing the cluster, right? Wrong! Well, at least that‘s not the whole truth. There are plenty of security options available when using K8s: RBAC, securityContexts, Network Policies, PodSecurityPolicies, Kernel Security Modules, Services Meshes, etc.

But which ones are relevant for developers? And which are the most important ones?

In this talk, I will describe my personal K8s security best practice established throughout the last years while developing applications on Kubernetes clusters. It contains security options that can be applied with reasonable effort in our everyday lives as software developers and shows the effects of these options on our application’s security.

schnatterer

February 08, 2019
Tweet

More Decks by schnatterer

Other Decks in Technology

Transcript

  1. //
    // 3 THINGS EVERY DEVELOPER SHOULD
    3 THINGS EVERY DEVELOPER SHOULD
    KNOW ABOUT K8S SECURITY
    KNOW ABOUT K8S SECURITY
    JOHANNES SCHNATTERER
    CLOUDOGU GMBH
    VERSION: 201902080841-BEBF381
    @JSCHNATTERER
    1 . 1

    View Slide

  2. Requirements for dev and ops
    Requirements for dev and ops
    https://iso25000.com/images/figures/en/iso25010.png
    1 . 2

    View Slide

  3. End of Budget / Close to Deadline
    End of Budget / Close to Deadline
    Don't try this at home!
    1 . 3

    View Slide

  4. Otherwise
    You might make the news...
    You might make the news...
    Like so many others
    Like so many others
    1 . 4

    View Slide

  5. https://haveibeenpwned.com/PwnedWebsites
    2 . 1

    View Slide

  6. Plenty of security options
    Plenty of security options
    2 . 2

    View Slide

  7. runAsNonRoot runAsUser privileged procMount
    allowPrivilegeEscalation readOnlyRootFilesystem
    PodSecurityPolicy seccomp Linux Capabilities AppArmor
    SELinux Falco
    Open Policy Agent NetworkPolicy gVisor
    Kata Containers Nabla Containers mTLS KubeSec
    KubeBench kubetest Clair Vault Grafeas
    notary Bastion Host
    Certificate
    Rotation Threat detection
    SecOps
    securityContext
    RBAC
    Service Mesh
    3

    View Slide


  8. 4 . 1

    View Slide


  9. 4 . 2

    View Slide


  10. 4 . 3

    View Slide

  11. 3 things every developer should know about K8s security
    3 things every developer should know about K8s security
    • a very opinionated list of actions that make a huge difference with manageable effort
    • distilled from the experience of the last years developing and operating apps on k8s
    4 . 4

    View Slide

  12. 1. RBAC
    1. RBAC
    5 . 1

    View Slide

  13. https://memegenerator.net/instance/83566913/homer-simpson-boring
    5 . 2

    View Slide

  14. • RBAC active by default since K8s 1.6
    • ... but not if you migrated!
    5 . 3

    View Slide

  15. • Every Container is mounted the token of its service account at
    /var/run/secrets/kubernetes.io/serviceaccount/token
    • With RBAC the default service account is only authorized to read publicly accessible API info
    • ⚠ With legacy authz the default service account is cluster admin
    • You can test if your pod is authorized by executing the following in it:
    • If a pod does not need access to K8s API, mounting the token can be disabled in the pod spec:
    automountServiceAccountToken: false
    curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
    -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    https://${KUBERNETES_SERVICE_HOST}/api/v1/secrets
    5 . 4

    View Slide


  16. Demo

    Demo
    Cluster legacy authz
    Namespace 'default'
    Cluster RBAC
    Namespace 'default'
    Web Console Web Console
    HTTPS


    legacy-authz
    RBAC
    5 . 5

    View Slide

  17. 2. Network Policies (
    2. Network Policies (netpol
    netpol)
    )
    A kind of firewall for communication between pods.
    • Apply to pods (podSelector)
    • within a namespace
    • via labels
    • Ingress or egress
    • to/from pods (in namespaces) or CIDRs (egress only)
    • for specific ports (optional)
    • Are enforced by the CNI Plugin (e.g. Calico)
    • ⚠ No Network Policies: All traffic allowed
    6 . 1

    View Slide

  18. Recommendation: Whitelist ingress traffic
    Recommendation: Whitelist ingress traffic
    In every namespace except kube-system:
    • Deny all ingress traffic between pods ...
    • ... and then whitelist all allowed routes.
    See also: Securing Cluster Networking with Network Policies - Ahmet Balkan

    https://github.com/ahmetb/kubernetes-network-policy-recipes
    https://www.youtube.com/watch?v=3gGpMmYeEO8
    6 . 2

    View Slide

  19. Advanced: ingress to
    Advanced: ingress to kube-system
    kube-system namespace
    namespace
    ⚠ You might stop the apps in your cluster from working
    For example, don't forget to:
    • Allow external access to ingress controller
    (otherwise no more external access on any cluster resource)
    • Allow access to kube-dns/core-dns to every namespace
    (otherwise no more service discovery by name)
    6 . 3

    View Slide

  20. Advanced: egress
    Advanced: egress
    • Verbose solution:
    • Deny all egress traffic between pods ...
    • ... and then whitelist all allowed routes...
    • ... repeating all ingress rules.
    • More pragmatic solution:
    • Allow only egress traffic within the cluster...
    • ... and then whitelist pods that need access to the internet.
    6 . 4

    View Slide


  21. Net pol pitfalls

    Net pol pitfalls
    • Don't forget to whitelist your monitoring tools (e.g. Prometheus)
    • A restart of the pods might be necessary for the netpol to become effective
    (e.g. Prometheus)
    • In order to match namespaces, labels need to be added to the namespaces, e.g.
    • Matching both pods and namespace is only possible from k8s 1.11+
    • Restricting kube-system might be more of a challenge (DNS, ingress controller)
    • egress rules are more recent feature than ingress rules and seem less sophisticated
    • Policies might not be supported by CNI Plugin.
    Make sure to test them!

    • On GKE: "at least 2 nodes of type n1-standard-1" are required
    kubectl label namespace/kube-system namespace=kube-system
    https://www.inovex.de/blog/test-kubernetes-network-policies/
    6 . 5

    View Slide


  22. Demo

    Demo
    Cluster Network Policies
    Namespace 'kube-system'
    Namespace 'default' Namespace 'production'
    Traefik
    Web Console nosqlclient
    mongodb
    HTTP
    web-console:80 nosqlclient:3000
    mongodb:/mongodb:27017


    nosqlclient
    web-console
    6 . 6

    View Slide


  23. Wrap-Up: Network Policies

    Wrap-Up: Network Policies
    • My recommendations
    • Definitely use DENY all ingress rule in non-kube-system namespaces
    • Use with care
    • rules in kube-system
    • egress rules
    • Helpful to get started - describe what a netpol does:
    kubectl describe netpol
    • Limitations:
    • netpols will not work in multi-cloud / cluster-federation scenarios
    • Service Meshes provides similar features and work also work with multiple clusters
    https://twitter.com/sebiwicb/status/962963928484630530

    View Slide

  24. 3. Security Context
    3. Security Context
    Defines privilege and access control settings for a Pod or Container
    https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
    7 . 1

    View Slide

  25. Recommendation per Container
    Recommendation per Container
    • allowPrivilegeEscalation: false
    • mitigates a process within the container from gaining higher privileges than its parent (the container
    process)
    • E.g. sudo, setuid, Kernel vulnerabilities
    • runAsNonRoot - Container is not started when the user is root
    • readOnlyRootFilesystem: true
    • Mounts the whole file system in the container read-only. Writing only allowed in volumes.
    • Makes sure that config or code within the container cannot be manipulated.
    • It's also more efficient (no CoW).
    7 . 2

    View Slide

  26. There is also a securityContext on pod level, but those settings cannot be applied there.
    apiVersion: v1
    kind: Pod
    #...
    spec:
    containers:
    - name: cntnr
    securityContext:
    runAsNonRoot: true
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    7 . 3

    View Slide


  27. Security context pitfalls

    Security context pitfalls
    • readOnlyRootFilesystem - most applications need temp folders to write to
    • Run image locally using docker, access app ( run automated e2e/integration tests)
    • Then use docker diff to see a diff between container layer and image
    • and mount all folders listed there as emptyDir volumes in your pod
    • runAsNonRoot
    • Non-root verification only supports numeric user.
    Either set e.g.
    • runAsUser: 1000 in securityContext of pod or
    • USER 1000 in Dockerfile of image.
    • Some official images run as root by default.
    Possible solutions:
    • Find a trusted image that does not run as root
    e.g. for nginx, or postgres:
    • Derive from the original image and create your own non-root image
    e.g. nginx:
    https://hub.docker.com/r/bitnami/
    https://github.com/schnatterer/nginx-unpriv
    7 . 4

    View Slide


  28. Demo

    Demo
    Security Context & PSP
    Namespace 'wild-west'
    nginx bitnami/nginx
    docker-sudo
    7 . 5

    View Slide


  29. Wrap-Up: Security Context

    Wrap-Up: Security Context
    My recommendations
    • Security Context
    • Start with readOnlyRootFilesystem, runAsNonRoot and allowPrivilegeEscalation:
    false
    • Only differ if there's absolutely no other way
    • BTW - you can enforce Security Context Settings by using Pod Security Policies.
    However, those cause a lot more effort to maintain.
    7 . 6

    View Slide

  30. Summary
    Summary
    IMHO ever person working with k8s should at least adhere to the following:
    • Enable RBAC!
    • Don't allow arbitrary connection between pods.
    (e.g use Network Policies to whitelist ingresses)
    • Don't allow privilege escalation via the security context of each container
    • Try to run your containers
    • as non-root user and
    • with a read-only file system.
    8 . 1

    View Slide

  31. Johannes Schnatterer
    Johannes Schnatterer
    Cloudogu GmbH


    Demo Source:
    https://cloudogu.com/schulungen
    https://cloudogu.com/blog
    @jschnatterer
    @cloudogu
    https://github.com/cloudogu/k8s-security-demos
    8 . 2

    View Slide