Drei Dinge, die jeder Entwickler über Kubernetes Security wissen sollte - heise devSec 2019

1d07191e05a05762ac97f841e6845924?s=47 schnatterer
September 26, 2019

Drei Dinge, die jeder Entwickler über Kubernetes Security wissen sollte - heise devSec 2019

Viele Entwickler deployen ihre Apps auf Kubernetes-Clustern, die von anderen betrieben werden, wie Amazon oder "dem Betrieb". Das bedeutet, dass auch andere für die Cluster-Security verantwortlich sind, richtig? Nein, so einfach ist es nicht!
In Kubernetes (K8s) gibt es vielfältige Security-Optionen: RBAC, securityContexts, Network Policies, PodSecurityPolicies, etc. Doch welches sind die relevanten für Entwickler?
In diesem Talk werde ich meine persönliche K8s Security Best Practice vorstellen, die sich in den letzten Jahren als DevOps mit K8s-Clustern destilliert hat. Dabei wird praktisch gezeigt, wie mit überschaubarem Aufwand in der täglichen Arbeit die Anwendungssicherheit erhöht werden kann.

Vorkenntnisse
Grundlagen der Benutzung von K8s-Clustern

Lernziele
Mit überschaubarem Aufwand wichtige Angriffsvektoren auf Container in K8s abschwächen

1d07191e05a05762ac97f841e6845924?s=128

schnatterer

September 26, 2019
Tweet

Transcript

  1. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 1/43 // // 3 THINGS EVERY DEVELOPER SHOULD 3 THINGS EVERY DEVELOPER SHOULD KNOW ABOUT K8S SECURITY KNOW ABOUT K8S SECURITY JOHANNES SCHNATTERER CLOUDOGU GMBH VERSION: 201909252020-645D5FD 1
  2. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 2/43 https://haveibeenpwned.com/PwnedWebsites @haveibeenpwned 2
  3. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 3/43 https://haveibeenpwned.com/PwnedWebsites @haveibeenpwned 2
  4. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 4/43 https://de.wikipedia.org/wiki/Elfenbeinturm#/media/Datei:Altenmarkt_Kapelle_-_Elfenbeinerner_Turm.jpg 3 . 1
  5. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 5/43 3 . 2
  6. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 6/43 3 . 3
  7. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 7/43 What about Security? What about Security? 3 . 4
  8. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 8/43 3 . 5
  9. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 9/43 Plenty of security options Plenty of security options 3 . 6
  10. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 10/43 securityContext runAsNonRoot runAsUser privileged procMount allowPrivilegeEscalation readOnlyRootFilesystem PodSecurityPolicy RBAC seccomp Linux Capabilities AppArmor SELinux NetworkPolicy Falco Open Policy Agent gVisor Kata Containers Nabla Containers Service Mesh mTLS KubeSec KubeBench 4
  11. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 11/43 3 things every developer should know about K8s security 3 things every developer should know about K8s security • a very opinionated list of actions that make a huge difference with manageable effort • distilled from the experience of the last years developing and operating apps on k8s 5
  12. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 12/43 1. RBAC 1. RBAC 6 . 1
  13. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 13/43 https://memegenerator.net/instance/83566913/homer-simpson-boring 6 . 2
  14. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 14/43 • RBAC active by default since K8s 1.6 • ... but not if you migrated! 6 . 3
  15. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 15/43 • Every Container is mounted the token of its service account at /var/run/secrets/kubernetes.io/serviceaccount/token • With RBAC the default service account is only authorized to read publicly accessible API info • With legacy authz the default service account is cluster admin • You can test if your pod is authorized by executing the following in it: • If a pod does not need access to K8s API, mounting the token can be disabled in the pod spec: automountServiceAccountToken: false curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \ -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \ https://${KUBERNETES_SERVICE_HOST}/api/v1/secrets 6 . 4
  16. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 16/43 Demo Demo Cluster legacy authz Namespace 'default' Cluster RBAC Namespace 'default' Web Console Web Console HTTPS • • legacy-authz RBAC 6 . 5
  17. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 17/43 2. Network Policies ( 2. Network Policies (netpol netpol) ) 7 . 1
  18. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 18/43 A kind of firewall for communication between pods. • Apply to pods (podSelector) • within a namespace • via labels • Ingress or egress • to/from pods (in namespaces) or CIDRs (egress only) • for specific ports (optional) • Are enforced by the CNI Plugin (e.g. Calico) • No Network Policies: All traffic allowed 7 . 2
  19. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 19/43 Helpful to get started: Helpful to get started: • • Securing Cluster Networking with Network Policies - Ahmet Balkan • Interactively describes what a netpol does: https://github.com/ahmetb/kubernetes-network-policy-recipes https://www.youtube.com/watch?v=3gGpMmYeEO8 kubectl describe netpol <name> 7 . 3
  20. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 20/43 Recommendation: Whitelist ingress traffic Recommendation: Whitelist ingress traffic In every namespace except kube-system: • Deny all ingress traffic between pods ... • ... and then whitelist all allowed routes. 7 . 4
  21. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 21/43 Advanced: ingress to Advanced: ingress to kube-system kube-system namespace namespace You might stop the apps in your cluster from working For example, don't forget to: • Allow external access to ingress controller (otherwise no more external access on any cluster resource) • Allow access to kube-dns/core-dns to every namespace (otherwise no more service discovery by name) 7 . 5
  22. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 22/43 Advanced: egress Advanced: egress • Verbose solution: • Deny all egress traffic between pods ... • ... and then whitelist all allowed routes... • ... repeating all ingress rules. • More pragmatic solution: • Allow only egress traffic within the cluster... • ... and then whitelist pods that need access to the internet. 7 . 6
  23. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 23/43 Net pol pitfalls Net pol pitfalls • Don't forget to whitelist your monitoring tools (e.g. Prometheus) • A restart of the pods might be necessary for the netpol to become effective (e.g. Prometheus) • In order to match namespaces, labels need to be added to the namespaces, e.g. • Matching both pods and namespace is only possible from k8s 1.11+ • Restricting kube-system might be more of a challenge (DNS, ingress controller) • egress rules are more recent feature than ingress rules and seem less sophisticated • Policies might not be supported by CNI Plugin. Make sure to test them! • On GKE: "at least 2 nodes of type n1-standard-1" are required kubectl label namespace/kube-system namespace=kube-system https://www.inovex.de/blog/test-kubernetes-network-policies/ 7 . 7
  24. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 24/43 Limitations Limitations • no option for cluster-wide policies • whitelisting egress for domain names instead of CIDRs • filtering on L7 (e.g. HTTP or gRPC) • netpols will not work in multi-cloud / cluster-federation scenarios Possible solutions: • Proprietary extensions of CNI Plugin (e.g. cilium or calico) • Service Meshes provides similar features and work also work with multiple clusters. Service Meshes operate on L7, NetPol on L3/4 ➜ different strengths, support each other https://istio.io/blog/2017/0.1-using-network-policy/ 7 . 8
  25. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 25/43 Demo Demo Cluster Network Policies Namespace 'kube-system' Namespace 'default' Namespace 'production' Traefik Web Console nosqlclient mongodb HTTP web-console:80 nosqlclient:3000 mongodb:/mongodb:27017 • • nosqlclient web-console 7 . 9
  26. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 26/43 Wrap-Up: Network Policies Wrap-Up: Network Policies My recommendations: • Definitely use DENY all ingress rule in non-kube-system namespaces • Use with care • rules in kube-system • egress rules 7 . 10
  27. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 27/43 3. Security Context 3. Security Context 8 . 1
  28. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 28/43 Defines privilege and access control settings for a Pod or Container See also: Secure Pods - Tim Allclair https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ https://www.youtube.com/watch?v=GLwmJh-j3rs 8 . 2
  29. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 29/43 Recommendation per Container Recommendation per Container There is also a securityContext on pod level, but not all of those settings cannot be applied there. apiVersion: v1 kind: Pod # ... metadata: annotations: seccomp.security.alpha.kubernetes.io/pod: runtime/default spec: containers: - name: restricted securityContext: runAsNonRoot: true runAsUser: 100000 runAsGroup: 100000 readOnlyRootFilesystem: true allowPrivilegeEscalation: false capabilities: drop: - ALL 8 . 3
  30. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 30/43 Recommendation per Container in Detail (1) Recommendation per Container in Detail (1) • allowPrivilegeEscalation: false • mitigates a process within the container from gaining higher privileges than its parent (the container process) • E.g. sudo, setuid, Kernel vulnerabilities • seccomp.security.alpha.kubernetes.io/pod: runtime/default • Enables e.g. docker's seccomp default profile that block 44/~300 Syscalls • Has mitigated some Kernel vulns in the past and might in the future : • no seccomp profile is also one of the findings of the k8s security audit: • "capabilities": { "drop": [ "ALL" ] } • Reduces attack surface • Drops even the default caps: https://docs.docker.com/engine/security/non-events/ https://www.cncf.io/blog/2019/08/06/open-sourcing-the-kubernetes-security-audit/ https://github.com/moby/moby/blob/master/oci/defaults.go#L14-L30 8 . 4
  31. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 31/43 Recommendation per Container in Detail (2) Recommendation per Container in Detail (2) • runAsNonRoot: true - Container is not started when the user is root • runAsUser and runAsGroup > 10000 • Reduces risk to run as user existing on host • In case of container escape UID/GID does not have privileges on host/filesystem • readOnlyRootFilesystem: true • Mounts the whole file system in the container read-only. Writing only allowed in volumes. • Makes sure that config or code within the container cannot be manipulated. • It's also more efficient (no CoW). 8 . 5
  32. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 32/43 Security context pitfalls Security context pitfalls • readOnlyRootFilesystem - most applications need temp folders to write to • Run image locally using docker, access app ( run automated e2e/integration tests) • Then use docker diff to see a diff between container layer and image • and mount all folders listed there as emptyDir volumes in your pod • capabilities - some images require capabilities • Start container locally with docker and --cap-drop ALL, then check logs for errors • Start again add caps as needed with e.g. --cap-add CAP_CHOWN, check logs for errors • Start again with additional caps and so forth. • Add all necessary caps to k8s resource • Alternative: Find an image of same app that does not require caps, e.g. nginxinc/nginx- unprivileged • runAsGroup - beta from K8s 1.14. Before that defaults to GID 0 https://github.com/kubernetes/enhancements/issues/213 8 . 6
  33. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 33/43 Security context pitfalls - Security context pitfalls - runAsNonRoot runAsNonRoot • Non-root verification only supports numeric user. • runAsUser: 100000 in securityContext of pod or • USER 100000 in Dockerfile of image. • Some official images run as root by default. • Find a trusted image that does not run as root e.g. for nginx, or postgres: • Derive from the original image and create your own non-root image e.g. nginx: • UID 100000 might not have permissions to read/write. Possible solutions: • Init Container sets permissions for PVCs • Wrong permissions in container ➜ chmod/chown in Dockerfile • Some applications require a user for UID in /etc/passwd • New image that contains a user for UID e.g. 100000 or • Create /etc/passwd with user in init container and mount into application container https://hub.docker.com/r/bitnami/ https://github.com/schnatterer/nginx-unpriv 8 . 7
  34. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 34/43 Tools Tools Find out if your cluster adheres to these and other good security practices: • - managable amount of checks • • a whole lot of checks, • even deny all ingress and egress NetPols and AppArmor Annotations ➜ Be prepared for a lot of findings ➜ Create your own good practices controlplaneio/kubesec Shopify/kubeaudit 8 . 8
  35. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 35/43 Demo Demo Security Context & PSP Namespace 'wild-west' nginx nginxinc/nginx-unprivileged docker-sudo 8 . 9
  36. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 36/43 Wrap-Up: Security Context Wrap-Up: Security Context My recommendations • Security Context • Start with least privilege • Only differ if there's absolutely no other way • BTW - you can enforce Security Context Settings by using Pod Security Policies. However, those cause a lot more effort to maintain. 8 . 10
  37. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 37/43 4 things every developer should know about K8s security? 4 things every developer should know about K8s security? 9 . 1
  38. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 38/43 4 things every developer should know about K8s security? 4 things every developer should know about K8s security? Pod Security Policies (PSP) Pod Security Policies (PSP) 9 . 1
  39. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 39/43 a cluster-level resource [..] that define a set of conditions that a pod must run with in order to be accepted into the system https://kubernetes.io/docs/concepts/policy/pod-security-policy/ 9 . 2
  40. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 40/43 • can be used to enforce security context cluster-wide • has additional options such as block pods that try to • enter node's Linux namespaces (net, PID, etc.) • mounting the docker socket, • binding ports to nodes, • starting privileged containers • etc. • more effort than security context and different syntax as in securityContext ➜ Still highly recommended! 9 . 3
  41. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 41/43 Too much ground to cover for 45 min! See Demo Repo on last slide 9 . 4
  42. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 42/43 Summary Summary IMHO ever person working with k8s should at least adhere to the following: • Enable RBAC! • Don't allow arbitrary connection between pods. (e.g. use Network Policies to whitelist ingresses) • Start with least privilege for your containers: • Block privilege escalation via the security context of each container • Enable the seccomp default module via annotation of each pod • Try to run your containers • as non-root user, with UID & GID >= 10000, • with a read-only file system and • without capabilities. • Least privilege rules can either be set per container (securityContext) or cluster-wide (PodSecurityPolicy) 10 . 1
  43. 9/25/2019 3 things every developer should know about K8s security

    http://172.17.0.2:8000/?print-pdf#/ 43/43 Johannes Schnatterer Johannes Schnatterer Cloudogu GmbH K8s Security series on JavaSPEKTRUM starting 05/2019 See also Demo Source: https://cloudogu.com/schulungen https://cloudogu.com/blog @jschnatterer @cloudogu https://github.com/cloudogu/k8s-security-demos 10 . 2