3 things every developer should know about K8s security - KubeCologne

A lot of developers deploy their apps on a Kubernetes Cluster that is „managed by someone else“™ like Google, Amazon, Microsoft or the ops department. This also means someone else is responsible for securing the cluster, right? Wrong! Well, at least that‘s not the whole truth. There are plenty of security options available when using K8s: RBAC, securityContexts, Network Policies, PodSecurityPolicies, Kernel Security Modules, Services Meshes, etc.

But which ones are relevant for developers? And which are the most important ones?

In this talk, I will describe my personal K8s security best practice established throughout the last years while developing applications on Kubernetes clusters. It contains security options that can be applied with reasonable effort in our everyday lives as software developers and shows the effects of these options on our application’s security.

Transcript

1. // // 3 THINGS EVERY DEVELOPER SHOULD 3 THINGS EVERY DEVELOPER SHOULD KNOW ABOUT K8S SECURITY KNOW ABOUT K8S SECURITY JOHANNES SCHNATTERER CLOUDOGU GMBH VERSION: 201902080841-BEBF381 @JSCHNATTERER 1 . 1
2. Requirements for dev and ops Requirements for dev and ops https://iso25000.com/images/figures/en/iso25010.png 1 . 2
3. End of Budget / Close to Deadline End of Budget / Close to Deadline Don't try this at home! 1 . 3
4. Otherwise You might make the news... You might make the news... Like so many others Like so many others 1 . 4
5. https://haveibeenpwned.com/PwnedWebsites 2 . 1
6. Plenty of security options Plenty of security options 2 . 2
7. runAsNonRoot runAsUser privileged procMount allowPrivilegeEscalation readOnlyRootFilesystem PodSecurityPolicy seccomp Linux Capabilities AppArmor SELinux Falco Open Policy Agent NetworkPolicy gVisor Kata Containers Nabla Containers mTLS KubeSec KubeBench kubetest Clair Vault Grafeas notary Bastion Host Certificate Rotation Threat detection SecOps securityContext RBAC Service Mesh 3
8. ⏱ 4 . 1
9. 4 . 2
10. 4 . 3
11. 3 things every developer should know about K8s security 3 things every developer should know about K8s security • a very opinionated list of actions that make a huge difference with manageable effort • distilled from the experience of the last years developing and operating apps on k8s 4 . 4
12. 1. RBAC 1. RBAC 5 . 1
13. https://memegenerator.net/instance/83566913/homer-simpson-boring 5 . 2
14. • RBAC active by default since K8s 1.6 • ... but not if you migrated! 5 . 3
15. • Every Container is mounted the token of its service account at /var/run/secrets/kubernetes.io/serviceaccount/token • With RBAC the default service account is only authorized to read publicly accessible API info • ⚠ With legacy authz the default service account is cluster admin • You can test if your pod is authorized by executing the following in it: • If a pod does not need access to K8s API, mounting the token can be disabled in the pod spec: automountServiceAccountToken: false curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \ -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \ https://${KUBERNETES_SERVICE_HOST}/api/v1/secrets 5 . 4
16. Demo Demo Cluster legacy authz Namespace 'default' Cluster RBAC Namespace 'default' Web Console Web Console HTTPS • • legacy-authz RBAC 5 . 5
17. 2. Network Policies ( 2. Network Policies (netpol netpol) ) A kind of firewall for communication between pods. • Apply to pods (podSelector) • within a namespace • via labels • Ingress or egress • to/from pods (in namespaces) or CIDRs (egress only) • for specific ports (optional) • Are enforced by the CNI Plugin (e.g. Calico) • ⚠ No Network Policies: All traffic allowed 6 . 1
18. Recommendation: Whitelist ingress traffic Recommendation: Whitelist ingress traffic In every namespace except kube-system: • Deny all ingress traffic between pods ... • ... and then whitelist all allowed routes. See also: Securing Cluster Networking with Network Policies - Ahmet Balkan https://github.com/ahmetb/kubernetes-network-policy-recipes https://www.youtube.com/watch?v=3gGpMmYeEO8 6 . 2
19. Advanced: ingress to Advanced: ingress to kube-system kube-system namespace namespace ⚠ You might stop the apps in your cluster from working For example, don't forget to: • Allow external access to ingress controller (otherwise no more external access on any cluster resource) • Allow access to kube-dns/core-dns to every namespace (otherwise no more service discovery by name) 6 . 3
20. Advanced: egress Advanced: egress • Verbose solution: • Deny all egress traffic between pods ... • ... and then whitelist all allowed routes... • ... repeating all ingress rules. • More pragmatic solution: • Allow only egress traffic within the cluster... • ... and then whitelist pods that need access to the internet. 6 . 4
21. Net pol pitfalls Net pol pitfalls • Don't forget to whitelist your monitoring tools (e.g. Prometheus) • A restart of the pods might be necessary for the netpol to become effective (e.g. Prometheus) • In order to match namespaces, labels need to be added to the namespaces, e.g. • Matching both pods and namespace is only possible from k8s 1.11+ • Restricting kube-system might be more of a challenge (DNS, ingress controller) • egress rules are more recent feature than ingress rules and seem less sophisticated • Policies might not be supported by CNI Plugin. Make sure to test them! • On GKE: "at least 2 nodes of type n1-standard-1" are required kubectl label namespace/kube-system namespace=kube-system https://www.inovex.de/blog/test-kubernetes-network-policies/ 6 . 5
22. Demo Demo Cluster Network Policies Namespace 'kube-system' Namespace 'default' Namespace 'production' Traefik Web Console nosqlclient mongodb HTTP web-console:80 nosqlclient:3000 mongodb:/mongodb:27017 • • nosqlclient web-console 6 . 6
23. Wrap-Up: Network Policies Wrap-Up: Network Policies • My recommendations • Definitely use DENY all ingress rule in non-kube-system namespaces • Use with care • rules in kube-system • egress rules • Helpful to get started - describe what a netpol does: kubectl describe netpol <name> • Limitations: • netpols will not work in multi-cloud / cluster-federation scenarios • Service Meshes provides similar features and work also work with multiple clusters https://twitter.com/sebiwicb/status/962963928484630530
24. 3. Security Context 3. Security Context Defines privilege and access control settings for a Pod or Container https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ 7 . 1
25. Recommendation per Container Recommendation per Container • allowPrivilegeEscalation: false • mitigates a process within the container from gaining higher privileges than its parent (the container process) • E.g. sudo, setuid, Kernel vulnerabilities • runAsNonRoot - Container is not started when the user is root • readOnlyRootFilesystem: true • Mounts the whole file system in the container read-only. Writing only allowed in volumes. • Makes sure that config or code within the container cannot be manipulated. • It's also more efficient (no CoW). 7 . 2
26. There is also a securityContext on pod level, but those settings cannot be applied there. apiVersion: v1 kind: Pod #... spec: containers: - name: cntnr securityContext: runAsNonRoot: true allowPrivilegeEscalation: false readOnlyRootFilesystem: true 7 . 3
27. Security context pitfalls Security context pitfalls • readOnlyRootFilesystem - most applications need temp folders to write to • Run image locally using docker, access app ( run automated e2e/integration tests) • Then use docker diff to see a diff between container layer and image • and mount all folders listed there as emptyDir volumes in your pod • runAsNonRoot • Non-root verification only supports numeric user. Either set e.g. • runAsUser: 1000 in securityContext of pod or • USER 1000 in Dockerfile of image. • Some official images run as root by default. Possible solutions: • Find a trusted image that does not run as root e.g. for nginx, or postgres: • Derive from the original image and create your own non-root image e.g. nginx: https://hub.docker.com/r/bitnami/ https://github.com/schnatterer/nginx-unpriv 7 . 4
28. Demo Demo Security Context & PSP Namespace 'wild-west' nginx bitnami/nginx docker-sudo 7 . 5
29. Wrap-Up: Security Context Wrap-Up: Security Context My recommendations • Security Context • Start with readOnlyRootFilesystem, runAsNonRoot and allowPrivilegeEscalation: false • Only differ if there's absolutely no other way • BTW - you can enforce Security Context Settings by using Pod Security Policies. However, those cause a lot more effort to maintain. 7 . 6
30. Summary Summary IMHO ever person working with k8s should at least adhere to the following: • Enable RBAC! • Don't allow arbitrary connection between pods. (e.g use Network Policies to whitelist ingresses) • Don't allow privilege escalation via the security context of each container • Try to run your containers • as non-root user and • with a read-only file system. 8 . 1
31. Johannes Schnatterer Johannes Schnatterer Cloudogu GmbH Demo Source: https://cloudogu.com/schulungen https://cloudogu.com/blog @jschnatterer @cloudogu https://github.com/cloudogu/k8s-security-demos 8 . 2