3 things every developer should know about K8s security - KubeCologne

A lot of developers deploy their apps on a Kubernetes Cluster that is „managed by someone else“™ like Google, Amazon, Microsoft or the ops department. This also means someone else is responsible for securing the cluster, right? Wrong! Well, at least that‘s not the whole truth. There are plenty of security options available when using K8s: RBAC, securityContexts, Network Policies, PodSecurityPolicies, Kernel Security Modules, Services Meshes, etc.

But which ones are relevant for developers? And which are the most important ones?

In this talk, I will describe my personal K8s security best practice established throughout the last years while developing applications on Kubernetes clusters. It contains security options that can be applied with reasonable effort in our everyday lives as software developers and shows the effects of these options on our application’s security.

Transcript

1. //
   // 3 THINGS EVERY DEVELOPER SHOULD
   3 THINGS EVERY DEVELOPER SHOULD
   KNOW ABOUT K8S SECURITY
   KNOW ABOUT K8S SECURITY
   JOHANNES SCHNATTERER
   CLOUDOGU GMBH
   VERSION: 201902080841-BEBF381
   @JSCHNATTERER
   1 . 1
   

   View Slide

2. Requirements for dev and ops
   Requirements for dev and ops
   https://iso25000.com/images/figures/en/iso25010.png
   1 . 2
   

   View Slide

3. End of Budget / Close to Deadline
   End of Budget / Close to Deadline
   Don't try this at home!
   1 . 3
   

   View Slide

4. Otherwise
   You might make the news...
   You might make the news...
   Like so many others
   Like so many others
   1 . 4
   

   View Slide

5. https://haveibeenpwned.com/PwnedWebsites
   2 . 1
   

   View Slide

6. Plenty of security options
   Plenty of security options
   2 . 2
   

   View Slide

7. runAsNonRoot runAsUser privileged procMount
   allowPrivilegeEscalation readOnlyRootFilesystem
   PodSecurityPolicy seccomp Linux Capabilities AppArmor
   SELinux Falco
   Open Policy Agent NetworkPolicy gVisor
   Kata Containers Nabla Containers mTLS KubeSec
   KubeBench kubetest Clair Vault Grafeas
   notary Bastion Host
   Certificate
   Rotation Threat detection
   SecOps
   securityContext
   RBAC
   Service Mesh
   3
   

   View Slide

8. ⏱
   4 . 1
   

   View Slide

9. 
   4 . 2
   

   View Slide

10. 
    4 . 3
    

    View Slide

11. 3 things every developer should know about K8s security
    3 things every developer should know about K8s security
    • a very opinionated list of actions that make a huge difference with manageable effort
    • distilled from the experience of the last years developing and operating apps on k8s
    4 . 4
    

    View Slide

12. 1. RBAC
    1. RBAC
    5 . 1
    

    View Slide

13. https://memegenerator.net/instance/83566913/homer-simpson-boring
    5 . 2
    

    View Slide

14. • RBAC active by default since K8s 1.6
    • ... but not if you migrated!
    5 . 3
    

    View Slide

15. • Every Container is mounted the token of its service account at
    /var/run/secrets/kubernetes.io/serviceaccount/token
    • With RBAC the default service account is only authorized to read publicly accessible API info
    • ⚠ With legacy authz the default service account is cluster admin
    • You can test if your pod is authorized by executing the following in it:
    • If a pod does not need access to K8s API, mounting the token can be disabled in the pod spec:
    automountServiceAccountToken: false
    curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
    -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    https://${KUBERNETES_SERVICE_HOST}/api/v1/secrets
    5 . 4
    

    View Slide

16. 
    Demo
    
    Demo
    Cluster legacy authz
    Namespace 'default'
    Cluster RBAC
    Namespace 'default'
    Web Console Web Console
    HTTPS
    •
    •
    legacy-authz
    RBAC
    5 . 5
    

    View Slide

17. 2. Network Policies (
    2. Network Policies (netpol
    netpol)
    )
    A kind of firewall for communication between pods.
    • Apply to pods (podSelector)
    • within a namespace
    • via labels
    • Ingress or egress
    • to/from pods (in namespaces) or CIDRs (egress only)
    • for specific ports (optional)
    • Are enforced by the CNI Plugin (e.g. Calico)
    • ⚠ No Network Policies: All traffic allowed
    6 . 1
    

    View Slide

18. Recommendation: Whitelist ingress traffic
    Recommendation: Whitelist ingress traffic
    In every namespace except kube-system:
    • Deny all ingress traffic between pods ...
    • ... and then whitelist all allowed routes.
    See also: Securing Cluster Networking with Network Policies - Ahmet Balkan
    
    https://github.com/ahmetb/kubernetes-network-policy-recipes
    https://www.youtube.com/watch?v=3gGpMmYeEO8
    6 . 2
    

    View Slide

19. Advanced: ingress to
    Advanced: ingress to kube-system
    kube-system namespace
    namespace
    ⚠ You might stop the apps in your cluster from working
    For example, don't forget to:
    • Allow external access to ingress controller
    (otherwise no more external access on any cluster resource)
    • Allow access to kube-dns/core-dns to every namespace
    (otherwise no more service discovery by name)
    6 . 3
    

    View Slide

20. Advanced: egress
    Advanced: egress
    • Verbose solution:
    • Deny all egress traffic between pods ...
    • ... and then whitelist all allowed routes...
    • ... repeating all ingress rules.
    • More pragmatic solution:
    • Allow only egress traffic within the cluster...
    • ... and then whitelist pods that need access to the internet.
    6 . 4
    

    View Slide

21. 
    Net pol pitfalls
    
    Net pol pitfalls
    • Don't forget to whitelist your monitoring tools (e.g. Prometheus)
    • A restart of the pods might be necessary for the netpol to become effective
    (e.g. Prometheus)
    • In order to match namespaces, labels need to be added to the namespaces, e.g.
    • Matching both pods and namespace is only possible from k8s 1.11+
    • Restricting kube-system might be more of a challenge (DNS, ingress controller)
    • egress rules are more recent feature than ingress rules and seem less sophisticated
    • Policies might not be supported by CNI Plugin.
    Make sure to test them!
    
    • On GKE: "at least 2 nodes of type n1-standard-1" are required
    kubectl label namespace/kube-system namespace=kube-system
    https://www.inovex.de/blog/test-kubernetes-network-policies/
    6 . 5
    

    View Slide

22. 
    Demo
    
    Demo
    Cluster Network Policies
    Namespace 'kube-system'
    Namespace 'default' Namespace 'production'
    Traefik
    Web Console nosqlclient
    mongodb
    HTTP
    web-console:80 nosqlclient:3000
    mongodb:/mongodb:27017
    •
    •
    nosqlclient
    web-console
    6 . 6
    

    View Slide

23. 
    Wrap-Up: Network Policies
    
    Wrap-Up: Network Policies
    • My recommendations
    • Definitely use DENY all ingress rule in non-kube-system namespaces
    • Use with care
    • rules in kube-system
    • egress rules
    • Helpful to get started - describe what a netpol does:
    kubectl describe netpol
    • Limitations:
    • netpols will not work in multi-cloud / cluster-federation scenarios
    • Service Meshes provides similar features and work also work with multiple clusters
    https://twitter.com/sebiwicb/status/962963928484630530
    

    View Slide

24. 3. Security Context
    3. Security Context
    Defines privilege and access control settings for a Pod or Container
    https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
    7 . 1
    

    View Slide

25. Recommendation per Container
    Recommendation per Container
    • allowPrivilegeEscalation: false
    • mitigates a process within the container from gaining higher privileges than its parent (the container
    process)
    • E.g. sudo, setuid, Kernel vulnerabilities
    • runAsNonRoot - Container is not started when the user is root
    • readOnlyRootFilesystem: true
    • Mounts the whole file system in the container read-only. Writing only allowed in volumes.
    • Makes sure that config or code within the container cannot be manipulated.
    • It's also more efficient (no CoW).
    7 . 2
    

    View Slide

26. There is also a securityContext on pod level, but those settings cannot be applied there.
    apiVersion: v1
    kind: Pod
    #...
    spec:
    containers:
    - name: cntnr
    securityContext:
    runAsNonRoot: true
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    7 . 3
    

    View Slide

27. 
    Security context pitfalls
    
    Security context pitfalls
    • readOnlyRootFilesystem - most applications need temp folders to write to
    • Run image locally using docker, access app ( run automated e2e/integration tests)
    • Then use docker diff to see a diff between container layer and image
    • and mount all folders listed there as emptyDir volumes in your pod
    • runAsNonRoot
    • Non-root verification only supports numeric user.
    Either set e.g.
    • runAsUser: 1000 in securityContext of pod or
    • USER 1000 in Dockerfile of image.
    • Some official images run as root by default.
    Possible solutions:
    • Find a trusted image that does not run as root
    e.g. for nginx, or postgres:
    • Derive from the original image and create your own non-root image
    e.g. nginx:
    https://hub.docker.com/r/bitnami/
    https://github.com/schnatterer/nginx-unpriv
    7 . 4
    

    View Slide

28. 
    Demo
    
    Demo
    Security Context & PSP
    Namespace 'wild-west'
    nginx bitnami/nginx
    docker-sudo
    7 . 5
    

    View Slide

29. 
    Wrap-Up: Security Context
    
    Wrap-Up: Security Context
    My recommendations
    • Security Context
    • Start with readOnlyRootFilesystem, runAsNonRoot and allowPrivilegeEscalation:
    false
    • Only differ if there's absolutely no other way
    • BTW - you can enforce Security Context Settings by using Pod Security Policies.
    However, those cause a lot more effort to maintain.
    7 . 6
    

    View Slide

30. Summary
    Summary
    IMHO ever person working with k8s should at least adhere to the following:
    • Enable RBAC!
    • Don't allow arbitrary connection between pods.
    (e.g use Network Policies to whitelist ingresses)
    • Don't allow privilege escalation via the security context of each container
    • Try to run your containers
    • as non-root user and
    • with a read-only file system.
    8 . 1
    

    View Slide

31. Johannes Schnatterer
    Johannes Schnatterer
    Cloudogu GmbH
    
    
    Demo Source:
    https://cloudogu.com/schulungen
    https://cloudogu.com/blog
    @jschnatterer
    @cloudogu
    https://github.com/cloudogu/k8s-security-demos
    8 . 2
    

    View Slide