Good Practices for Secure Kubernetes AppOps

As a user of a managed Kubernetes cluster, someone else is responsible for security, right? Wrong! Well, at least that‘s not the whole truth. There are plenty of security options available when using K8s: RBAC, securityContexts, Network Policies, PodSecurityPolicies, Kernel Security Modules, etc.

But which ones are relevant for developers? And which are the most important ones?

This talk describes our good practices for K8s security established throughout the last years while developing and operating apps on K8s clusters. It features security options that can be applied with reasonable effort in our everyday lives as developers and shows the effects of these options on application security.

Transcript

1. /
   //
   // GOOD PRACTICES FOR SECURE
   GOOD PRACTICES FOR SECURE
   KUBERNETES APPOPS
   KUBERNETES APPOPS
   JOHANNES SCHNATTERER
   CLOUDOGU GMBH
   VERSION: 202003191636-4481CC1
   1
   

   View Slide

2. /
   Outline
   Outline
   How to improve application security
   How to improve application security
   using Kubernetes security built-ins
   using Kubernetes security built-ins
   pragmatically
   pragmatically
   2 . 1
   

   View Slide

3. /
   K8s built-in security
   K8s built-in security
   mechanisms
   mechanisms
   • Network Policies
   • Security Context
   • Pod Security Policies
   2 . 2
   

   View Slide

4. /
   Plenty of Options
   Plenty of Options
   • Secure by default?
   • How to improve pragmatically?
   2 . 3
   

   View Slide

5. /
   Network Policies
   Network Policies
   (netpol)
   (netpol)
   3 . 1
   

   View Slide

6. /
   A "firewall" for communication between pods.
   • Applied to pods
   • within namespace
   • via labels
   • Ingress / egress
   • to/from pods (in namespaces) or CIDRs (egress only)
   • for specific ports (optional)
   • Enforced by the CNI Plugin (e.g. Calico)
   • No Network Policies: All traffic allowed
   3 . 2
   

   View Slide

7. /
   Helpful to get started
   Helpful to get started
   •
   • Securing Cluster Networking with Network Policies - Ahmet Balkan
   • Interactively describes what a netpol does:
   https://github.com/ahmetb/kubernetes-network-policy-recipes
   https://www.youtube.com/watch?v=3gGpMmYeEO8
   kubectl describe netpol
   3 . 3
   

   View Slide

8. /
   Recommendation: Whitelist ingress
   Recommendation: Whitelist ingress
   traffic
   traffic
   In every namespace except kube-system:
   • Deny ingress between pods,
   • then whitelist all allowed routes.
   3 . 4
   

   View Slide

9. /
   Advanced: ingress to
   Advanced: ingress to kube-system
   kube-system
   Might stop the apps in your cluster from working
   Don't forget to:
   • Allow external access to ingress controller
   • Allow access to kube-dns/core-dns to every namespace
   3 . 5
   

   View Slide

10. /
    Advanced: egress
    Advanced: egress
    • Verbose solution:
    • Deny egress between pods,
    • then whitelist all allowed routes,
    • repeating all ingress rules.
    • More pragmatic solution:
    • Allow only egress within the cluster,
    • then whitelist pods that need access to internet.
    3 . 6
    

    View Slide

11. /
    Net pol pitfalls
    Net pol pitfalls
    • Whitelisting monitoring tools (e.g. Prometheus)
    • Restart might be necessary (e.g. Prometheus)
    • No labels on namespaces by default
    • egress more recent than ingress rules and less sophisticated
    • Policies might not be supported by CNI Plugin.
    Testing!
    https://www.inovex.de/blog/test-kubernetes-network-policies/
    3 . 7
    

    View Slide

12. /
    More Features?
    More Features?
    • Proprietary extensions of CNI Plugin (e.g. cilium or calico)
    • Service Meshes: similar features, also work with multiple clusters
    different strengths, support each other
    https://istio.io/blog/2017/0.1-using-network-policy/
    3 . 8
    

    View Slide

13. /
    Demo
    Demo
    K8s Cluster
    Namespace 'kube-system'
    Namespace 'default' Namespace 'production'
    Traefik
    NetPols
    Web Console
    NetPols NetPols
    nosqlclient
    mongodb
    HTTP
    web-console:80 nosqlclient:3000
    mongodb:/mongodb:27017
    •
    •
    nosqlclient
    web-console
    3 . 9
    

    View Slide

14. /
    Wrap-Up: Network Policies
    Wrap-Up: Network Policies
    My recommendations:
    • Ingress whitelisting in non-kube-system namespaces
    • Use with care
    • whitelisting in kube-system
    • egress whitelisting for cluster-external traffic
    3 . 10
    

    View Slide

15. /
    Security Context
    Security Context
    4 . 1
    

    View Slide

16. /
    • Security Context: Defines security parameters per pod/container
    container runtime
    • Secure Pods - Tim Allclair
    • Cluster-wide security parameters: See Pod Security Policies
    https://www.youtube.com/watch?v=GLwmJh-j3rs
    4 . 2
    

    View Slide

17. /
    Recommendations per Container
    Recommendations per Container
    apiVersion: v1
    kind: Pod
    metadata:
    annotations:
    seccomp.security.alpha.kubernetes.io/pod: runtime/default
    spec:
    containers:
    - name: restricted
    securityContext:
    runAsNonRoot: true
    runAsUser: 100000
    runAsGroup: 100000
    readOnlyRootFilesystem: true
    allowPrivilegeEscalation: false
    capabilities:
    drop:
    - ALL
    enableServiceLinks: false
    automountServiceAccountToken: false # When not communicating with API Server
    4 . 3
    

    View Slide

18. /
    Recommendation per Container in
    Recommendation per Container in
    Detail
    Detail
    4 . 4
    

    View Slide

19. /
    Enable seccomp
    Enable seccomp
    • Enables e.g. docker's seccomp default profile that block 44/~300
    Syscalls
    • Has mitigated Kernel vulns in past and might in future
    • See also k8s security audit:
    https://docs.docker.com/engine/security/non-events/
    https://www.cncf.io/blog/2019/08/06/open-sourcing-the-
    kubernetes-security-audit/
    4 . 5
    

    View Slide

20. /
    Run as unprivileged user
    Run as unprivileged user
    • runAsNonRoot: true
    Container is not started when the user is root
    • runAsUser and runAsGroup > 10000
    • Reduces risk to run as user existing on host
    • In case of container escape UID/GID does not have
    privileges on host
    • E.g. mitigates vuln in runc (used by Docker among others)
    https://kubernetes.io/blog/2019/02/11/runc-and-cve-2019-5736/
    4 . 6
    

    View Slide

21. /
    No Privilege escalation
    No Privilege escalation
    • Container can't increase privileges
    • E.g. sudo, setuid, Kernel vulnerabilities
    4 . 7
    

    View Slide

22. /
    Read-only root file system
    Read-only root file system
    • Starts container without read-write layer
    • Writing only allowed in volumes
    • Config or code within the container cannot be manipulated
    4 . 8
    

    View Slide

23. /
    Drop Capabilities
    Drop Capabilities
    • Drops even the default caps:
    • E.g. Mitigates CapNetRaw attack - DNS Spoofing on
    Kubernetes Clusters
    https://github.com/moby/moby/blob/3152f94/oci/caps/defaults.go
    https://blog.aquasec.com/dns-spoofing-kubernetes-clusters
    4 . 9
    

    View Slide

24. /
    Bonus: No Services in Environment
    Bonus: No Services in Environment
    • By default: Each K8s service written to each container's env vars
    Docker Link legacy, no longer needed
    • But convenient info for attacker where to go next
    4 . 10
    

    View Slide

25. /
    Bonus: Disable access to K8s API
    Bonus: Disable access to K8s API
    • SA Token in every pod for api-server authn
    • If not needed, disable!
    • No authentication possible
    • Lesser risk of security misconfig or vulns in authz
    curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
    -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"
    https://${KUBERNETES_SERVICE_HOST}/api/v1/
    4 . 11
    

    View Slide

26. /
    Security context pitfalls
    Security context pitfalls
    4 . 12
    

    View Slide

27. /
    Read-only root file system
    Read-only root file system
    Application might need temp folder to write to
    • Run image locally using docker, access app
    Run automated e2e/integration tests
    • Review container's read-write layer via
    • Mount folders as emptyDir volumes in pod
    docker diff
    4 . 13
    

    View Slide

28. /
    Drop Capabilities
    Drop Capabilities
    Some images require capabilities
    • Find out needed Caps locally:
    • Add necessary caps to k8s resource
    • Alternative: Find image with same app that does not require caps,
    e.g. nginxinc/nginx-unprivileged
    docker run --rm --cap-drop ALL
    # Check error
    docker run --rm --cap-drop ALL --cap-add CAP_CHOWN
    # Keep adding caps until no more error
    4 . 14
    

    View Slide

29. /
    Run as unprivileged user
    Run as unprivileged user
    • Some official images run as root by default.
    • Find a trusted image that does not run as root
    e.g. for mongo or postgres:
    • Create your own non-root image
    (potentially basing on original image)
    e.g. nginx:
    • Non-root verification only supports numeric user.
    • runAsUser: 100000 in securityContext of pod or
    • USER 100000 in Dockerfile of image.
    https://hub.docker.com/r/bitnami/
    https://github.com/schnatterer/nginx-unpriv
    4 . 15
    

    View Slide

30. /
    • UID 100000 might not have permissions. Solutions:
    • Init Container sets permissions for PVCs
    • Permissions in image chmod/chown in Dockerfile
    • Run in root Group - GID 0
    https://docs.openshift.com/container-
    platform/4.3/openshift_images/create-images.html#images-
    create-guide-openshift_create-images
    4 . 16
    

    View Slide

31. /
    Tools
    Tools
    Find out if your cluster adheres to these and other good security
    practices:
    • - managable amount of checks
    •
    • a whole lot of checks,
    • even deny all ingress and egress NetPols and AppArmor
    Annotations
    Be prepared for a lot of findings
    Create your own good practices
    controlplaneio/kubesec
    Shopify/kubeaudit
    4 . 17
    

    View Slide

32. /
    Demo
    Demo
    K8s Cluster
    Namespace 'sec-ctx'
    nginx nginxinc/nginx-unprivileged
    docker-sudo
    4 . 18
    

    View Slide

33. /
    Wrap-Up: Security Context
    Wrap-Up: Security Context
    My recommendations:
    • Start with least privilege
    • Only differ if there's absolutely no other way
    4 . 19
    

    View Slide

34. /
    Pod Security Policies
    Pod Security Policies
    (PSP)
    (PSP)
    5 . 1
    

    View Slide

35. /
    • enforces security context cluster-wide
    • additional options enforcing secure defaults
    • more effort than security context and different syntax
    Still highly recommended!
    5 . 2
    

    View Slide

36. /
    Recommendations
    Recommendations
    • Same as Security Context
    • Plus: Enforce secure defaults.
    Block pods from
    • entering node's Linux namespaces, e.g. net, PID
    (includes binding ports to nodes directly),
    • mounting arbitrary host paths (from node)
    (includes docker socket),
    • starting privileged containers,
    • changing apparmor profile
    5 . 3
    

    View Slide

37. /
    Security Context Recommendations
    Security Context Recommendations
    PSP
    PSP
    apiVersion: policy/v1beta1
    kind: PodSecurityPolicy
    metadata:
    annotations:
    seccomp.security.alpha.kubernetes.io/defaultProfileName: runtime/default
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: runtime/default
    spec:
    requiredDropCapabilities:
    - ALL
    allowedCapabilities: []
    defaultAllowPrivilegeEscalation: false
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    runAsUser: # Same for runAsGroup, supplementalGroups, fsGroup
    rule: MustRunAs
    ranges:
    - min: 100000
    max: 999999
    5 . 4
    

    View Slide

38. /
    Additional Recommendations
    Additional Recommendations
    apiVersion: policy/v1beta1
    kind: PodSecurityPolicy
    metadata:
    annotations:
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    spec:
    hostIPC: false
    hostPID: false
    hostNetwork: false
    hostPorts: []
    privileged: false
    allowedHostPaths: []
    volumes:
    - configMap
    - emptyDir
    - projected
    - secret
    - downwardAPI
    - persistentVolumeClaim
    5 . 5
    

    View Slide

39. /
    Usage
    Usage
    1 Activate Admission controler via API-Server
    (also necessary for most managed k8s)
    2 Define PSP (YAML)
    3 Activate via RBAC
    Example:
    https://github.com/cloudogu/k8s-security-demos/blob/master/4-
    pod-security-policies/demo/01-psp-restrictive.yaml
    5 . 6
    

    View Slide

40. /
    Activation via RBAC
    Activation via RBAC
    PodSecurityPolicy (Cluster-)Role (Cluster-)RoleBinding ServiceAccount Pod
    authorize use bind bind assign
    5 . 7
    

    View Slide

41. /
    PSP pitfalls
    PSP pitfalls
    • Loose coupling in RBAC fail late with typos
    • AdmissionController
    • only evaluates Pods before starting
    • if not active PSP are ignored
    • if active but no PSP defined no pod can be started
    • Different PSP API group in (Cluster)Role
    • < 1.16: apiGroups [ extensions ]
    • ≥ 1.16: apiGroups [ policy ]
    5 . 8
    

    View Slide

42. /
    PSP Debugging Hints
    PSP Debugging Hints
    # Query active PSP
    kubectl get pod -o jsonpath='{.metadata.annotations.kubernetes\.io/psp}'
    # Check authorization
    kubectl auth can-i use psp/privileged --as=system:serviceaccount::
    # Show which SA's are authorized (kubectl plugin)
    kubectl who-can use psp/
    # Show roles of a SA (kubectl plugin)
    kubectl rbac-lookup # e.g. subject = sa name
    5 . 9
    

    View Slide

43. /
    PSP Limitations
    PSP Limitations
    • Unavailable options in PSPs
    • enableServiceLinks: false
    • automountServiceAccountToken: false
    • Future of PSPs uncertain
    Still easiest way for cluster-wide least privilege
    https://github.com/kubernetes/enhancements/issues/5
    5 . 10
    

    View Slide

44. /
    What if pod requires more privileges?
    What if pod requires more privileges?
    "Whitelisting" via RBAC.
    PodSecurityPolicy (Cluster-)Role (Cluster-)RoleBinding ServiceAccount Pod
    authorize use bind bind assign
    1 Duplicate least privilege PSP
    2 Grant required privilege in new PSP
    3 Allow PSP via a Role (namespaced)
    4 Create ServiceAccount
    5 Create RoleBinding
    6 Assign ServiceAccount to Pod
    https://github.com/cloudogu/k8s-security-demos/blob/master/4-
    pod-security-policies/demo/02a-psp-whitelist.yaml
    5 . 11
    

    View Slide

45. /
    Demo
    Demo
    K8s Cluster
    Namespace 'psp'
    psp/privileged psp/restricted
    nginxinc/nginx-unprivileged
    nginx
    5 . 12
    

    View Slide

46. /
    Summary
    Summary
    • Don't allow arbitrary connections between pods, e.g. via NetPols
    • Start with least privilege for your containers
    • using either Security Context or
    • PSP
    6 . 1
    

    View Slide

47. /
    Johannes Schnatterer
    Johannes Schnatterer
    Cloudogu GmbH
    K8s AppOps security series on JavaSPEKTRUM 05/2019+
    See also
    Demo Source:
    cloudogu.com/schulungen
    cloudogu.com/blog/tag/k8s-security
    @cloudogu
    @jschnatterer
    github.com/cloudogu/k8s-security-demos
    6 . 2
    

    View Slide