$30 off During Our Annual Pro Sale. View Details »

Good Practices for Secure Kubernetes AppOps

Good Practices for Secure Kubernetes AppOps

As a user of a managed Kubernetes cluster, someone else is responsible for security, right? Wrong! Well, at least that‘s not the whole truth. There are plenty of security options available when using K8s: RBAC, securityContexts, Network Policies, PodSecurityPolicies, Kernel Security Modules, etc.

But which ones are relevant for developers? And which are the most important ones?

This talk describes our good practices for K8s security established throughout the last years while developing and operating apps on K8s clusters. It features security options that can be applied with reasonable effort in our everyday lives as developers and shows the effects of these options on application security.

schnatterer

March 18, 2020
Tweet

More Decks by schnatterer

Other Decks in Programming

Transcript

  1. /
    //
    // GOOD PRACTICES FOR SECURE
    GOOD PRACTICES FOR SECURE
    KUBERNETES APPOPS
    KUBERNETES APPOPS
    JOHANNES SCHNATTERER
    CLOUDOGU GMBH
    VERSION: 202003191636-4481CC1
    1

    View Slide

  2. /
    Outline
    Outline
    How to improve application security
    How to improve application security
    using Kubernetes security built-ins
    using Kubernetes security built-ins
    pragmatically
    pragmatically
    2 . 1

    View Slide

  3. /
    K8s built-in security
    K8s built-in security
    mechanisms
    mechanisms
    • Network Policies
    • Security Context
    • Pod Security Policies
    2 . 2

    View Slide

  4. /
    Plenty of Options
    Plenty of Options
    • Secure by default?
    • How to improve pragmatically?
    2 . 3

    View Slide

  5. /
    Network Policies
    Network Policies
    (netpol)
    (netpol)
    3 . 1

    View Slide

  6. /
    A "firewall" for communication between pods.
    • Applied to pods
    • within namespace
    • via labels
    • Ingress / egress
    • to/from pods (in namespaces) or CIDRs (egress only)
    • for specific ports (optional)
    • Enforced by the CNI Plugin (e.g. Calico)
    • No Network Policies: All traffic allowed
    3 . 2

    View Slide

  7. /
    Helpful to get started
    Helpful to get started

    • Securing Cluster Networking with Network Policies - Ahmet Balkan
    • Interactively describes what a netpol does:
    https://github.com/ahmetb/kubernetes-network-policy-recipes
    https://www.youtube.com/watch?v=3gGpMmYeEO8
    kubectl describe netpol
    3 . 3

    View Slide

  8. /
    Recommendation: Whitelist ingress
    Recommendation: Whitelist ingress
    traffic
    traffic
    In every namespace except kube-system:
    • Deny ingress between pods,
    • then whitelist all allowed routes.
    3 . 4

    View Slide

  9. /
    Advanced: ingress to
    Advanced: ingress to kube-system
    kube-system
    Might stop the apps in your cluster from working
    Don't forget to:
    • Allow external access to ingress controller
    • Allow access to kube-dns/core-dns to every namespace
    3 . 5

    View Slide

  10. /
    Advanced: egress
    Advanced: egress
    • Verbose solution:
    • Deny egress between pods,
    • then whitelist all allowed routes,
    • repeating all ingress rules.
    • More pragmatic solution:
    • Allow only egress within the cluster,
    • then whitelist pods that need access to internet.
    3 . 6

    View Slide

  11. /
    Net pol pitfalls
    Net pol pitfalls
    • Whitelisting monitoring tools (e.g. Prometheus)
    • Restart might be necessary (e.g. Prometheus)
    • No labels on namespaces by default
    • egress more recent than ingress rules and less sophisticated
    • Policies might not be supported by CNI Plugin.
    Testing!
    https://www.inovex.de/blog/test-kubernetes-network-policies/
    3 . 7

    View Slide

  12. /
    More Features?
    More Features?
    • Proprietary extensions of CNI Plugin (e.g. cilium or calico)
    • Service Meshes: similar features, also work with multiple clusters
    different strengths, support each other
    https://istio.io/blog/2017/0.1-using-network-policy/
    3 . 8

    View Slide

  13. /
    Demo
    Demo
    K8s Cluster
    Namespace 'kube-system'
    Namespace 'default' Namespace 'production'
    Traefik
    NetPols
    Web Console
    NetPols NetPols
    nosqlclient
    mongodb
    HTTP
    web-console:80 nosqlclient:3000
    mongodb:/mongodb:27017


    nosqlclient
    web-console
    3 . 9

    View Slide

  14. /
    Wrap-Up: Network Policies
    Wrap-Up: Network Policies
    My recommendations:
    • Ingress whitelisting in non-kube-system namespaces
    • Use with care
    • whitelisting in kube-system
    • egress whitelisting for cluster-external traffic
    3 . 10

    View Slide

  15. /
    Security Context
    Security Context
    4 . 1

    View Slide

  16. /
    • Security Context: Defines security parameters per pod/container
    container runtime
    • Secure Pods - Tim Allclair
    • Cluster-wide security parameters: See Pod Security Policies
    https://www.youtube.com/watch?v=GLwmJh-j3rs
    4 . 2

    View Slide

  17. /
    Recommendations per Container
    Recommendations per Container
    apiVersion: v1
    kind: Pod
    metadata:
    annotations:
    seccomp.security.alpha.kubernetes.io/pod: runtime/default
    spec:
    containers:
    - name: restricted
    securityContext:
    runAsNonRoot: true
    runAsUser: 100000
    runAsGroup: 100000
    readOnlyRootFilesystem: true
    allowPrivilegeEscalation: false
    capabilities:
    drop:
    - ALL
    enableServiceLinks: false
    automountServiceAccountToken: false # When not communicating with API Server
    4 . 3

    View Slide

  18. /
    Recommendation per Container in
    Recommendation per Container in
    Detail
    Detail
    4 . 4

    View Slide

  19. /
    Enable seccomp
    Enable seccomp
    • Enables e.g. docker's seccomp default profile that block 44/~300
    Syscalls
    • Has mitigated Kernel vulns in past and might in future
    • See also k8s security audit:
    https://docs.docker.com/engine/security/non-events/
    https://www.cncf.io/blog/2019/08/06/open-sourcing-the-
    kubernetes-security-audit/
    4 . 5

    View Slide

  20. /
    Run as unprivileged user
    Run as unprivileged user
    • runAsNonRoot: true
    Container is not started when the user is root
    • runAsUser and runAsGroup > 10000
    • Reduces risk to run as user existing on host
    • In case of container escape UID/GID does not have
    privileges on host
    • E.g. mitigates vuln in runc (used by Docker among others)
    https://kubernetes.io/blog/2019/02/11/runc-and-cve-2019-5736/
    4 . 6

    View Slide

  21. /
    No Privilege escalation
    No Privilege escalation
    • Container can't increase privileges
    • E.g. sudo, setuid, Kernel vulnerabilities
    4 . 7

    View Slide

  22. /
    Read-only root file system
    Read-only root file system
    • Starts container without read-write layer
    • Writing only allowed in volumes
    • Config or code within the container cannot be manipulated
    4 . 8

    View Slide

  23. /
    Drop Capabilities
    Drop Capabilities
    • Drops even the default caps:
    • E.g. Mitigates CapNetRaw attack - DNS Spoofing on
    Kubernetes Clusters
    https://github.com/moby/moby/blob/3152f94/oci/caps/defaults.go
    https://blog.aquasec.com/dns-spoofing-kubernetes-clusters
    4 . 9

    View Slide

  24. /
    Bonus: No Services in Environment
    Bonus: No Services in Environment
    • By default: Each K8s service written to each container's env vars
    Docker Link legacy, no longer needed
    • But convenient info for attacker where to go next
    4 . 10

    View Slide

  25. /
    Bonus: Disable access to K8s API
    Bonus: Disable access to K8s API
    • SA Token in every pod for api-server authn
    • If not needed, disable!
    • No authentication possible
    • Lesser risk of security misconfig or vulns in authz
    curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
    -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"
    https://${KUBERNETES_SERVICE_HOST}/api/v1/
    4 . 11

    View Slide

  26. /
    Security context pitfalls
    Security context pitfalls
    4 . 12

    View Slide

  27. /
    Read-only root file system
    Read-only root file system
    Application might need temp folder to write to
    • Run image locally using docker, access app
    Run automated e2e/integration tests
    • Review container's read-write layer via
    • Mount folders as emptyDir volumes in pod
    docker diff
    4 . 13

    View Slide

  28. /
    Drop Capabilities
    Drop Capabilities
    Some images require capabilities
    • Find out needed Caps locally:
    • Add necessary caps to k8s resource
    • Alternative: Find image with same app that does not require caps,
    e.g. nginxinc/nginx-unprivileged
    docker run --rm --cap-drop ALL
    # Check error
    docker run --rm --cap-drop ALL --cap-add CAP_CHOWN
    # Keep adding caps until no more error
    4 . 14

    View Slide

  29. /
    Run as unprivileged user
    Run as unprivileged user
    • Some official images run as root by default.
    • Find a trusted image that does not run as root
    e.g. for mongo or postgres:
    • Create your own non-root image
    (potentially basing on original image)
    e.g. nginx:
    • Non-root verification only supports numeric user.
    • runAsUser: 100000 in securityContext of pod or
    • USER 100000 in Dockerfile of image.
    https://hub.docker.com/r/bitnami/
    https://github.com/schnatterer/nginx-unpriv
    4 . 15

    View Slide

  30. /
    • UID 100000 might not have permissions. Solutions:
    • Init Container sets permissions for PVCs
    • Permissions in image chmod/chown in Dockerfile
    • Run in root Group - GID 0
    https://docs.openshift.com/container-
    platform/4.3/openshift_images/create-images.html#images-
    create-guide-openshift_create-images
    4 . 16

    View Slide

  31. /
    Tools
    Tools
    Find out if your cluster adheres to these and other good security
    practices:
    • - managable amount of checks

    • a whole lot of checks,
    • even deny all ingress and egress NetPols and AppArmor
    Annotations
    Be prepared for a lot of findings
    Create your own good practices
    controlplaneio/kubesec
    Shopify/kubeaudit
    4 . 17

    View Slide

  32. /
    Demo
    Demo
    K8s Cluster
    Namespace 'sec-ctx'
    nginx nginxinc/nginx-unprivileged
    docker-sudo
    4 . 18

    View Slide

  33. /
    Wrap-Up: Security Context
    Wrap-Up: Security Context
    My recommendations:
    • Start with least privilege
    • Only differ if there's absolutely no other way
    4 . 19

    View Slide

  34. /
    Pod Security Policies
    Pod Security Policies
    (PSP)
    (PSP)
    5 . 1

    View Slide

  35. /
    • enforces security context cluster-wide
    • additional options enforcing secure defaults
    • more effort than security context and different syntax
    Still highly recommended!
    5 . 2

    View Slide

  36. /
    Recommendations
    Recommendations
    • Same as Security Context
    • Plus: Enforce secure defaults.
    Block pods from
    • entering node's Linux namespaces, e.g. net, PID
    (includes binding ports to nodes directly),
    • mounting arbitrary host paths (from node)
    (includes docker socket),
    • starting privileged containers,
    • changing apparmor profile
    5 . 3

    View Slide

  37. /
    Security Context Recommendations
    Security Context Recommendations
    PSP
    PSP
    apiVersion: policy/v1beta1
    kind: PodSecurityPolicy
    metadata:
    annotations:
    seccomp.security.alpha.kubernetes.io/defaultProfileName: runtime/default
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: runtime/default
    spec:
    requiredDropCapabilities:
    - ALL
    allowedCapabilities: []
    defaultAllowPrivilegeEscalation: false
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    runAsUser: # Same for runAsGroup, supplementalGroups, fsGroup
    rule: MustRunAs
    ranges:
    - min: 100000
    max: 999999
    5 . 4

    View Slide

  38. /
    Additional Recommendations
    Additional Recommendations
    apiVersion: policy/v1beta1
    kind: PodSecurityPolicy
    metadata:
    annotations:
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    spec:
    hostIPC: false
    hostPID: false
    hostNetwork: false
    hostPorts: []
    privileged: false
    allowedHostPaths: []
    volumes:
    - configMap
    - emptyDir
    - projected
    - secret
    - downwardAPI
    - persistentVolumeClaim
    5 . 5

    View Slide

  39. /
    Usage
    Usage
    1 Activate Admission controler via API-Server
    (also necessary for most managed k8s)
    2 Define PSP (YAML)
    3 Activate via RBAC
    Example:
    https://github.com/cloudogu/k8s-security-demos/blob/master/4-
    pod-security-policies/demo/01-psp-restrictive.yaml
    5 . 6

    View Slide

  40. /
    Activation via RBAC
    Activation via RBAC
    PodSecurityPolicy (Cluster-)Role (Cluster-)RoleBinding ServiceAccount Pod
    authorize use bind bind assign
    5 . 7

    View Slide

  41. /
    PSP pitfalls
    PSP pitfalls
    • Loose coupling in RBAC fail late with typos
    • AdmissionController
    • only evaluates Pods before starting
    • if not active PSP are ignored
    • if active but no PSP defined no pod can be started
    • Different PSP API group in (Cluster)Role
    • < 1.16: apiGroups [ extensions ]
    • ≥ 1.16: apiGroups [ policy ]
    5 . 8

    View Slide

  42. /
    PSP Debugging Hints
    PSP Debugging Hints
    # Query active PSP
    kubectl get pod -o jsonpath='{.metadata.annotations.kubernetes\.io/psp}'
    # Check authorization
    kubectl auth can-i use psp/privileged --as=system:serviceaccount::
    # Show which SA's are authorized (kubectl plugin)
    kubectl who-can use psp/
    # Show roles of a SA (kubectl plugin)
    kubectl rbac-lookup # e.g. subject = sa name
    5 . 9

    View Slide

  43. /
    PSP Limitations
    PSP Limitations
    • Unavailable options in PSPs
    • enableServiceLinks: false
    • automountServiceAccountToken: false
    • Future of PSPs uncertain
    Still easiest way for cluster-wide least privilege
    https://github.com/kubernetes/enhancements/issues/5
    5 . 10

    View Slide

  44. /
    What if pod requires more privileges?
    What if pod requires more privileges?
    "Whitelisting" via RBAC.
    PodSecurityPolicy (Cluster-)Role (Cluster-)RoleBinding ServiceAccount Pod
    authorize use bind bind assign
    1 Duplicate least privilege PSP
    2 Grant required privilege in new PSP
    3 Allow PSP via a Role (namespaced)
    4 Create ServiceAccount
    5 Create RoleBinding
    6 Assign ServiceAccount to Pod
    https://github.com/cloudogu/k8s-security-demos/blob/master/4-
    pod-security-policies/demo/02a-psp-whitelist.yaml
    5 . 11

    View Slide

  45. /
    Demo
    Demo
    K8s Cluster
    Namespace 'psp'
    psp/privileged psp/restricted
    nginxinc/nginx-unprivileged
    nginx
    5 . 12

    View Slide

  46. /
    Summary
    Summary
    • Don't allow arbitrary connections between pods, e.g. via NetPols
    • Start with least privilege for your containers
    • using either Security Context or
    • PSP
    6 . 1

    View Slide

  47. /
    Johannes Schnatterer
    Johannes Schnatterer
    Cloudogu GmbH
    K8s AppOps security series on JavaSPEKTRUM 05/2019+
    See also
    Demo Source:
    cloudogu.com/schulungen
    cloudogu.com/blog/tag/k8s-security
    @cloudogu
    @jschnatterer
    github.com/cloudogu/k8s-security-demos
    6 . 2

    View Slide