Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Lock your containers down!

Lock your containers down!

In this talk, we will take a look at different layers of security on Kubernetes and look at security best practices for containers running on Kubernetes. We will also talk about using OIDC to secure the kubernetes clusters so that it can scale with your organization. We will also talk about using Secrets effectively.

Deepu K Sasidharan

December 16, 2022
Tweet

More Decks by Deepu K Sasidharan

Other Decks in Programming

Transcript

  1. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Coming Up Next
    Lock your containers
    down!
    Deepu K Sasidharan (he/him)
    Staff Dev Advocate
    #DevDaysAmsterdam

    View Slide

  2. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    A Global Developer
    Conference by Auth0
    Lock your
    containers
    down!
    Deepu K Sasidharan
    Staff Developer Advocate
    A Global Developer
    Conference by Auth0

    View Slide

  3. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Understanding
    Kubernetes Security

    View Slide

  4. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Transport security
    All API communication is done via TLS using
    valid certificates

    View Slide

  5. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Authentication
    All API requests are authenticated with one of
    the several authentication mechanisms
    supported by Kubernetes

    View Slide

  6. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Authorization
    All authenticated requests are authorized using
    one or more of the supported authorization
    models

    View Slide

  7. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Admission control
    All authorized requests, except read/get
    requests, are validated by admission control
    modules

    View Slide

  8. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Kubernetes Security Best practices
    https://a0.to/k8s-security-best-practices

    View Slide

  9. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Use RBAC

    View Slide

  10. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Why?
    ● Most secure Authorization mechanism for
    Kubernetes
    ● Most widely used and most flexible
    ● Ideal for enterprise and medium-large orgs
    ● Easy to model business rules

    View Slide

  11. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    How?
    ● Check if RBAC is enabled
    ○ kubectl cluster-info dump | grep authorization-mode
    ● Use --authorization-mode flag for the API server to
    enable RBAC
    ● Create Role/ClusterRole and
    RoleBinding/ClusterRoleBinding as required

    View Slide

  12. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    How?
    apiVersion : rbac.authorization.k8s.io/v1
    kind: Role
    metadata :
    namespace : fancy-namespace
    name: pod-service-reader
    rules:
    - apiGroups : [""] # "" indicates the core API group
    resources : ["pods", "services” ]
    verbs: [ "get", "watch", "list"]

    apiVersion : rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata :
    name: read-pods-services
    namespace : fancy-namespace
    roleRef:
    kind: Role #this must be Role or ClusterRole
    name: pod-service-reader # this must match the name of the Role or ClusterRole you wish to bind to
    apiGroup : rbac.authorization.k8s.io
    subjects : # subject can be individual users or a group of users. Group is defined in the external
    authentication service, in this case, an OIDC server
    - kind: Group
    name: k8s-restricted-users

    View Slide

  13. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Use OpenID
    Connect

    View Slide

  14. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Why?
    ● Most secure Authentication mechanism
    ● Most scalable
    ● Ideal for clusters accessed by large teams
    as it provides a single sign-on solution
    ● Easy to onboard and offboard users

    View Slide

  15. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    How?

    View Slide

  16. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    How?
    How to Secure Your Kubernetes Cluster with
    OpenID Connect and RBAC
    https://a0.to/k8s-api-server-oidc

    View Slide

  17. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Use Secure
    Secrets

    View Slide

  18. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Why?
    ● Kubernetes Secrets are not very secure as
    its just base64 encoded strings
    ● Kubernetes Secrets cannot be stored in
    version control
    ● Kubernetes Secrets does not work with
    external secret managers

    View Slide

  19. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    How?
    ● Use Sealed Secrets
    ○ Uses asymmetric crypto encryption and
    supports certificate rotation
    ○ Can be stored in version control
    ○ Encrypted using unique key per cluster,
    namespace and secret
    ○ Can manage existing secrets
    ○ Ideal for small teams

    View Slide

  20. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    How?

    View Slide

  21. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    How?
    ● Use External Secrets Operator
    ○ Secrets are stored in external secret managers
    and is much more secure
    ○ Secrets are kept in sync
    ○ Works with HashiCorp Vault, Google Secrets
    Manager, AWS Secrets Manager and so on

    View Slide

  22. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    How?

    View Slide

  23. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    How?
    ● Use Secrets Store CSI driver
    ○ Secrets are stored in external secret managers
    and is much more secure
    ○ Secrets are mounted as volume on the pod
    ○ Secrets are kept in sync
    ○ Supports secret rotation
    ○ Works with HashiCorp Vault, Google Secrets
    Manager, AWS Secrets Manager and so on

    View Slide

  24. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    How?

    View Slide

  25. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    How?
    Shhhh... Kubernetes Secrets Are Not Really
    Secret!
    https://auth0.com/blog/kubernetes-secrets-management/

    View Slide

  26. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Keep Kubernetes
    version up to date

    View Slide

  27. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Why? ● Fix CVEs and other security bugs
    ● Latest features and security updates

    View Slide

  28. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    How?
    ● Check the Kubernetes security and
    disclosure information website to see if
    there are known security vulnerabilities for
    your version
    ● If you are using a managed PaaS, upgrade
    using built-in mechanism
    ● For on-prem installations, use tools like
    kOps, kubeadm, and so on, for easy
    upgrades

    View Slide

  29. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Restrict kubelet,
    API, and SSH
    access

    View Slide

  30. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Why?
    ● Restrict unintended access
    ● Non-admin users should not have API,
    SSH access

    View Slide

  31. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    How?
    ● Secure API server using OIDC and RBAC
    ● Disable SSH for non-admin users
    ● Secure kubelet’s HTTP endpoints

    View Slide

  32. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Control traffic
    between pods and
    clusters

    View Slide

  33. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Why?
    ● A compromised pod could compromise
    another leading to a chain reaction
    ● Larger attack surface
    ● Better traffic control and better security

    View Slide

  34. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    How?
    ● Use Kubernetes network policies to
    control traffic between pods and clusters
    ● Allow only necessary traffic between pods

    View Slide

  35. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Use namespaces
    to isolate
    workloads

    View Slide

  36. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Why?
    ● Isolating workloads in namespaces
    reduces attack surface
    ● Easier to manage with RBAC

    View Slide

  37. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    How?
    ● Avoid using default namespace
    ● Tune RBAC to restrict access to only
    required namespaces
    ● Use Kubernetes network policies to
    control traffic between namespaces

    View Slide

  38. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Limit resource
    usages

    View Slide

  39. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Why? ● Avoid denial of service (DoS) attacks
    ● Reduce attack surface

    View Slide

  40. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    How?
    ● Use resources quotas and limit ranges to
    set limits at the namespace level
    ● Set resource limits at container level as
    well

    View Slide

  41. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Use monitoring
    tools and enable
    audit logging

    View Slide

  42. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Why?
    ● Detect unauthorized access attempts
    ● Keep an eye on the traffic
    ● Prevent breaches before with alarms

    View Slide

  43. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    How?
    ● Enable audit logging for the cluster
    ● Use a monitoring tool to monitor
    ingress/egress networking traffic

    View Slide

  44. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Infrastructure best
    practices

    View Slide

  45. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    How?
    ● Ensure that all communication is done via
    TLS.
    ● Protect etcd with TLS, Firewall, and
    Encryption and restrict access to it using
    strong credentials.
    ● Set up IAM access policies in a supported
    environment like a PaaS.
    ● Secure the Kubernetes Control Plane.
    ● Rotate infrastructure credentials
    frequently.
    ● Restrict cloud metadata API access when
    running in a PaaS like AWS, Azure, or GCP.

    View Slide

  46. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Container Best practices
    https://a0.to/container-security

    View Slide

  47. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Do not run
    containers as root

    View Slide

  48. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Why?
    ● Principle of least privilege to reduce attack
    surface
    ● Avoid container escape and privilege
    escalations

    View Slide

  49. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    How?
    ● Use a least privileged user
    ● Use --chown=user:user when using
    Docker copy commands

    View Slide

  50. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Use minimal
    up-to-date official
    base images

    View Slide

  51. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Why? ● Reduce attack surface
    ● Latest bug fixes and security patches

    View Slide

  52. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    How?
    ● Use deterministic image tags - FROM
    node:14.2.0-alpine3.11 instead of FROM
    node:14-alpine
    ● Install only production dependencies
    ● Use official verified images for popular
    software. Prefer LTS versions.
    ● Use a trusted registry for non-official
    images and always verify the image
    publisher

    View Slide

  53. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Prevent loading
    unwanted kernel
    modules

    View Slide

  54. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Why? ● Reduce attack surface
    ● Better performance

    View Slide

  55. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    How?
    ● Restrict using rules in
    /etc/modprobe.d/kubernetes-blacklist.conf
    of the node
    ● Uninstall the unwanted modules from the
    node

    View Slide

  56. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Enable container
    image scanning in
    your CI/CD phase

    View Slide

  57. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Why? ● Detect known vulnerabilities before they
    are exploited

    View Slide

  58. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    How?
    ● Enable image scanning in CI/CD phase
    ● Use OSS tools like clair, Anchore or
    commercial tools like Snyk

    View Slide

  59. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Audit images

    View Slide

  60. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Why? ● Check for security best practices

    View Slide

  61. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    How?
    ● Use Docker Bench for Security to audit
    your container images

    View Slide

  62. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Use pod security
    policies

    View Slide

  63. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Why? ● Reduce attack surface
    ● Prevent privilege escalation

    View Slide

  64. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    How?
    ● Use Pod Security Admission to limit a
    container’s access to the host further

    View Slide

  65. © Okta and/or its affiliates. All rights reserved. Confidential Information of Okta – For Recipient’s Internal Use Only
    Thank you!
    Deepu K Sasidharan
    @[email protected] | deepu.tech

    View Slide

  66. View Slide