$30 off During Our Annual Pro Sale. View Details »

Multi-tenancy in Kubernetes: Sharing a Cluster with Hundreds of Users

Multi-tenancy in Kubernetes: Sharing a Cluster with Hundreds of Users

Security primitives provided by Kubernetes in a multi-tenant context and Hasura's experiences of running a multi-tenant k8s cluster in production.

Presented at Infrastructure Engineering @Scale Meetup at LinkedIn Office, Bangalore.

Shahidh K Muhammed

May 26, 2018
Tweet

More Decks by Shahidh K Muhammed

Other Decks in Technology

Transcript

  1. Multi-tenancy in Kubernetes:
    Sharing a Cluster with
    Hundreds of Users

    View Slide

  2. Shahidh K
    Muhammed
    @shahidh_k

    View Slide

  3. Outline
    ● What is multi-tenancy?
    ● Why multi-tenancy?
    ● Challenges involved in multi-tenancy
    ● Security primitives in Kubernetes
    ● Managing shared resources in Kubernetes
    ● Simplifying operations on Kubernetes - Operator

    View Slide

  4. What is
    Multi-tenancy?
    “Mode of operation of software where
    multiple independent instances of one or
    more applications operate in a shared
    environment” -- Gartner

    View Slide

  5. What is Multi-tenancy
    source

    View Slide

  6. Types of Multi-tenancy
    Soft Multi-tenancy
    ● Non adversarial tenants
    ● Different department/teams in the same
    company
    ● Not trying to harm other tenants
    ● Focus on preventing accidents
    Hard Multi-tenancy
    ● Adversarial tenants
    ● Different kinds of users who has no relation
    to each other
    ● Trying to exploit the system
    ● Focus on securing and isolating each tenant

    View Slide

  7. Why multi-tenancy in
    Kubernetes?

    View Slide

  8. Resource sharing
    ● Containers from multiple-tenants can be fit into one node.
    ● Tight packing of containers helps save resources especially for bursty workloads
    ● Cost savings
    ● Operational burden?

    View Slide

  9. One cluster for all But…
    ● Need to manage each tenant
    ● Different type of complexity
    ● Setup and maintain a single
    cluster
    ● Homogenous environment for
    all tenants

    View Slide

  10. Multi-tenant Security in
    Kubernetes

    View Slide

  11. Kubernetes Security Primitives
    ● Namespacing for isolation
    ● Comprehensive authn and authz: RBAC
    ● Make containers (pods) secure: PodSecurityPolicy
    ● Make network secure: NetworkPolicy

    View Slide

  12. Challenges involved in security
    ● Containers share host kernel (breakout?)
    ● Containers share network
    ● Containers share node resources: cpu/ram/disk (noisy-neighbour?)
    ● Tenants share Kubernetes itself!

    View Slide

  13. Tackling
    the
    challenges

    View Slide

  14. Containers sharing host kernel
    ● Do not run containers as root
    ● Limit sys calls (strong seccomp profile)
    ● Apparmour/Selinux (filesystem access)
    ● Linux Capabilities (CAP_SYS_ADMIN)
    ● Do not share host PID, host IPC, etc

    View Slide

  15. Containers sharing host kernel
    apiVersion: extensions/v1beta1
    kind: PodSecurityPolicy
    metadata:
    name: restricted
    annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default'
    apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'
    seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default'
    apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default'
    spec:
    privileged: false
    # Required to prevent escalations to root.
    ...

    View Slide

  16. PodSecurityPolicy
    spec:
    privileged: false
    # Required to prevent escalations to root.
    allowPrivilegeEscalation: false
    # This is redundant with non-root + disallow privilege escalation,
    # but we can provide it for defense in depth.
    requiredDropCapabilities:
    - ALL
    # Allow core volume types.
    volumes:
    - 'configMap'
    ...

    View Slide

  17. PodSecurityPolicy
    volumes:
    - 'configMap'
    - 'emptyDir'
    - 'projected'
    - 'secret'
    - 'downwardAPI'
    # Assume that persistentVolumes set up by the cluster admin are safe to use.
    - 'persistentVolumeClaim'
    hostNetwork: false
    hostIPC: false
    hostPID: false
    runAsUser:
    ...

    View Slide

  18. PodSecurityPolicy
    runAsUser:
    # Require the container to run without root privileges.
    rule: 'MustRunAsNonRoot'
    seLinux:
    # This policy assumes the nodes are using AppArmor rather than SELinux.
    rule: 'RunAsAny'
    supplementalGroups:
    rule: 'MustRunAs'
    ranges:
    # Forbid adding the root group.
    - min: 1
    max: 65535

    View Slide

  19. Containers sharing host kernel
    ● The next big thing: Nested kernels
    ● Bring Your Own Kernel (BYOK)
    ● Each container contains guest kernel which provides isolation from host kernel
    ● Introducing Kata containers

    View Slide

  20. Kata Containers
    source

    View Slide

  21. Containers sharing network
    source

    View Slide

  22. NetworkPolicy
    Enforced by Network Plugins
    ● Calico
    ● Romana
    ● Weave
    ● Cilium

    View Slide

  23. Containers sharing node resources
    ● Resource quotas and limits
    ● Uses linux cgroups
    ● cAdvisor
    apiVersion: v1
    kind: LimitRange
    metadata:
    name: mem-limit-range
    Spec:
    Limits:
    - default:
    memory: 512Mi
    cpu: 100m
    defaultRequest:
    memory: 256Mi
    type: Container

    View Slide

  24. Tenants sharing Kubernetes
    ● Tenants share control plane components like API server, scheduler etc
    ● Control plane isolation?
    ● Needs “hypervisor” for control plane
    ● Yet, unsolved

    View Slide

  25. Challenges involving operations
    ● Provisioning tenants
    ○ Build a tenant operator
    ● Identity management and access control (IAM)
    ○ Build IAM system outside Kubernetes
    ● Observability
    ○ RBAC enabled Heapster?
    None of this is available today!

    View Slide

  26. On that note:
    ● Kubernetes has many primitives for enabling multi tenancy today
    ● More things coming: Kata containers
    ● Many things missing: provisioning, IAM, observability
    ● Join Kubernetes Multitenancy Working Group if you’re interested

    View Slide

  27. Multi-tenancy
    at Hasura

    View Slide

  28. Danava
    ● ~50 nodes (auto-scaled)
    ● ~1000 projects
    ● ~2000 namespaces
    ● >10000 deployments
    ● >2000 active pods
    ● >2000 active services
    ● CRD+K8s Operator - state machine
    ● Monitoring - shutdown inactive
    projects

    View Slide

  29. Shahidh K
    Muhammed
    @shahidh_k
    Thanks for listening! Questions?

    View Slide

  30. References
    ● Hypernetes: Bringing Security and Multi-tenancy to Kubernetes
    ● Building on Kubernetes: Bringing Google-Scale Container Orchestration and Management to the
    Enterprise
    ● Hard Multi-Tenancy in Kubernetes
    ● Multitenancy Deep Dive
    ● Kubernetes Multitenancy Working Group

    View Slide