Multi-tenancy in Kubernetes: Sharing a Cluster with Hundreds of Users

Multi-tenancy in Kubernetes: Sharing a Cluster with Hundreds of Users

Security primitives provided by Kubernetes in a multi-tenant context and Hasura's experiences of running a multi-tenant k8s cluster in production.

Presented at Infrastructure Engineering @Scale Meetup at LinkedIn Office, Bangalore.

0913331bcf360849ca2acec0612081b2?s=128

Shahidh K Muhammed

May 26, 2018
Tweet

Transcript

  1. Multi-tenancy in Kubernetes: Sharing a Cluster with Hundreds of Users

  2. Shahidh K Muhammed @shahidh_k

  3. Outline • What is multi-tenancy? • Why multi-tenancy? • Challenges

    involved in multi-tenancy • Security primitives in Kubernetes • Managing shared resources in Kubernetes • Simplifying operations on Kubernetes - Operator
  4. What is Multi-tenancy? “Mode of operation of software where multiple

    independent instances of one or more applications operate in a shared environment” -- Gartner
  5. What is Multi-tenancy source

  6. Types of Multi-tenancy Soft Multi-tenancy • Non adversarial tenants •

    Different department/teams in the same company • Not trying to harm other tenants • Focus on preventing accidents Hard Multi-tenancy • Adversarial tenants • Different kinds of users who has no relation to each other • Trying to exploit the system • Focus on securing and isolating each tenant
  7. Why multi-tenancy in Kubernetes?

  8. Resource sharing • Containers from multiple-tenants can be fit into

    one node. • Tight packing of containers helps save resources especially for bursty workloads • Cost savings • Operational burden?
  9. One cluster for all But… • Need to manage each

    tenant • Different type of complexity • Setup and maintain a single cluster • Homogenous environment for all tenants
  10. Multi-tenant Security in Kubernetes

  11. Kubernetes Security Primitives • Namespacing for isolation • Comprehensive authn

    and authz: RBAC • Make containers (pods) secure: PodSecurityPolicy • Make network secure: NetworkPolicy
  12. Challenges involved in security • Containers share host kernel (breakout?)

    • Containers share network • Containers share node resources: cpu/ram/disk (noisy-neighbour?) • Tenants share Kubernetes itself!
  13. Tackling the challenges

  14. Containers sharing host kernel • Do not run containers as

    root • Limit sys calls (strong seccomp profile) • Apparmour/Selinux (filesystem access) • Linux Capabilities (CAP_SYS_ADMIN) • Do not share host PID, host IPC, etc
  15. Containers sharing host kernel apiVersion: extensions/v1beta1 kind: PodSecurityPolicy metadata: name:

    restricted annotations: seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default' apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default' seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default' apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default' spec: privileged: false # Required to prevent escalations to root. ...
  16. PodSecurityPolicy spec: privileged: false # Required to prevent escalations to

    root. allowPrivilegeEscalation: false # This is redundant with non-root + disallow privilege escalation, # but we can provide it for defense in depth. requiredDropCapabilities: - ALL # Allow core volume types. volumes: - 'configMap' ...
  17. PodSecurityPolicy volumes: - 'configMap' - 'emptyDir' - 'projected' - 'secret'

    - 'downwardAPI' # Assume that persistentVolumes set up by the cluster admin are safe to use. - 'persistentVolumeClaim' hostNetwork: false hostIPC: false hostPID: false runAsUser: ...
  18. PodSecurityPolicy runAsUser: # Require the container to run without root

    privileges. rule: 'MustRunAsNonRoot' seLinux: # This policy assumes the nodes are using AppArmor rather than SELinux. rule: 'RunAsAny' supplementalGroups: rule: 'MustRunAs' ranges: # Forbid adding the root group. - min: 1 max: 65535
  19. Containers sharing host kernel • The next big thing: Nested

    kernels • Bring Your Own Kernel (BYOK) • Each container contains guest kernel which provides isolation from host kernel • Introducing Kata containers
  20. Kata Containers source

  21. Containers sharing network source

  22. NetworkPolicy Enforced by Network Plugins • Calico • Romana •

    Weave • Cilium
  23. Containers sharing node resources • Resource quotas and limits •

    Uses linux cgroups • cAdvisor apiVersion: v1 kind: LimitRange metadata: name: mem-limit-range Spec: Limits: - default: memory: 512Mi cpu: 100m defaultRequest: memory: 256Mi type: Container
  24. Tenants sharing Kubernetes • Tenants share control plane components like

    API server, scheduler etc • Control plane isolation? • Needs “hypervisor” for control plane • Yet, unsolved
  25. Challenges involving operations • Provisioning tenants ◦ Build a tenant

    operator • Identity management and access control (IAM) ◦ Build IAM system outside Kubernetes • Observability ◦ RBAC enabled Heapster? None of this is available today!
  26. On that note: • Kubernetes has many primitives for enabling

    multi tenancy today • More things coming: Kata containers • Many things missing: provisioning, IAM, observability • Join Kubernetes Multitenancy Working Group if you’re interested
  27. Multi-tenancy at Hasura

  28. Danava • ~50 nodes (auto-scaled) • ~1000 projects • ~2000

    namespaces • >10000 deployments • >2000 active pods • >2000 active services • CRD+K8s Operator - state machine • Monitoring - shutdown inactive projects
  29. Shahidh K Muhammed @shahidh_k Thanks for listening! Questions?

  30. References • Hypernetes: Bringing Security and Multi-tenancy to Kubernetes •

    Building on Kubernetes: Bringing Google-Scale Container Orchestration and Management to the Enterprise • Hard Multi-Tenancy in Kubernetes • Multitenancy Deep Dive • Kubernetes Multitenancy Working Group