Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Multi-tenancy in Kubernetes: Sharing a Cluster with Hundreds of Users

Multi-tenancy in Kubernetes: Sharing a Cluster with Hundreds of Users

Security primitives provided by Kubernetes in a multi-tenant context and Hasura's experiences of running a multi-tenant k8s cluster in production.

Presented at Infrastructure Engineering @Scale Meetup at LinkedIn Office, Bangalore.

Shahidh K Muhammed

May 26, 2018

More Decks by Shahidh K Muhammed

Other Decks in Technology


  1. Outline • What is multi-tenancy? • Why multi-tenancy? • Challenges

    involved in multi-tenancy • Security primitives in Kubernetes • Managing shared resources in Kubernetes • Simplifying operations on Kubernetes - Operator
  2. What is Multi-tenancy? “Mode of operation of software where multiple

    independent instances of one or more applications operate in a shared environment” -- Gartner
  3. Types of Multi-tenancy Soft Multi-tenancy • Non adversarial tenants •

    Different department/teams in the same company • Not trying to harm other tenants • Focus on preventing accidents Hard Multi-tenancy • Adversarial tenants • Different kinds of users who has no relation to each other • Trying to exploit the system • Focus on securing and isolating each tenant
  4. Resource sharing • Containers from multiple-tenants can be fit into

    one node. • Tight packing of containers helps save resources especially for bursty workloads • Cost savings • Operational burden?
  5. One cluster for all But… • Need to manage each

    tenant • Different type of complexity • Setup and maintain a single cluster • Homogenous environment for all tenants
  6. Kubernetes Security Primitives • Namespacing for isolation • Comprehensive authn

    and authz: RBAC • Make containers (pods) secure: PodSecurityPolicy • Make network secure: NetworkPolicy
  7. Challenges involved in security • Containers share host kernel (breakout?)

    • Containers share network • Containers share node resources: cpu/ram/disk (noisy-neighbour?) • Tenants share Kubernetes itself!
  8. Containers sharing host kernel • Do not run containers as

    root • Limit sys calls (strong seccomp profile) • Apparmour/Selinux (filesystem access) • Linux Capabilities (CAP_SYS_ADMIN) • Do not share host PID, host IPC, etc
  9. Containers sharing host kernel apiVersion: extensions/v1beta1 kind: PodSecurityPolicy metadata: name:

    restricted annotations: seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default' apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default' seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default' apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default' spec: privileged: false # Required to prevent escalations to root. ...
  10. PodSecurityPolicy spec: privileged: false # Required to prevent escalations to

    root. allowPrivilegeEscalation: false # This is redundant with non-root + disallow privilege escalation, # but we can provide it for defense in depth. requiredDropCapabilities: - ALL # Allow core volume types. volumes: - 'configMap' ...
  11. PodSecurityPolicy volumes: - 'configMap' - 'emptyDir' - 'projected' - 'secret'

    - 'downwardAPI' # Assume that persistentVolumes set up by the cluster admin are safe to use. - 'persistentVolumeClaim' hostNetwork: false hostIPC: false hostPID: false runAsUser: ...
  12. PodSecurityPolicy runAsUser: # Require the container to run without root

    privileges. rule: 'MustRunAsNonRoot' seLinux: # This policy assumes the nodes are using AppArmor rather than SELinux. rule: 'RunAsAny' supplementalGroups: rule: 'MustRunAs' ranges: # Forbid adding the root group. - min: 1 max: 65535
  13. Containers sharing host kernel • The next big thing: Nested

    kernels • Bring Your Own Kernel (BYOK) • Each container contains guest kernel which provides isolation from host kernel • Introducing Kata containers
  14. Containers sharing node resources • Resource quotas and limits •

    Uses linux cgroups • cAdvisor apiVersion: v1 kind: LimitRange metadata: name: mem-limit-range Spec: Limits: - default: memory: 512Mi cpu: 100m defaultRequest: memory: 256Mi type: Container
  15. Tenants sharing Kubernetes • Tenants share control plane components like

    API server, scheduler etc • Control plane isolation? • Needs “hypervisor” for control plane • Yet, unsolved
  16. Challenges involving operations • Provisioning tenants ◦ Build a tenant

    operator • Identity management and access control (IAM) ◦ Build IAM system outside Kubernetes • Observability ◦ RBAC enabled Heapster? None of this is available today!
  17. On that note: • Kubernetes has many primitives for enabling

    multi tenancy today • More things coming: Kata containers • Many things missing: provisioning, IAM, observability • Join Kubernetes Multitenancy Working Group if you’re interested
  18. Danava • ~50 nodes (auto-scaled) • ~1000 projects • ~2000

    namespaces • >10000 deployments • >2000 active pods • >2000 active services • CRD+K8s Operator - state machine • Monitoring - shutdown inactive projects
  19. References • Hypernetes: Bringing Security and Multi-tenancy to Kubernetes •

    Building on Kubernetes: Bringing Google-Scale Container Orchestration and Management to the Enterprise • Hard Multi-Tenancy in Kubernetes • Multitenancy Deep Dive • Kubernetes Multitenancy Working Group