Multi-tenancy in Kubernetes: Sharing a Cluster with Hundreds of Users

Security primitives provided by Kubernetes in a multi-tenant context and Hasura's experiences of running a multi-tenant k8s cluster in production.

Presented at Infrastructure Engineering @Scale Meetup at LinkedIn Office, Bangalore.

Transcript

1. Multi-tenancy in Kubernetes: Sharing a Cluster with Hundreds of Users
2. Shahidh K Muhammed @shahidh_k
3. Outline • What is multi-tenancy? • Why multi-tenancy? • Challenges involved in multi-tenancy • Security primitives in Kubernetes • Managing shared resources in Kubernetes • Simplifying operations on Kubernetes - Operator
4. What is Multi-tenancy? “Mode of operation of software where multiple independent instances of one or more applications operate in a shared environment” -- Gartner
5. What is Multi-tenancy source
6. Types of Multi-tenancy Soft Multi-tenancy • Non adversarial tenants • Different department/teams in the same company • Not trying to harm other tenants • Focus on preventing accidents Hard Multi-tenancy • Adversarial tenants • Different kinds of users who has no relation to each other • Trying to exploit the system • Focus on securing and isolating each tenant
7. Why multi-tenancy in Kubernetes?
8. Resource sharing • Containers from multiple-tenants can be fit into one node. • Tight packing of containers helps save resources especially for bursty workloads • Cost savings • Operational burden?
9. One cluster for all But… • Need to manage each tenant • Different type of complexity • Setup and maintain a single cluster • Homogenous environment for all tenants
10. Multi-tenant Security in Kubernetes
11. Kubernetes Security Primitives • Namespacing for isolation • Comprehensive authn and authz: RBAC • Make containers (pods) secure: PodSecurityPolicy • Make network secure: NetworkPolicy
12. Challenges involved in security • Containers share host kernel (breakout?) • Containers share network • Containers share node resources: cpu/ram/disk (noisy-neighbour?) • Tenants share Kubernetes itself!
13. Tackling the challenges
14. Containers sharing host kernel • Do not run containers as root • Limit sys calls (strong seccomp profile) • Apparmour/Selinux (filesystem access) • Linux Capabilities (CAP_SYS_ADMIN) • Do not share host PID, host IPC, etc
15. Containers sharing host kernel apiVersion: extensions/v1beta1 kind: PodSecurityPolicy metadata: name: restricted annotations: seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default' apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default' seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default' apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default' spec: privileged: false # Required to prevent escalations to root. ...
16. PodSecurityPolicy spec: privileged: false # Required to prevent escalations to root. allowPrivilegeEscalation: false # This is redundant with non-root + disallow privilege escalation, # but we can provide it for defense in depth. requiredDropCapabilities: - ALL # Allow core volume types. volumes: - 'configMap' ...
17. PodSecurityPolicy volumes: - 'configMap' - 'emptyDir' - 'projected' - 'secret' - 'downwardAPI' # Assume that persistentVolumes set up by the cluster admin are safe to use. - 'persistentVolumeClaim' hostNetwork: false hostIPC: false hostPID: false runAsUser: ...
18. PodSecurityPolicy runAsUser: # Require the container to run without root privileges. rule: 'MustRunAsNonRoot' seLinux: # This policy assumes the nodes are using AppArmor rather than SELinux. rule: 'RunAsAny' supplementalGroups: rule: 'MustRunAs' ranges: # Forbid adding the root group. - min: 1 max: 65535
19. Containers sharing host kernel • The next big thing: Nested kernels • Bring Your Own Kernel (BYOK) • Each container contains guest kernel which provides isolation from host kernel • Introducing Kata containers
20. Kata Containers source
21. Containers sharing network source
22. NetworkPolicy Enforced by Network Plugins • Calico • Romana • Weave • Cilium
23. Containers sharing node resources • Resource quotas and limits • Uses linux cgroups • cAdvisor apiVersion: v1 kind: LimitRange metadata: name: mem-limit-range Spec: Limits: - default: memory: 512Mi cpu: 100m defaultRequest: memory: 256Mi type: Container
24. Tenants sharing Kubernetes • Tenants share control plane components like API server, scheduler etc • Control plane isolation? • Needs “hypervisor” for control plane • Yet, unsolved
25. Challenges involving operations • Provisioning tenants ◦ Build a tenant operator • Identity management and access control (IAM) ◦ Build IAM system outside Kubernetes • Observability ◦ RBAC enabled Heapster? None of this is available today!
26. On that note: • Kubernetes has many primitives for enabling multi tenancy today • More things coming: Kata containers • Many things missing: provisioning, IAM, observability • Join Kubernetes Multitenancy Working Group if you’re interested
27. Multi-tenancy at Hasura
28. Danava • ~50 nodes (auto-scaled) • ~1000 projects • ~2000 namespaces • >10000 deployments • >2000 active pods • >2000 active services • CRD+K8s Operator - state machine • Monitoring - shutdown inactive projects
29. Shahidh K Muhammed @shahidh_k Thanks for listening! Questions?
30. References • Hypernetes: Bringing Security and Multi-tenancy to Kubernetes • Building on Kubernetes: Bringing Google-Scale Container Orchestration and Management to the Enterprise • Hard Multi-Tenancy in Kubernetes • Multitenancy Deep Dive • Kubernetes Multitenancy Working Group