What is Multi-tenancy? “Mode of operation of software where multiple independent instances of one or more applications operate in a shared environment” -- Gartner
Types of Multi-tenancy Soft Multi-tenancy ● Non adversarial tenants ● Different department/teams in the same company ● Not trying to harm other tenants ● Focus on preventing accidents Hard Multi-tenancy ● Adversarial tenants ● Different kinds of users who has no relation to each other ● Trying to exploit the system ● Focus on securing and isolating each tenant
Resource sharing ● Containers from multiple-tenants can be fit into one node. ● Tight packing of containers helps save resources especially for bursty workloads ● Cost savings ● Operational burden?
One cluster for all But… ● Need to manage each tenant ● Different type of complexity ● Setup and maintain a single cluster ● Homogenous environment for all tenants
Containers sharing host kernel ● Do not run containers as root ● Limit sys calls (strong seccomp profile) ● Apparmour/Selinux (filesystem access) ● Linux Capabilities (CAP_SYS_ADMIN) ● Do not share host PID, host IPC, etc
PodSecurityPolicy spec: privileged: false # Required to prevent escalations to root. allowPrivilegeEscalation: false # This is redundant with non-root + disallow privilege escalation, # but we can provide it for defense in depth. requiredDropCapabilities: - ALL # Allow core volume types. volumes: - 'configMap' ...
PodSecurityPolicy runAsUser: # Require the container to run without root privileges. rule: 'MustRunAsNonRoot' seLinux: # This policy assumes the nodes are using AppArmor rather than SELinux. rule: 'RunAsAny' supplementalGroups: rule: 'MustRunAs' ranges: # Forbid adding the root group. - min: 1 max: 65535
Containers sharing host kernel ● The next big thing: Nested kernels ● Bring Your Own Kernel (BYOK) ● Each container contains guest kernel which provides isolation from host kernel ● Introducing Kata containers
Tenants sharing Kubernetes ● Tenants share control plane components like API server, scheduler etc ● Control plane isolation? ● Needs “hypervisor” for control plane ● Yet, unsolved
Challenges involving operations ● Provisioning tenants ○ Build a tenant operator ● Identity management and access control (IAM) ○ Build IAM system outside Kubernetes ● Observability ○ RBAC enabled Heapster? None of this is available today!
On that note: ● Kubernetes has many primitives for enabling multi tenancy today ● More things coming: Kata containers ● Many things missing: provisioning, IAM, observability ● Join Kubernetes Multitenancy Working Group if you’re interested
References ● Hypernetes: Bringing Security and Multi-tenancy to Kubernetes ● Building on Kubernetes: Bringing Google-Scale Container Orchestration and Management to the Enterprise ● Hard Multi-Tenancy in Kubernetes ● Multitenancy Deep Dive ● Kubernetes Multitenancy Working Group