Multi-tenancy in Kubernetes: Sharing a Cluster with Hundreds of Users

Security primitives provided by Kubernetes in a multi-tenant context and Hasura's experiences of running a multi-tenant k8s cluster in production.

Presented at Infrastructure Engineering @Scale Meetup at LinkedIn Office, Bangalore.

Transcript

1. Multi-tenancy in Kubernetes:
   Sharing a Cluster with
   Hundreds of Users
   

   View Slide

2. Shahidh K
   Muhammed
   @shahidh_k
   

   View Slide

3. Outline
   ● What is multi-tenancy?
   ● Why multi-tenancy?
   ● Challenges involved in multi-tenancy
   ● Security primitives in Kubernetes
   ● Managing shared resources in Kubernetes
   ● Simplifying operations on Kubernetes - Operator
   

   View Slide

4. What is
   Multi-tenancy?
   “Mode of operation of software where
   multiple independent instances of one or
   more applications operate in a shared
   environment” -- Gartner
   

   View Slide

5. What is Multi-tenancy
   source
   

   View Slide

6. Types of Multi-tenancy
   Soft Multi-tenancy
   ● Non adversarial tenants
   ● Different department/teams in the same
   company
   ● Not trying to harm other tenants
   ● Focus on preventing accidents
   Hard Multi-tenancy
   ● Adversarial tenants
   ● Different kinds of users who has no relation
   to each other
   ● Trying to exploit the system
   ● Focus on securing and isolating each tenant
   

   View Slide

7. Why multi-tenancy in
   Kubernetes?
   

   View Slide

8. Resource sharing
   ● Containers from multiple-tenants can be fit into one node.
   ● Tight packing of containers helps save resources especially for bursty workloads
   ● Cost savings
   ● Operational burden?
   

   View Slide

9. One cluster for all But…
   ● Need to manage each tenant
   ● Different type of complexity
   ● Setup and maintain a single
   cluster
   ● Homogenous environment for
   all tenants
   

   View Slide

10. Multi-tenant Security in
    Kubernetes
    

    View Slide

11. Kubernetes Security Primitives
    ● Namespacing for isolation
    ● Comprehensive authn and authz: RBAC
    ● Make containers (pods) secure: PodSecurityPolicy
    ● Make network secure: NetworkPolicy
    

    View Slide

12. Challenges involved in security
    ● Containers share host kernel (breakout?)
    ● Containers share network
    ● Containers share node resources: cpu/ram/disk (noisy-neighbour?)
    ● Tenants share Kubernetes itself!
    

    View Slide

13. Tackling
    the
    challenges
    

    View Slide

14. Containers sharing host kernel
    ● Do not run containers as root
    ● Limit sys calls (strong seccomp profile)
    ● Apparmour/Selinux (filesystem access)
    ● Linux Capabilities (CAP_SYS_ADMIN)
    ● Do not share host PID, host IPC, etc
    

    View Slide

15. Containers sharing host kernel
    apiVersion: extensions/v1beta1
    kind: PodSecurityPolicy
    metadata:
    name: restricted
    annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default'
    apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'
    seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default'
    apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default'
    spec:
    privileged: false
    # Required to prevent escalations to root.
    ...
    

    View Slide

16. PodSecurityPolicy
    spec:
    privileged: false
    # Required to prevent escalations to root.
    allowPrivilegeEscalation: false
    # This is redundant with non-root + disallow privilege escalation,
    # but we can provide it for defense in depth.
    requiredDropCapabilities:
    - ALL
    # Allow core volume types.
    volumes:
    - 'configMap'
    ...
    

    View Slide

17. PodSecurityPolicy
    volumes:
    - 'configMap'
    - 'emptyDir'
    - 'projected'
    - 'secret'
    - 'downwardAPI'
    # Assume that persistentVolumes set up by the cluster admin are safe to use.
    - 'persistentVolumeClaim'
    hostNetwork: false
    hostIPC: false
    hostPID: false
    runAsUser:
    ...
    

    View Slide

18. PodSecurityPolicy
    runAsUser:
    # Require the container to run without root privileges.
    rule: 'MustRunAsNonRoot'
    seLinux:
    # This policy assumes the nodes are using AppArmor rather than SELinux.
    rule: 'RunAsAny'
    supplementalGroups:
    rule: 'MustRunAs'
    ranges:
    # Forbid adding the root group.
    - min: 1
    max: 65535
    

    View Slide

19. Containers sharing host kernel
    ● The next big thing: Nested kernels
    ● Bring Your Own Kernel (BYOK)
    ● Each container contains guest kernel which provides isolation from host kernel
    ● Introducing Kata containers
    

    View Slide

20. Kata Containers
    source
    

    View Slide

21. Containers sharing network
    source
    

    View Slide

22. NetworkPolicy
    Enforced by Network Plugins
    ● Calico
    ● Romana
    ● Weave
    ● Cilium
    

    View Slide

23. Containers sharing node resources
    ● Resource quotas and limits
    ● Uses linux cgroups
    ● cAdvisor
    apiVersion: v1
    kind: LimitRange
    metadata:
    name: mem-limit-range
    Spec:
    Limits:
    - default:
    memory: 512Mi
    cpu: 100m
    defaultRequest:
    memory: 256Mi
    type: Container
    

    View Slide

24. Tenants sharing Kubernetes
    ● Tenants share control plane components like API server, scheduler etc
    ● Control plane isolation?
    ● Needs “hypervisor” for control plane
    ● Yet, unsolved
    

    View Slide

25. Challenges involving operations
    ● Provisioning tenants
    ○ Build a tenant operator
    ● Identity management and access control (IAM)
    ○ Build IAM system outside Kubernetes
    ● Observability
    ○ RBAC enabled Heapster?
    None of this is available today!
    

    View Slide

26. On that note:
    ● Kubernetes has many primitives for enabling multi tenancy today
    ● More things coming: Kata containers
    ● Many things missing: provisioning, IAM, observability
    ● Join Kubernetes Multitenancy Working Group if you’re interested
    

    View Slide

27. Multi-tenancy
    at Hasura
    

    View Slide

28. Danava
    ● ~50 nodes (auto-scaled)
    ● ~1000 projects
    ● ~2000 namespaces
    ● >10000 deployments
    ● >2000 active pods
    ● >2000 active services
    ● CRD+K8s Operator - state machine
    ● Monitoring - shutdown inactive
    projects
    

    View Slide

29. Shahidh K
    Muhammed
    @shahidh_k
    Thanks for listening! Questions?
    

    View Slide

30. References
    ● Hypernetes: Bringing Security and Multi-tenancy to Kubernetes
    ● Building on Kubernetes: Bringing Google-Scale Container Orchestration and Management to the
    Enterprise
    ● Hard Multi-Tenancy in Kubernetes
    ● Multitenancy Deep Dive
    ● Kubernetes Multitenancy Working Group
    

    View Slide