GPU Sharing done right

In this presentation, Scott McAllister and I explain how to share GPUs across multiple Kubernetes clusters the right way.

This version of the talk was given at HashiConf San Francisco, in September 2025.

Transcript

1. ©2025 HASHICORP GPU sharing done right Senior Developer Advocate, HashiCorp Kerim Satirli Principal Developer Advocate, vCluster Scott McAllister
2. Senior Developer Advocate, Infrastructure & Orchestration @ksatirli Kerim Satirli
3. Principal Developer Advocate vCluster @stmcallister Scott McAllister
4. 01 Where we're at
5. Traditional clusters are like gamers.
6. Traditional clusters don't like to share GPU.
7. AI use cases GPU demand Interest in AI What’s happening
8. Multi-Process Service (MPS) Process-1 Process-2 Process-3 GPU CUDA Context
9. GPU User User User User User Multi-Instance GPUs (MIG) GPU instance 0 GPU instance 1 GPU instance 2 GPU instance 3 GPU instance 4
10. Time-slicing (Or: custom scheduling) GPU Compute Process 1 Process 2 Process 3 Process 4 T2 T3 T4 T1 T2 T3 T4 T1 T1 T2 Time slice
11. Per-team GPU cluster Tenant namespaces GPU node GPU node KAI scheduler Tenant A Tenant namespaces GPU node GPU node KAI scheduler Tenant B
12. Per-team GPU cluster Tenant namespaces GPU node GPU node KAI scheduler Tenant A Tenant namespaces GPU node GPU node KAI scheduler Tenant B
13. Namespace-based multi-tenancy tenant-ns-a GPU node GPU node KAI scheduler tenant-ns-b GPU node GPU node Tenant A Tenant B
14. pod-1 custom-resource Team-shared cluster deployment pod-1 custom-resource Data store Control manager vcluster syncer API server Namespace in virtual cluster Virtual cluster context etcd Control manager Scheduler API server Control plane synced-pod-1 vcluster-pod Namespace in K8s K8s cluster context vcluster-pod Tenant Connects to cluster API server Controls virtual cluster Admin Connects to K8s API server Controls K8s context
15. Team-shared cluster etcd Control manager Scheduler API server Control plane pod-1 Namespace in team clusters Team-cluster context custom-resource
16. GPU sharing only works when you understand your trust boundaries.
17. but not isolation. Namespaces provide logical segmentation,
18. Per-team clusters address isolation, but not efficiency.
19. Multi-team clusters address efficiency, but not isolation.
20. 02 Where we need to be at
21. Multi-tenant cluster Tenant A Tenant B tenant-a-proj-1 tenant-b-proj-1 Kubernetes cluster Other Operations vcluster-a vcluster-b DGX node DGX node DGX node DGX node
22. Team-specific cluster etcd Control manager Scheduler API server Control plane pod-1 custom-resource Deployments
23. Team-specific cluster etcd Control manager Scheduler API server Control plane pod-1 Deployments custom-resource Vault
24. Team-specific security etcd Control manager Scheduler Secrets Team 2 Namespaces Team 3 Team 1 Control plane Deployments Vault pod-1
25. Team-shared cluster Control plane Deployments Vault pod-1 Access for 4h Control plane Deployments Vault pod-1 Access for S3 Team 1 Team 2
26. # ... listener "tcp" { address = "127.0.0.1:8300" chroot_namespace = "team-1" tls_cert_file = "/mnt/vault/chain.pem" tls_key_file = "/mnt/vault/private-key.pem" } # ... Configuration vault-config.hcl
27. # ... listener "tcp" { address = "127.0.0.1:8300" chroot_namespace = "team-1" tls_cert_file = "/mnt/vault/chain.pem" tls_key_file = "/mnt/vault/private-key.pem" } # ... Configuration vault-config.hcl
28. 03 What we need to remember
29. Architect for failure.
30. Isolation is key.
31. Defense in-depth is defense that works.
32. Virtual clusters limit blast-radius.
33. Virtual clusters enable easier auditing.
34. GPU Fairness is an evolving problem.
35. Thank you speakerdeck.com/ksatirli
36. Eliminate forever credentials