Kraftlet: Kubernetes at 1M Pod Scale and Millisecond Speeds

Transcript

1. unikraft.cloud Kraftlet: Kubernetes at 1M Pod scale Bringing microVM isolation and speed to the cluster you already run Petar Cvitanovi Schedule a demo support@unikraft.cloud
2. 02 A story about our Kubernetes integration… unikraft.cloud Scale without limits
3. Where we started 03 What is Unikraft? Fast Standby Isolated Boots microVMs in under 10 ms Idle instances go into standby and wake up on the next request Lightweight VMs with strong isolation unikraft.cloud Scale without limits
4. 04 Our clients love it and run it in production And many of them asked us the same thing: can we use it from Kubernetes? unikraft.cloud Scale without limits
5. Denisity: Kubernetes node vs Unikraft box A Kubernetes node ~110 06 A Unikraft box VS Pods per node, the upstream-tested limit 10k+ microVMs on a single box So we set out to lift the limits on individual nodes. unikraft.cloud Scale without limits
6. How Unikraft runs 05 One box, one HTTP API A single Unikraft box HTTP API Client CLI, SDK or any HTTP client Instances Services Volumes microVMs, booted from an image Expose and loadbalance instances Persistent storage for instances Install it on any machine: one runtime plus an API that creates and manages the microVMs on it. unikraft.cloud Scale without limits
7. Idea 1 · Operator 07 An operator maps one-to-one to Unikraft Simple to run and maintain 1-to-1 mapping to Unikraft API A CRD per resource Instance, Service and Volume, each reconciled into Unikraft API calls. Every platform knob Replicas, memory, scale-to-zero, service groups and volumes, right in the spec. Still maintained on GitHub for simple management without scheduling. unikraft.cloud apiVersion: unikraft.com/v1alpha1 kind: Instance metadata: name: my-instance spec: image: pcvitanovic/my-app:latest memory_mb: 256 replicas: 2 scale_to_zero: policy: "on" stateful: true cooldown_time_ms: 1000 service_group: name: my-service volumes: name: my-volume at: /data Scale without limits
8. Idea 1 · Operator 08 One root problem: it doesn't run Pods An Instance is not a Pod The operator manages its own CRDs, so nothing in Kubernetes that is built around Pods applies to it. No ecosystem No scheduler Strimzi, CNPG and other operators create Pods. Those Pods can never land on Unikraft. Kubernetes schedules Pods, not Instances. Spreading them across boxes means writing our own. unikraft.cloud Scale without limits
9. 09 Idea 2: Start looking at Unikraft as a runtime unikraft.cloud Scale without limits
10. Idea 2 · CRI 10 Why a CRI is the right abstraction Pod Scheduler kubelet CRI Unikraft runtime Pods, natively The ecosystem works Kubernetes schedules Instances are configured through the Pod object. No new CRDs to learn. Strimzi, CNPG, Deployments, Services: anything that creates or selects Pods runs on Unikraft. The scheduler places Pods on nodes, so we don't need a scheduler of our own. It fixes both operator problems at the root: Unikraft runs Pods. unikraft.cloud Scale without limits
11. Idea 2 · CRI 11 Root problem: the kubelet is in charge The runtime sits below the kubelet A CRI only plugs in underneath, so the kubelet's design and limits become ours. One kubelet per box Built for ~110 Pods Less Pod lifecycle flexibility The kubelet has to run where the runtime runs, so a whole box becomes one node. It lists every container each second and re-syncs every Pod about once a minute. A runtime can only return errors. It can't fail or delete a Pod, or read cluster state. unikraft.cloud Scale without limits
12. Idea 3 · Kraftlet 12 Kraftlet owns the Pod lifecycle Kubernetes cluster (EKS, GKE, AKS…) Unikraft Box Control plane Instance worker-node-1 etcd Kraftlet Kubelet Instance Service K8s API container runtime Volume container Scheduler container Volume Built on Virtual Kubelet Runs anywhere The CNCF project for nodes backed by an API instead of a machine. Kraftlet is its Unikraft provider. A node like any other, but it talks to the platform API, so it needn't run on the box. unikraft.cloud Scale without limits
13. Idea 3 · Kraftlet 13 Scale out: many Kraftlets, one box Kubernetes cluster (EKS, GKE, AKS…) Unikraft Box Control plane Instance Kraftlet-0 worker-node-1 etcd Instance Kubelet Kraftlet-1 Service K8s API container runtime Volume container Scheduler Kraftlet-n container Volume Each Kraftlet is a node Add Kraftlets to scale Small blast radius It registers with the API server like any node and owns its share of the Pods. Density is capped by the box, not by one process managing it. If one Kraftlet fails, only its Pods are affected. unikraft.cloud Scale without limits
14. Storage · CSI 14 Volumes: Kraftlet drives CSI remotely External APIs (e.g. AWS EC2 API) Kubernetes cluster (EKS, GKE, AKS…) 2. call external APIs Control plane CSI Controller 6. create Instance with attached volume etcd 1. watch for PVCs Instance K8s API 5. create volume Managed Volume 3. watch for Pods Scheduler Unikraft Cloud Kraftlet CSI Node (AWS EBS) 4. NodeStageVolume unikraft.cloud Scale without limits
15. Networking · CNI 15 Networking: CNI through a remote-cni Kubernetes cluster (EKS, GKE, AKS…) Unikraft Cloud 1. watch for Pods libcni Control plane Worker node 2. CNI ADD etcd Kubelet remote-cni 3. invoke plugins Kraftlet K8s API CNI plugins 4. create Instance with custom network interface container runtime Unikraft platform container Scheduler 5. create container Instance unikraft.cloud Scale without limits
16. About the title 16 What 1M Pod scale means Stock Kubernetes 1M Pods ÷ ~110 per node = ~10k nodes With Kraftlet 1M Pods ÷ 10k per node = 100 nodes The nodes stop being the limit At 1M Pods, what caps the cluster is the API server and etcd holding a million Pod objects. unikraft.cloud Scale without limits
17. Try it yourself 17 Run on any Kubernetes distro 1 Install the chart Point it at your Unikraft metro and API token. 2 Kraftlet joins as a node It registers with the API server and shows up as Ready. 3 Schedule Pods onto it Add the Kraftlet toleration and a nodeSelector. Each Pod runs as a microVM. $ helm install kraftlet \ --namespace kraftlet --create-namespace \ --set ukc.metro=$UKC_METRO \ --set ukc.token=$UKC_TOKEN \ oci://ghcr.io/unikraft-cloud/helm-charts/kraftlet $ kubectl get nodes NAME STATUS ROLES kraftlet Ready agent Full guide: unikraft.com/docs/integrations/kubernetes EKS GKE AKS On-prem Any distro unikraft.cloud Scale without limits
18. Your tooling, cluster and workflows. Just faster and more secure. unikraft.com/docs/integrations/kubernetes console.unikraft.cloud/signup
19. Backup · Cold start 19 From Pod to running microVM in ~70 ms 3.6 ms 68 ms 220 ms microVM boot, p50 cold start, one Pod at a time, p50 cold start, burst of 80 Pods, p50 Where the time goes (p50) 1 Pod 12.5 10 Pods ~16 50 Pods 80 Pods Kubernetes 44.5 Kraftlet 79.1 ms 22.1 ~45 84.4 ms ~23 ~67 Unikraft platform ~43 136.8 ms ~27 122.1 41.7 27.4 191.2 ms From 1 to 80 Pods, Kubernetes grows 9.8×. Kraftlet stays flat (0.9×) and the platform barely moves (1.2×). unikraft.cloud Scale without limits