From 5 to 1,300+ Clusters: Declarative Scaling for Private Kubernetes

Transcript

1. #KubeCon #CloudNativeCon From 5 to 1,300+ Clusters: Declarative Scaling for Private Kubernetes Private / On-Prem / OpenStack Shota Yoshimura Senior Platform Engineer, LY Corporation
2. Where We Started — 2016: VMs on OpenStack Our Challenges • Provisioning environments often took more than a week • Nodes stayed unpatched for long periods — a security risk • Node failures meant on-call pages — often at 3 a.m. So we set out to build declarative infrastructure.
3. Architecture Kubernetes as a Service API Req Custom Resource apiVersion: zlab.co.jp/v1 kind: KubernetesCluster metadata: - name: foo # Control Plane controlPlaneMachineGroups: - name: cp flavor: 4v-8G-64G replicas: 2 # Worker workerMachineGroups: - name: worker flavor: 4v-16G-64G replicas: 3 Apply KubernetesCluster Watch name: foo KubernetesCluster Controller Create Create MachineDeployment Watch name: foo-<group> Create MachineDeployment Controller Control Plane Nodes VM MachineSet Watch name: foo-<group>-<hash> # Ingress - name: ingress flavor: 4v-8G-64G replicas: 2 MachineSet Controller VM Worker Nodes Create VM Machine name:foo-<group>-<hash>-<rand> Watch Machine Controller VM VM Ingress Nodes VM Inspired by Kubernetes' own Deployment — applied to VMs. VM
4. Ten Years Later — 2026 Scale grew without linear team growth. 1,300+ clusters 40,000+ nodes ~1M containers 700+ clusters Operated by 15 engineers 400+ clusters 5 clusters 2017 20 clusters 2018 2019 2020 …. 2026 Today, one platform engineer operates 90 clusters.
5. Want the deep dive? Wednesday at 12:10 PM · 30 min · 5F | 501 Scan for breakout session