qaware.de Mario-Leander Reimer [email protected] © 2004 Efficient Kubernetes Fleet Management with Cluster API and GitOps

2 Mario-Leander Reimer Managing Director | CTO @LeanderReimer #cloudnativenerd #qaware #gernperDude

10:04 PM - 27. November 2017 5:38 PM - 24. Februar 2019

Cloud-native Application Engineering Cloud-native Platform Engineering Responsibility Model of Cloud-native Software Engineering QAware | 4 IaaS Network, Compute, Storage (Network, Compute, Storage, Integration, ...) CaaS (Managed Kubernetes Services) PaaS (General Platform Components, CI/CD Toolchain, Catalogs & Portals, Application Blueprints) Application-specific Software Infrastructure Cloud-friendly & cloud-native Applications Architect Build Run

Platform engineering is the discipline of designing and building toolchains and workflows that enable self-service capabilities for software engineering organizations in the cloud-native era. Platform engineers provide an integrated product most often referred to as an “Internal Developer Platform” covering the operational necessities of the entire lifecycle of an application. https://platformengineering.org/blog/what-is-platform-engineering

An IDP and your platform engineers are key enablers for high productivity of the stream-aligned DevOps teams. QAware | 6 ■ Responsible to build and operation a platform to enable and support the teams in their day to day development work. ■ The platform aims to hide the inherent complexity to reduce the cognitive load for the other teams. – Standardization (Compliance, Security, …) – Developer Self-Service ■ Fully automated software delivery is the goal! https://hennyportman.wordpress.com/2020/05/25/review-team-topologies/

All the layers and components of your internal developer platform architecture need to be managed. Via GitOps. QAware | 7 7 QAware Developer Control Plane Integration and Delivery Plane Monitoring and Logging Plane Security Plane IDE Service Catalog / API Catalog Developer Portal Application Source Code Infrastructure & Platform Source Code Observability Secrets & Identity Manager CI Pipeline Registry CD Pipeline Resource Plane Compute Data Integration Networking Platform Orchestrator Certificates & Encryption GitOps https://humanitec.com/reference-architectures

How many platform instances? How many teams need to be supported? How to address multi-tenancy & isolation? How many stages are required? How to support ephemeral environments?

Option A: One Central Multi-Tenant Platform Limited Scalability e.g. Prometheus, Opensearch, GitOps Soft Isolation e.g. Docker, Namespaces Challenging Multi-Tenancy e.g. RBAC, Monitoring Stack Complex Coordination e.g. K8s Deprecations, CRDs Single Point of Failure e.g. API Gateway Route 9 QAware

Option B: One Platform per Tenant 10 Cluster Sprawl e.g. massive day-2 maintenance efforts High OPEX e.g. a lot of (underutilized) infrastructure Provider Limits e.g. external IPs, load balancer, VMs, … Complex Automation e.g. a lot of custom code and pipelines required Expert Team e.g. big professional platform team required QAware

How high is the extraneous cognitive load? Can GitOps be used to manage the platform? Are there k8s-native options & approaches? Which tooling to use?

+ +

lreimer/ k8s-fleet-capi-gitops

Conceptual Showcase Architecture QAware | 14 Cluster API Management Cluster and Tenant Cluster Repo Tenant Platform Repo Management Cluster Manage Tenant Platform Repo Tenant 00 Platform Blueprint Fork Tenant 99 Virtual Tenant 01 Virtual Tenant Platform Repo Management vCluster

Cluster-API in a Nutshell QAware | 15 Cluster API is a Kubernetes sub-project focused on providing declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters. ■ Kubernetes Special Interest Group (SIG) Cluster Lifecycle ■ K8s-style APIs to Automate Cluster Lifecycle Management ■ Provides several CRDs, e.g. Machine, MachineSet and MachineDeployment and many more ■ Infrastructure like VMs, networks, load balancer as well as K8s configuration are also defined declaratively ■ Enables consistent and repeatable cluster deployments ■ Wide variety (30) of infrastructure providers supported ■ Many commercial platform providers also adopt Cluster-API Cluster KubeadmControlPlane GCPCluster GCPMachineTemplate KubeadmConfigTemplate MachineDeployment GCPMachineTemplate

Cluster-API Concepts QAware | 16

Lightweight Multi-Tenant K8s mit vCluster QAware | 17

qaware.de QAware GmbH Aschauer Straße 30 81549 München Tel. +49 89 232315-0 [email protected] linkedin.com/company/qaware-gmbh xing.com/companies/qawaregmbh slideshare.net/qaware github.com/qaware

We take responsibility and risks: From prototypes to large programs. We deliver. Guaranteed. Our cross-functional teams of consultants, developers and managers see themselves as enablers. We transform your organisation directly through project collaboration. With three guarantees: 1. Guarantee of success: We take responsibility and share your risks, for example through fixed prices. 2. Quality guarantee: You receive sustain- able, reliable quality software – docu- mented via KPIs and contractually fixed. 3. Satisfaction guarantee: We tie part of our remuneration to your satisfaction. 200 Engineers Munich Mainz Darmstadt Rosenheim Successful in the most demanding projects for 18 years Cloud Native Transformation & Host replacement: Tour guide into the future Data Value & AI: Open up data, network it & make it valuable 35 m € revenue Expertise for you Business Booster: Enable & accelerate business-critical visions Guaranteed success ■ BMW Aftersales Info Research ■ MaidlTC AI Optimizer ■ BMW GenAI Plattform ■ BMW PSBOM Stückliste ■ MunichRe Underwriting CP ■ Raiffeisenbank Südtirol Next Top Provider NPS 100 Top employer: 97% say "QAware is a very good workplace" ■ Allianz LEAP und Syncier Cloud ■ Hellmann HeRo ■ Ericsson KDTMES