HDC24: A Life without VMs: High-Scale Kubernetes on Bare Metal

Transcript

1. VIKTOR ABRAMS @mittwald MARTIN HELMICH @mittwald A LIFE WITHOUT VMs HIGH - SCALE KUBERNETES ON BARE METAL HELLMANN DEVELOPER & MAKER CONFERENCE August 30th, 2024
2. https://web.archive.org/web/20040829084706/http://www.espelkamper-nachrichten.de/ MY FIRST ( PAID ) WEBSITE
3. HOSTING BACK THEN
4. HOSTING BACK THEN
5. APACHE PHP MYSQL HOSTING BACK THEN
6. APACHE PHP MYSQL ... AND NOW OPENSEARCH KEYCLOAK MOBILE APP ...OR NODE.JS DOCKER PWA NGINX 💥 ...OR P CDN HOSTING BACK THEN BOOP VARNISH
8. MITTWALD CLOUD PLATFORM ARCHITECTURAL GOALS FLEXIBILITY RESOURCE EFFICIENCY ( INTERNAL ) EASE OF USE *) apart from obvious ones, like security, reliability and performance *
9. MITTWALD CLOUD PLATFORM ARCHITECTURAL GOALS FLEXIBILITY STANDARDS CONFORMANCE KUBERNETES/COE CONTAINERIZED CUSTOMIZABLE OCI IMAGES QUALITY GOAL SOLUTION STRATEGY RESOURCE EFFICIENCY ELASTICITY OPTIMIZED SCHEDULING ( INTERNAL ) EASE OF USE DECLARATIVE APIS KUBERNETES OPERATORS
10. NODE BASIC KUBERNETES PRINCIPLES NODE NODE CLUSTER YEET USER POD POD POD POD ( SLIGHTLY SIMPLIFIED ) POD
11. martin @ local $ mw app install typo3 -q --version=12.4.17 --install-mode composer [...] martin @ local $ kubectl get pods --all-namespaces \ --field-selector spec.nodeName=shoot--chp-prod--gestringen-pm-sps-3669c938-z1-797d7-gtfmz NAMESPACE NAME READY STATUS RESTARTS AGE kube-system node-local-dns-69btw 1/1 Running 0 36d kube-system node-problem-detector-2bq2z 1/1 Running 0 36d kube-system openstack-manila-csi-nodeplugin-sk8sm 2/2 Running 0 36d monitoring kube-prometheus-stack-prometheus-node-exporter-dnq9l 1/1 Running 0 36d monitoring loki-stack-promtail-mntw2 1/1 Running 0 36d pg-s-piryq0 mysql-default-80-low-latency-knlw2-0 1/1 Running 0 12s pg-s-piryq0 p-elfmx0-64db467d4b-qzmdv 2/2 Running 0 2d20h pg-s-piryq0 p-elfmx0-l2qzd-b79b7fdf6-qxjl4 2/2 Running 0 12s pg-s-piryq0 p-elfmx0-logrotation-28698480-47mvh 0/1 Completed 0 23h
12. DEPLOYMENT REPLICASET POD STATEFULSET DAEMONSET SERVICE INGRESS INGRESS CLASS PERSISTENT VOLUME PERSISTENT VOLUMECLAIM STORAGE CLASS JOB CRON JOB CERTIFICATE SECRET CONFIG MAP NODE
14. ABSTRACTION
15. DEPLOYMENT REPLICASET POD STATEFULSET DAEMONSET SERVICE INGRESS INGRESS PERSISTENT VOLUME PERSISTENT VOLUMECLAIM STORAGE CLASS JOB CRON JOB CERTIFICATE SECRET CONFIG MAP NODE KUBERNTES CORE CONCEPTS PROJECT CUSTOMER PROJECT GROUP APP DATABASE DOMAIN HOSTING DOMAIN MODEL
16. apiVersion: apphosting.mittwald.cloud/v1 kind: App metadata: name: a-XXXXX namespace: s-YYYYY spec: app: typo3 version: 12.4.17 project: name: p-ZZZZZ requirements: - name: php version: ^8.3 - name: composer version: ^2 installPath: /my-typo3-site
17. CUSTOM RESOURCE CONTROLLER KUBERNETES API USER create / update watch notify reconcile desired state control loop
19. PRODUCT SPECIFIC K8S OPERATORS K8S CORE COMPONENTS
20. MACRO ARCHITECTURE PRODUCT SPECIFIC MICROSERVICES PRODUCT SPECIFIC K8S OPERATORS K8S CORE COMPONENTS IaaS CLOUD DATA CENTER
21. ORGANIZATIONAL STRUCTURE PRODUCT SPECIFIC MICROSERVICES PRODUCT SPECIFIC K8S OPERATORS K8S CORE COMPONENTS IaaS CLOUD DATA CENTER PRODUCT DEV TEAMS CLOUD PLATFORM TEAM IaaS + NETWORK TEAM DATA CENTER OPERATIONS
22. UNDER THE HOOD
23. WHAT IS YOUR PRODUCT? AND WHAT DO YOU NEED TO RUN IT?
24. m³ ( Pronunciation: "m-cubed") NATIVE KUBERNETES WEBHOSTING
25. m³ ( Pronunciation: "m-cubed") VISIBILITY RESOURCE MANAGEMENT BARE METAL PROVISIONING VIRTUAL CLUSTERS ROUTE REFLECTION MAIN GOALS
26. /etc /mnt /srv httpd php-fpm sshd VISIBILITY NAMESPACES /home python /mnt /var httpd /mnt /etc /srv httpd php-fpm MySQL 1. Processes 2. File system 3. Network
27. RESOURCE MANAGEMENT CONTROL GROUPS cpu1 cpu2 cpu3 httpd node php-fpm 1. Memory 2. Compute 3. I/O
28. BARE METAL PROVISIONING control- plane worker worker worker Rack 1 control -​ plane worker worker worker Rack 2 control -​ plane worker worker Rack 3 netbox Adapter dhcp bootp cloud- init CORE BGP EFI RAID EFI RAID LVM root Volume 1..n BGP worker cloud- image
29. VIRTUAL CLUSTERS systemd cri-o kubelet ... systemd cri-o kubelet ... systemd cri-o kubelet ... systemd cri-o kubelet ... Control-Plane Worker 1 Worker 2 Worker N kluster.m3.services Cluster Workergroup Cluster-Operator
30. ROUTE REFLECTION vnode 1 vnode 2 vnode 3 vnode 5 vnode 4 SWITCH 1 SWITCH 2 frr frr NODE 1 NODE 2
31. MAIN GOALS _ IP DUAL STACK _ PERFORMANCE FIRST _ MANAGEABLE UNITS (no micromanagement) _ COMPLETE INVENTORY OVERVIEW (datacenter infrastructure management; DCIM ) _ COMPLEXITY REDUCTION i.e.: less abstraction (compute, network, ...)
32. https://github.com/mittwald https://github.com/martin-helmich https://www.mittwald.de https://www.linkedin.com/company/mittwald https://www.linkedin.com/in/martinhelmich