T3DD24: How to build your own cloud platform

Hosting and infrastructure operations work differently today than they did 10 or 20 years ago. New demands in scalability, automation and technological flexibility necessitate new solutions. In this talk, I will give insights into the internal architecture of the mittwald cloud platform, and explain how we leveraged modern cloud technologies like OpenStack and Kubernetes to build a platform to not only support traditional LAMP environments, but also modern architectures like Headless CMS and new deployment strategies like containerized deployments.

I will also talk about the challenges that we faced (and maybe underestimated) in both development and operations, and which strategies we developed to overcome them.

Transcript

1. MARTIN HELMICH @mittwald HOW TO BUILD YOUR OWN CLOUD PLATFORM August 3rd, 2024
2. MARTIN HELMICH Head of Architecture & Developer Relations Lecturer, Software Engineering & Cloud Computing Sci-Fi-Nerd, Metalhead, Amateur Woodworker
3. Image Source
4. HOSTING BACK THEN
5. HOSTING BACK THEN
6. APACHE PHP MYSQL HOSTING BACK THEN
7. APACHE PHP MYSQL ... AND NOW OPENSEARCH KEYCLOAK MOBILE APP ...OR NODE.JS DOCKER PWA NGINX 💥 ...OR P CDN HOSTING BACK THEN BOOP VARNISH
9. MITTWALD CLOUD PLATFORM ARCHITECTURAL GOALS FLEXIBILITY RESOURCE EFFICIENCY ( INTERNAL ) EASE OF USE *) apart from obvious ones, like security, reliability and performance *
10. MITTWALD CLOUD PLATFORM ARCHITECTURAL GOALS FLEXIBILITY STANDARDS CONFORMANCE KUBERNETES/COE CONTAINERIZED CUSTOMIZABLE OCI IMAGES QUALITY GOAL SOLUTION STRATEGY RESOURCE EFFICIENCY ELASTICITY OPTIMIZED SCHEDULING ( INTERNAL ) EASE OF USE DECLARATIVE APIS KUBERNETES OPERATORS
11. NODE BASIC KUBERNETES PRINCIPLES NODE NODE CLUSTER YEET USER POD POD POD POD ( SLIGHTLY SIMPLIFIED ) POD
12. martin @ local $ mw app install typo3 -q --version=12.4.17 --install-mode composer [...] martin @ local $ kubectl get pods --all-namespaces \ --field-selector spec.nodeName=shoot--chp-prod--gestringen-pm-sps-3669c938-z1-797d7-gtfmz NAMESPACE NAME READY STATUS RESTARTS AGE kube-system node-local-dns-69btw 1/1 Running 0 36d kube-system node-problem-detector-2bq2z 1/1 Running 0 36d kube-system openstack-manila-csi-nodeplugin-sk8sm 2/2 Running 0 36d monitoring kube-prometheus-stack-prometheus-node-exporter-dnq9l 1/1 Running 0 36d monitoring loki-stack-promtail-mntw2 1/1 Running 0 36d pg-s-piryq0 mysql-default-80-low-latency-knlw2-0 1/1 Running 0 12s pg-s-piryq0 p-elfmx0-64db467d4b-qzmdv 2/2 Running 0 2d20h pg-s-piryq0 p-elfmx0-l2qzd-b79b7fdf6-qxjl4 2/2 Running 0 12s pg-s-piryq0 p-elfmx0-logrotation-28698480-47mvh 0/1 Completed 0 23h
13. POD
14. DEPLOYMENT REPLICASET POD STATEFULSET DAEMONSET SERVICE INGRESS INGRESS CLASS PERSISTENT VOLUME PERSISTENT VOLUMECLAIM STORAGE CLASS JOB CRON JOB CERTIFICATE SECRET CONFIG MAP NODE
17. ABSTRACTION
18. DEPLOYMENT REPLICASET POD STATEFULSET DAEMONSET SERVICE INGRESS INGRESS CLASS PERSISTENT VOLUME PERSISTENT VOLUMECLAIM STORAGE CLASS JOB CRON JOB CERTIFICATE SECRET CONFIG MAP NODE KUBERNTES CORE CONCEPTS PROJECT CUSTOMER PROJECT GROUP APP DATABASE DOMAIN HOSTING DOMAIN MODEL
19. apiVersion: apphosting.mittwald.cloud/v1 kind: App metadata: name: a-XXXXX namespace: s-YYYYY spec: app: typo3 version: 12.4.17 project: name: p-ZZZZZ requirements: - name: php version: ^8.3 - name: composer version: ^2 installPath: /my-typo3-site
20. CUSTOM RESOURCE CONTROLLER KUBERNETES API USER create / update watch notify reconcile desired state control loop
21. https://speakerdeck.com/martinhelmich/the-anatomy-of-docker-image
23. PRODUCT SPECIFIC K8S OPERATORS K8S CORE COMPONENTS
24. MACRO ARCHITECTURE PRODUCT SPECIFIC MICROSERVICES PRODUCT SPECIFIC K8S OPERATORS K8S CORE COMPONENTS IaaS CLOUD DATA CENTER
25. ORGANIZATIONAL STRUCTURE PRODUCT SPECIFIC MICROSERVICES PRODUCT SPECIFIC K8S OPERATORS K8S CORE COMPONENTS IaaS CLOUD DATA CENTER PRODUCT DEV TEAMS CLOUD PLATFORM TEAM IaaS + NETWORK TEAM DATA CENTER OPERATIONS
26. UNDER THE HOOD
27. ON PREM YES YES NO OPEN SOURCE NO HOW - TO CLOUD
28. OPEN - SOURCE CLUSTER PROVISIONING CLUSTER API GARDENER OPENSTACK MAGNUM
29. https://docs.openstack.org/security-guide/networking/architecture.html
31. WTF? THE STORAGE PROBLEM ( TRADE - OFFS WHEREVER YOU LOOK )
32. NODE POD LOCAL VOLUME - Fast - ( When used as block device) can be optimized for workloads (e.g. XFS for databases) - Limited scalability - Inefficient resource utilization - No networked access - No HA NODE POD ⛔ CEPH NETWORK BLOCK DEVICE - Fast(ish) - Can be optimized for workloads (e.g. XFS for databases) - Networked access by a single client - Reasonably scalable (limited by FS ) - Efficient resource utilization - Various HA configurations with different resource efficiencies and performances (replication & erasure coding) ⛔ CephFS NETWORK FILE SYSTEM - Fast(ish) with few large files - Slow with many small files (tolerable with LOTS of caching) - Networked access by multiple clients - Various HA configurations with different resource efficiencies and performances (replication & erasure coding) NETWORK OBJECT STORE - No filesystem access, only via API - Needs to be considered in architecture on application level - Very high scalability - Networked access by multiple clients RBD ROG THE STORAGE PROBLEM ( TRADE - OFFS WHEREVER YOU LOOK )
33. Gardener BARE METAL SERVERS
35. Lukas Tennie https://unsplash.com/photos/a-close-up-of-a-watch-face-with-the-gears-missing-3dyDozzCORw COMPLEXITY
36. Ralph_PH, CC - BY K I S S EEP T IMPLE, TUPID.
37. WHAT IS YOUR PRODUCT? AND WHAT DO YOU NEED TO RUN IT?
38. Gardener BARE METAL SERVERS
39. BARE METAL SERVERS
40. BARE METAL SERVERS Metallb OUR APPROACH : KUBERNETES - NATIVE EVERYTHING STORAGE VIRTUALIZATION NETWORKING metal-stack 3
41. https://metal-stack.io/ https://github.com/onmetal https://scs.community
42. ROADMAP INDIVIDUALIZED CUSTOMER SOLUTIONS 3RD PARTY EXTENSIBILITY SCALABILITY ELASTICITY + ( STRATEGIC ) CONFIGURABLE STORAGE OPTIONS ... AND WILD IDEAS SCALE - TO - ZERO HOSTING AI MODELS 3RD PARTY INFRASTRUCTURE ...AND MANY MORE
43. ACCEPT COMPLEXITY WHERE YOU NEED IT ( AND MANAGE ACCORDINGLY ) KEEP IT SIMPLE WHERE YOU DON'T KNOW YOUR PRODUCT
44. https://github.com/martin-helmich https://www.mittwald.de https://www.martin-helmich.de https://www.linkedin.com/in/martinhelmich