10 Years of Kubernetes Patterns Evolution

Much like the timeless 'Gang of Four' design patterns for object-oriented programming, Kubernetes has spawned its own timeless design patterns for distributed applications design. These patterns are not utilizing objects, classes, and interfaces, but new primitives pioneered by Kubernetes, such as Pods, Jobs, and Deployments, that span across processes, and nodes. In this talk, Bilgin & Roland will explore the most widely used Kubernetes design patterns and their evolution over the 10 years of Kubernetes existence. These patterns prevent reinventing the wheel by offering time-tested solutions to common challenges. Understanding this pattern language is crucial for grasping the Kubernetes' mindset, boosting your ability to comprehend, discuss, and design Kubernetes-based applications that are good cloud-native citizens set to last another decade.

Transcript

1. 10 Years of Kubernetes Patterns Evolution Bilgin Ibryam, Diagrid, @bibryam Roland Huß, Red Hat, @ro14nd@hachyderm.io k8spatterns.com
2. Kubernetes Patterns 35 https://k8spatterns.com
3. 3 Patterns
4. Patterns 34
5. Patterns 34
6. 6 Kubernetes
7. Kubernetes • Open Source container orchestration system started by Google in 2014 ⎈ Scheduling ⎈ Self-healing ⎈ Horizontal and vertical scaling ⎈ Service discovery ⎈ Application Deployments • Declarative resource-centric REST API 34
8. Architecture 33 Control Plane
9. Pod 32
10. Resources 32
11. Pattern Categories 32
12. Foundational Patterns 31
13. 13 Health Probe
14. How to communicate an application’s health state to Kubernetes Health Probe
15. Container Observability Options 30 Process APIs Health Check APIs
16. Liveness & Readiness • Liveness Probe ◦ Restarting containers if liveness probes fails • Readiness Probe ◦ Removing from service endpoint if readiness probe fails • Startup Probe ◦ Restarting containers if startup probes fails ▪ Probe methods ▪ HTTP endpoint ▪ gRPC endpoint 29 ▪ TCP socket endpoint ▪ Unix command return value
17. Health Probe Example containers: - image: k8spatterns/random-generator:1.0 name: random-generator livenessProbe: httpGet: path: /actuator/health port: 8080 readinessProbe: grpc: port: 5000 startupProbe: exec: command: [ "stat", "/opt/jboss/wildfly/tmp/startup-marker"] initialDelaySeconds: 30 periodSeconds: 60 failureThreshold: 15 28
18. Foundational Patterns Dependencies 27 Automated Placement Predictable Demands Declarative Deployment Managed Lifecycle Health Probe
19. Structural Patterns 26
20. Sidecar
21. How to enhance the functionality of an application without changing it Sidecar
22. Sidecar • Runtime collaboration of containers • Connected via shared resources: ◦ Network ◦ Volumes • Similar what AOP is for programming • Separation of concerns 25
23. Sidecar 24
24. • Dedicated sidecar containers (since 1.29, beta): ◦ initContainers[].restartPolicy == Always ◦ Continues to run after init containers have finished ◦ Supports probes 23 Startup Order
25. Behavioral Patterns 22
26. Singleton Service
27. Singleton Service How to ensure that only one application instance is active 27
28. Out-of-Application Locking 21 kind: ReplicaSet metadata: name: file-poller spec: serviceName: file-poller replicas: 1
29. Out-of-Application Locking • ReplicaSet with (at least) 1 replica • Highly available Pod which is monitored and restarted in case of failures • Favors availability over consistency → more than one Pod can exists temporarily • StatefulSet with (at most) 1 replica • Favors consistency over availability • Favors availability over consistency → less than one Pod can exists temporarily 21
30. In-Application Locking 20
31. In-Application Locking • Distributed lock shared by simultaneously running applications • Active-Passive topology • Distributed lock implementations e.g. ◦ Zookeeper ◦ Consul ◦ Redis ◦ etcd, ◦ Kubernetes ConfigMaps, Lease resource 19
32. Distributed lock API 18 ✖ Lock acquired Waiting for the lock curl -X POST http://localhost:3500/v1.0-alpha1/lock/lockstore -H 'Content-Type: application/json' -d ' {"resourceId":"my_file_name", "lockOwner":"abc123", "expiryInSeconds": 60}'
33. Configuration Patterns 17
34. Immutable Configuration
35. How to configure your application with immutable container images Immutable Configuration 35
36. Immutable Configuration • Configuration that can not be changed • Different sets of configuration for different environments (dev/prod) • Versioning and auditing for configuration • Solution: Configuration stored in container images ◦ Versioned via tags and digests ◦ Distributed via image registry 17
37. Configuration provided by Init Container 13
38. 13 Directly sharing the container filesystem • Sharing the process namespace, including the file system via shareProcessNamespace: true • Technique for directly accessing another containers filesystem via /proc/<pid>/root • Used in modelcars in KServe for accessing large language models
39. Security Patterns 12
40. Process Containment
41. How to protect the platform against deployed code Process Containment 41
42. Security Patterns 12
43. Run Containers with a Non-Root User Forces any container in the Pod to run with user ID 1000 and group ID 2000 19 apiVersion: v1 kind: Pod metadata: name: web-app spec: securityContext: runAsUser: 1000 runAsGroup: 2000 containers: - name: app apiVersion: v1 kind: Pod metadata: name: web-app spec: securityContext: runAsNonRoot: true containers: - name: app Prevent any container from starting with a root user—that is, a user with UID 0.
44. Lock the Capabilities Prevents privilege escalation 19 apiVersion: v1 kind: Pod metadata: name: web-server spec: containers: - name: httpd image: httpd securityContext: allowPrivilegeEscalation: false apiVersion: v1 kind: Pod metadata: name: web-server spec: securityContext: capabilities drop: ['ALL'] add: ['NET_BIND_SERVICE'] containers: - name: httpd Reduce capabilities to required ones only
45. Avoiding a Mutable Container Filesystem Set read only file system 19 apiVersion: v1 kind: Pod metadata: name: web-server spec: containers: - name: httpd image: httpd securityContext: readOnlyRootFilesystem: true apiVersion: v1 kind: Pod metadata: name: web-server spec: containers: - name: httpd image: httpd securityContext: runAsNonRoot: true allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: - ALL add: ['NET_BIND_SERVICE'] Overview of options we covered:
46. Advanced Patterns 8
47. Controller
48. How to get from the current state to a declared target state Controller 48
49. State Reconciliation • Kubernetes as distributed state manager • Make the actual state more like the declared target state. 🔎 Observe - Discover the actual state 🤔 Analyze - Determine difference to target state 🔨 Act- Perform actions to drive the actual to the desired state 8
50. Observe - Analyze - Act 7
51. Operator
52. How to encapsulate operational knowledge into executable software Operator 52
53. Definition “” An operator is a Kubernetes controller that understands two domains: Kubernetes and something else. By combining knowledge of both areas, it can automate tasks that usually require a human operator that understands both domains. Jimmy Zelinskie http://bit.ly/2Fjlx1h Operator = Controller + CustomResourceDefinition 6
54. CustomResourceDefinition Custom resource is modelling a custom domain and managed through the Kubernetes API apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: configwatchers.k8spatterns.io spec: scope: Namespaced group: k8spatterns.io version: v1 names: kind: ConfigWatcher plural: configwatchers validation: openAPIV3Schema: ... 5
55. Custom Resource kind: ConfigWatcher apiVersion: k8spatterns.io/v1 metadata: name: webapp-config-watcher spec: configMap: webapp-config podSelector: app: webapp 5
56. Operator 4
57. CRD Classification • Installation CRDs ◦ Installing and operating applications ◦ Backup and Restore ◦ Monitoring and self-healing ◦ Example: Prometheus for installing Prometheus & components • Application CRDs ◦ Application specific domain concepts ◦ Example: ServiceMonitor for registering Kubernetes service to be scraped by Prometheus 3
58. Operator Hub 3
59. May the duck be with you … Wednesday, 13:00 Thursday, 12:30 https://k8spatterns.com