10 Years of Kubernetes Patterns Evolution

Slide 1

Slide 1 text

10 Years of Kubernetes Patterns Evolution Bilgin Ibryam, Diagrid, @bibryam Roland Huß, Red Hat, @[email protected] k8spatterns.com

Slide 2

Slide 2 text

Kubernetes Patterns 35 https://k8spatterns.com

Slide 3

Slide 3 text

3 Patterns

Slide 4

Slide 4 text

Patterns 34

Slide 5

Slide 5 text

Patterns 34

Slide 6

Slide 6 text

6 Kubernetes

Slide 7

Slide 7 text

Kubernetes ● Open Source container orchestration system started by Google in 2014 ⎈ Scheduling ⎈ Self-healing ⎈ Horizontal and vertical scaling ⎈ Service discovery ⎈ Application Deployments ● Declarative resource-centric REST API 34

Slide 8

Slide 8 text

Architecture 33 Control Plane

Slide 9

Slide 9 text

Pod 32

Slide 10

Slide 10 text

Resources 32

Slide 11

Slide 11 text

Pattern Categories 32

Slide 12

Slide 12 text

Foundational Patterns 31

Slide 13

Slide 13 text

13 Health Probe

Slide 14

Slide 14 text

How to communicate an application’s health state to Kubernetes Health Probe

Slide 15

Slide 15 text

Container Observability Options 30 Process APIs Health Check APIs

Slide 16

Slide 16 text

Liveness & Readiness ● Liveness Probe ○ Restarting containers if liveness probes fails ● Readiness Probe ○ Removing from service endpoint if readiness probe fails ● Startup Probe ○ Restarting containers if startup probes fails ■ Probe methods ■ HTTP endpoint ■ gRPC endpoint 29 ■ TCP socket endpoint ■ Unix command return value

Slide 17

Slide 17 text

Health Probe Example containers: - image: k8spatterns/random-generator:1.0 name: random-generator livenessProbe: httpGet: path: /actuator/health port: 8080 readinessProbe: grpc: port: 5000 startupProbe: exec: command: [ "stat", "/opt/jboss/wildfly/tmp/startup-marker"] initialDelaySeconds: 30 periodSeconds: 60 failureThreshold: 15 28

Slide 18

Slide 18 text

Foundational Patterns Dependencies 27 Automated Placement Predictable Demands Declarative Deployment Managed Lifecycle Health Probe

Slide 19

Slide 19 text

Structural Patterns 26

Slide 20

Slide 20 text

Sidecar

Slide 21

Slide 21 text

How to enhance the functionality of an application without changing it Sidecar

Slide 22

Slide 22 text

Sidecar ● Runtime collaboration of containers ● Connected via shared resources: ○ Network ○ Volumes ● Similar what AOP is for programming ● Separation of concerns 25

Slide 23

Slide 23 text

Sidecar 24

Slide 24

Slide 24 text

● Dedicated sidecar containers (since 1.29, beta): ○ initContainers[].restartPolicy == Always ○ Continues to run after init containers have ﬁnished ○ Supports probes 23 Startup Order

Slide 25

Slide 25 text

Behavioral Patterns 22

Slide 26

Slide 26 text

Singleton Service

Slide 27

Slide 27 text

Singleton Service How to ensure that only one application instance is active 27

Slide 28

Slide 28 text

Out-of-Application Locking 21 kind: ReplicaSet metadata: name: file-poller spec: serviceName: file-poller replicas: 1

Slide 29

Slide 29 text

Out-of-Application Locking ● ReplicaSet with (at least) 1 replica • Highly available Pod which is monitored and restarted in case of failures • Favors availability over consistency → more than one Pod can exists temporarily ● StatefulSet with (at most) 1 replica • Favors consistency over availability • Favors availability over consistency → less than one Pod can exists temporarily 21

Slide 30

Slide 30 text

In-Application Locking 20

Slide 31

Slide 31 text

In-Application Locking ● Distributed lock shared by simultaneously running applications ● Active-Passive topology ● Distributed lock implementations e.g. ○ Zookeeper ○ Consul ○ Redis ○ etcd, ○ Kubernetes ConﬁgMaps, Lease resource 19

Slide 32

Slide 32 text

Distributed lock API 18 ✖ Lock acquired Waiting for the lock curl -X POST http://localhost:3500/v1.0-alpha1/lock/lockstore -H 'Content-Type: application/json' -d ' {"resourceId":"my_file_name", "lockOwner":"abc123", "expiryInSeconds": 60}'

Slide 33

Slide 33 text

Conﬁguration Patterns 17

Slide 34

Slide 34 text

Immutable Conﬁguration

Slide 35

Slide 35 text

How to conﬁgure your application with immutable container images Immutable Conﬁguration 35

Slide 36

Slide 36 text

Immutable Configuration ● Configuration that can not be changed ● Different sets of configuration for different environments (dev/prod) ● Versioning and auditing for configuration ● Solution: Configuration stored in container images ○ Versioned via tags and digests ○ Distributed via image registry 17

Slide 37

Slide 37 text

Conﬁguration provided by Init Container 13

Slide 38

Slide 38 text

13 Directly sharing the container filesystem ● Sharing the process namespace, including the file system via shareProcessNamespace: true ● Technique for directly accessing another containers filesystem via /proc//root ● Used in modelcars in KServe for accessing large language models

Slide 39

Slide 39 text

Security Patterns 12

Slide 40

Slide 40 text

Process Containment

Slide 41

Slide 41 text

How to protect the platform against deployed code Process Containment 41

Slide 42

Slide 42 text

Security Patterns 12

Slide 43

Slide 43 text

Run Containers with a Non-Root User Forces any container in the Pod to run with user ID 1000 and group ID 2000 19 apiVersion: v1 kind: Pod metadata: name: web-app spec: securityContext: runAsUser: 1000 runAsGroup: 2000 containers: - name: app apiVersion: v1 kind: Pod metadata: name: web-app spec: securityContext: runAsNonRoot: true containers: - name: app Prevent any container from starting with a root user—that is, a user with UID 0.

Slide 44

Slide 44 text

Lock the Capabilities Prevents privilege escalation 19 apiVersion: v1 kind: Pod metadata: name: web-server spec: containers: - name: httpd image: httpd securityContext: allowPrivilegeEscalation: false apiVersion: v1 kind: Pod metadata: name: web-server spec: securityContext: capabilities drop: ['ALL'] add: ['NET_BIND_SERVICE'] containers: - name: httpd Reduce capabilities to required ones only

Slide 45

Slide 45 text

Avoiding a Mutable Container Filesystem Set read only ﬁle system 19 apiVersion: v1 kind: Pod metadata: name: web-server spec: containers: - name: httpd image: httpd securityContext: readOnlyRootFilesystem: true apiVersion: v1 kind: Pod metadata: name: web-server spec: containers: - name: httpd image: httpd securityContext: runAsNonRoot: true allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: - ALL add: ['NET_BIND_SERVICE'] Overview of options we covered:

Slide 46

Slide 46 text

Advanced Patterns 8

Slide 47

Slide 47 text

Controller

Slide 48

Slide 48 text

How to get from the current state to a declared target state Controller 48

Slide 49

Slide 49 text

State Reconciliation ● Kubernetes as distributed state manager ● Make the actual state more like the declared target state. 🔎 Observe - Discover the actual state 🤔 Analyze - Determine difference to target state 🔨 Act- Perform actions to drive the actual to the desired state 8

Slide 50

Slide 50 text

Observe - Analyze - Act 7

Slide 51

Slide 51 text

Operator

Slide 52

Slide 52 text

How to encapsulate operational knowledge into executable software Operator 52

Slide 53

Slide 53 text

Deﬁnition “” An operator is a Kubernetes controller that understands two domains: Kubernetes and something else. By combining knowledge of both areas, it can automate tasks that usually require a human operator that understands both domains. Jimmy Zelinskie http://bit.ly/2Fjlx1h Operator = Controller + CustomResourceDeﬁnition 6

Slide 54

Slide 54 text

CustomResourceDeﬁnition Custom resource is modelling a custom domain and managed through the Kubernetes API apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: configwatchers.k8spatterns.io spec: scope: Namespaced group: k8spatterns.io version: v1 names: kind: ConfigWatcher plural: configwatchers validation: openAPIV3Schema: ... 5

Slide 55

Slide 55 text

Custom Resource kind: ConfigWatcher apiVersion: k8spatterns.io/v1 metadata: name: webapp-config-watcher spec: configMap: webapp-config podSelector: app: webapp 5

Slide 56

Slide 56 text

Operator 4

Slide 57

Slide 57 text

CRD Classiﬁcation ● Installation CRDs ○ Installing and operating applications ○ Backup and Restore ○ Monitoring and self-healing ○ Example: Prometheus for installing Prometheus & components ● Application CRDs ○ Application speciﬁc domain concepts ○ Example: ServiceMonitor for registering Kubernetes service to be scraped by Prometheus 3

Slide 58

Slide 58 text

Operator Hub 3

Slide 59

Slide 59 text

May the duck be with you … Wednesday, 13:00 Thursday, 12:30 https://k8spatterns.com