10 Years of Kubernetes
Patterns Evolution
Bilgin Ibryam, Diagrid, @bibryam
Roland Huß, Red Hat,
@ro14nd@hachyderm.io
k8spatterns.com
Slide 2
Slide 2 text
Kubernetes Patterns 35
https://k8spatterns.com
Slide 3
Slide 3 text
3
Patterns
Slide 4
Slide 4 text
Patterns 34
Slide 5
Slide 5 text
Patterns 34
Slide 6
Slide 6 text
6
Kubernetes
Slide 7
Slide 7 text
Kubernetes
● Open Source container orchestration system
started by Google in 2014
⎈ Scheduling
⎈ Self-healing
⎈ Horizontal and vertical scaling
⎈ Service discovery
⎈ Application Deployments
● Declarative resource-centric REST API
34
Slide 8
Slide 8 text
Architecture 33
Control Plane
Slide 9
Slide 9 text
Pod 32
Slide 10
Slide 10 text
Resources 32
Slide 11
Slide 11 text
Pattern Categories 32
Slide 12
Slide 12 text
Foundational Patterns 31
Slide 13
Slide 13 text
13
Health
Probe
Slide 14
Slide 14 text
How to communicate
an application’s health
state to Kubernetes
Health Probe
Slide 15
Slide 15 text
Container Observability Options 30
Process APIs
Health Check APIs
Slide 16
Slide 16 text
Liveness & Readiness
● Liveness Probe
○ Restarting containers if liveness probes fails
● Readiness Probe
○ Removing from service endpoint if readiness probe fails
● Startup Probe
○ Restarting containers if startup probes fails
■ Probe methods
■ HTTP endpoint
■ gRPC endpoint
29
■ TCP socket endpoint
■ Unix command return value
How to enhance the
functionality of an
application without
changing it
Sidecar
Slide 22
Slide 22 text
Sidecar
● Runtime collaboration of containers
● Connected via shared resources:
○ Network
○ Volumes
● Similar what AOP is for programming
● Separation of concerns
25
Slide 23
Slide 23 text
Sidecar 24
Slide 24
Slide 24 text
● Dedicated sidecar containers (since 1.29, beta):
○ initContainers[].restartPolicy == Always
○ Continues to run after init containers have finished
○ Supports probes
23
Startup Order
Slide 25
Slide 25 text
Behavioral Patterns 22
Slide 26
Slide 26 text
Singleton
Service
Slide 27
Slide 27 text
Singleton Service
How to ensure that
only one application
instance is active
27
Out-of-Application Locking
● ReplicaSet with (at least) 1 replica
• Highly available Pod which is monitored and restarted in case of
failures
• Favors availability over consistency
→ more than one Pod can exists temporarily
● StatefulSet with (at most) 1 replica
• Favors consistency over availability
• Favors availability over consistency
→ less than one Pod can exists temporarily
21
Distributed lock API 18
✖
Lock
acquired
Waiting for
the lock
curl -X POST http://localhost:3500/v1.0-alpha1/lock/lockstore -H 'Content-Type: application/json'
-d ' {"resourceId":"my_file_name", "lockOwner":"abc123", "expiryInSeconds": 60}'
Slide 33
Slide 33 text
Configuration Patterns 17
Slide 34
Slide 34 text
Immutable
Configuration
Slide 35
Slide 35 text
How to configure your
application with
immutable container
images
Immutable Configuration
35
Slide 36
Slide 36 text
Immutable Configuration
● Configuration that can not be changed
● Different sets of configuration for different
environments (dev/prod)
● Versioning and auditing for configuration
● Solution: Configuration stored in container images
○ Versioned via tags and digests
○ Distributed via image registry
17
Slide 37
Slide 37 text
Configuration provided by Init Container 13
Slide 38
Slide 38 text
13
Directly sharing the container filesystem
● Sharing the process namespace,
including the file system via
shareProcessNamespace: true
● Technique for directly accessing
another containers filesystem
via /proc//root
● Used in modelcars in KServe for
accessing large language
models
Slide 39
Slide 39 text
Security Patterns 12
Slide 40
Slide 40 text
Process Containment
Slide 41
Slide 41 text
How to protect the
platform against
deployed code
Process Containment
41
Slide 42
Slide 42 text
Security Patterns 12
Slide 43
Slide 43 text
Run Containers with a Non-Root User
Forces any container in the Pod to run with
user ID 1000 and group ID 2000
19
apiVersion: v1
kind: Pod
metadata:
name: web-app
spec:
securityContext:
runAsUser: 1000
runAsGroup: 2000
containers:
- name: app
apiVersion: v1
kind: Pod
metadata:
name: web-app
spec:
securityContext:
runAsNonRoot: true
containers:
- name: app
Prevent any container from starting with
a root user—that is, a user with UID 0.
Slide 44
Slide 44 text
Lock the Capabilities
Prevents privilege escalation
19
apiVersion: v1
kind: Pod
metadata:
name: web-server
spec:
containers:
- name: httpd
image: httpd
securityContext:
allowPrivilegeEscalation: false
apiVersion: v1
kind: Pod
metadata:
name: web-server
spec:
securityContext:
capabilities
drop: ['ALL']
add: ['NET_BIND_SERVICE']
containers:
- name: httpd
Reduce capabilities to required ones only
Slide 45
Slide 45 text
Avoiding a Mutable Container Filesystem
Set read only file system
19
apiVersion: v1
kind: Pod
metadata:
name: web-server
spec:
containers:
- name: httpd
image: httpd
securityContext:
readOnlyRootFilesystem: true
apiVersion: v1
kind: Pod
metadata:
name: web-server
spec:
containers:
- name: httpd
image: httpd
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: - ALL
add: ['NET_BIND_SERVICE']
Overview of options we covered:
Slide 46
Slide 46 text
Advanced Patterns 8
Slide 47
Slide 47 text
Controller
Slide 48
Slide 48 text
How to get from the
current state to a
declared target state
Controller
48
Slide 49
Slide 49 text
State Reconciliation
● Kubernetes as distributed state manager
● Make the actual state more like the declared target
state.
🔎 Observe - Discover the actual state
🤔 Analyze - Determine difference to target state
🔨 Act- Perform actions to drive the actual to the desired state
8
Slide 50
Slide 50 text
Observe - Analyze - Act 7
Slide 51
Slide 51 text
Operator
Slide 52
Slide 52 text
How to encapsulate
operational knowledge
into executable
software
Operator
52
Slide 53
Slide 53 text
Definition
“” An operator is a Kubernetes controller that understands
two domains: Kubernetes and something else. By
combining knowledge of both areas, it can automate tasks
that usually require a human operator that understands
both domains.
Jimmy Zelinskie
http://bit.ly/2Fjlx1h
Operator = Controller + CustomResourceDefinition
6
Slide 54
Slide 54 text
CustomResourceDefinition
Custom resource is modelling a custom domain and
managed through the Kubernetes API
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: configwatchers.k8spatterns.io
spec:
scope: Namespaced
group: k8spatterns.io
version: v1
names:
kind: ConfigWatcher
plural: configwatchers
validation:
openAPIV3Schema:
...
5