Design and Pa+erns of Distributed Systems Using Kubernetes as a real-world Example Michael Gasch (June 2023) 

2 (c) Michael Gasch 2023 @embano1 about://me • First Computer 1992 (C-64) • IHK Applied Computer Science/Max-Planck Society (-2007) • Systems Engineer Dell (-2015) • Engineer and Research VMware Office of the CTO (-2022) • Product Manager AWS EventBridge (2022-) • Self-taught Developer (Golang) • Public Speaking and Blogging (@embano1 www.mgasch.com) • ! and " 

Act 1 – Kubernetes Introduction 

4 (c) Michael Gasch 2023 @embano1 In 2013 Docker changed the IT World Credit: docker.com 

5 (c) Michael Gasch 2023 @embano1 Kubernetes won the Container OrchestraAon War 

6 (c) Michael Gasch 2023 @embano1 Overview Kubernetes Architecture Control Plane API Server etcd Controller Manager Scheduler … Access REST API SDKs Web UI kubectl Workers Kubelet Kubelet Kubelet Pod Pod Pod 

7 (c) Michael Gasch 2023 @embano1 Distributed Systems ain’t easy 

8 (c) Michael Gasch 2023 @embano1 Design Considerations Kubernetes Architecture Control Plane API Server etcd Controller Manager Scheduler … Access REST API SDKs Web UI kubectl Workers Kubelet Kubelet Kubelet Pod Pod Pod Availability Scalability Resiliency Security Flow Control Extensibility Observability Durability Consistency Deployment Versioning Responsiveness Programming Model Open Source Recovery Communication 

9 (c) Michael Gasch 2023 @embano1 (inherent) Complexity 

Act 2 – Taming Complexity with Patterns 

11 (c) Michael Gasch 2023 @embano1 • Common Definition: “A Solution to a Problem in a Context” • Applicable in lots of different Situations • Patterns are Solutions • Why as well as how • Code Examples Defini&on Patterns Source: https://martinfowler.com/articles/writingPatterns.html 

12 (c) Michael Gasch 2023 @embano1 • Consistent Core • Control (Feedback) Loops* • Idempotent Receiver • Leader-Follower • Quorum • Replicated Log • State Watch • Versioned Value Examples Patterns Source: h;ps://mar

Act 3 – Control Loops 

14 (c) Michael Gasch 2023 @embano1 Kubernetes Architecture Control Plane API Server etcd Controller Manager Scheduler … Access REST API SDKs Web UI kubectl Workers Kubelet Kubelet Kubelet Pod Pod Pod 

15 (c) Michael Gasch 2023 @embano1 Control Loops everywhere Kubernetes Architecture Control Plane API Server Workers = Control Loop 

16 (c) Michael Gasch 2023 @embano1 Commands vs Events Kubernetes Architecture Commands Events • Requests (intent) to do something • Named in the impera)ve, e.g. “CREATE” • Can be rejected • Higher coupling between sender and owner • Typically used in synchronous 1-to-1 request/response communicaEon • Something that has happened (a fact) • Named in past tense, e.g. “CREATED” • Cannot (semantically) be rejected by receiver • Lowest coupling between sender and owner • Asynchronous 1-to-many communication, e.g. publish/subscribe 

17 (c) Michael Gasch 2023 @embano1 Request Flow Kubernetes Architecture API Server REST REST Decoding Conversion & Defaulting Admission Persistency (etcd) … WATCH POST $ kubectl create –f my_replicaset.yaml apiVersion: extensions/v1beta1 kind: ReplicaSet spec: replicas: 2 Commands Events EVENT 

18 (c) Michael Gasch 2023 @embano1 Inside the Control Loop Kubernetes Architecture Observe Analyze Act apiVersion: extensions/v1beta1 kind: ReplicaSet spec: replicas: 2 desired := getDesiredState() current := getCurrentState() diff := desired – current if diff < 0 { deletePods() } if diff > 0 { createPods() } Command Event (Edge-Triggered) Event (Level-Triggered) 

19 (c) Michael Gasch 2023 @embano1 Controllers, oh my… Kubernetes Architecture 

20 (c) Michael Gasch 2023 @embano1 Choreography (over Coordina,on) 

21 (c) Michael Gasch 2023 @embano1 Asynchronous IntegraGon Kubernetes Architecture API Server CREATE apiVersion: extensions/v1beta1 kind: ReplicaSet spec: replicas: 2 CREATE Pod ReplicaSet CREATED ReplicaSet Controller BIND Pod Pod CREATED Scheduler Kubelet Pod BOUND UPDATE Pod (“running”) Time Command Event 

Act 4- Writing Controllers 

23 • Single Responsible Principle • Decoupling via event-driven Messaging • No central Coordinator A different Mindset WriAng Controllers Autonomous Processes 

24 • Eventual consistent by Design • Don’t rely on (assume) Order • Single Responsible Principle • Decoupling via event-driven Messaging • No central Coordinator A different Mindset Writing Controllers Autonomous Processes Concurrency & Asynchrony 

25 • API server (etcd) is the Source of Truth* • In-memory Cache via Reconciliation • Eventual consistent by Design • Don’t rely on (assume) Order • Single Responsible Principle • Decoupling via event-driven Messaging • No central Coordinator A different Mindset Writing Controllers Autonomous Processes Concurrency & Asynchrony Stateless over Stateful 

26 • Things will go wrong (crash) • No shared (wall) Clock • Anticipate Effects on the Rest of the System • API server (etcd) is the Source of Truth* • In-memory Cache via Reconciliation • Eventual consistent by Design • Don’t rely on (assume) Order • Single Responsible Principle • Decoupling via event-driven Messaging • No central Coordinator A different Mindset Writing Controllers Autonomous Processes Concurrency & Asynchrony Stateless over Stateful Defensive Programming 

27 • Delivery and Processing Guarantees only within Kubernetes • Things will go wrong (crash) • No shared (wall) Clock • Anticipate Effects on the Rest of the System • API server (etcd) is the Source of Truth* • In-memory Cache via Reconciliation • Eventual consistent by Design • Don’t rely on (assume) Order • Single Responsible Principle • Decoupling via event-driven Messaging • No central Coordinator A different Mindset Writing Controllers Autonomous Processes Concurrency & Asynchrony Stateless over Stateful Side Effects Defensive Programming 

Act 5 – State Watch Pattern Kubernetes ListerWatcher Implementation 

29 (c) Michael Gasch 2023 @embano1 Clients are interested in changes to the specific values on the server. It's difficult for clients to structure their logic if they need to poll the server conbnuously to look for changes. If clients open too many connecbons to the server for watching changes, it can overwhelm the server. Problem State Watch Pattern 

30 (c) Michael Gasch 2023 @embano1 Allow clients to register their interest with the server for specific state changes. The server nobfies the interested clients when state changes happen. The client maintains a Single Socket Channel with the server. The server sends state change nobficabons on this channel. Solution State Watch Pattern 

31 (c) Michael Gasch 2023 @embano1 Kubernetes ListerWatcher: LIST Phase State Watch Pattern Controller API SRV etcd GET https://127.0.0.1:65048/api/v1/namespaces/default/pods?limit=500 C 1 Get(“/registry/pods/default”).WithPrefix() A 2 RangeResponse { "header": { "cluster_id": 14358680983224840000, "member_id": 1033796535975940100, "revision": 3788, "raU_term": 2 }, "kvs": [...] } E 3 Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"resourceVersion":"3788"},"items":[...] A 4 

32 (c) Michael Gasch 2023 @embano1 Kubernetes ListerWatcher: WATCH Phase State Watch Pattern Controller API SRV etcd GET h]ps://127.0.0.1:65048/api/v1/namespaces/default/pods?resourceVersion=3788&watch=true C 5 Watch(“/registry/pods/default”).WithPrefix().WithRev(3788+1) A 6 WatchResponse { "Header": { "cluster_id": 14358680983224840000, "member_id": 1033796535975940100, "revision": 4067, "raft_term": 2 }, "Events": [{type: 0, kv: {"key":"/registry/pods/default/vcsim-7c578468cc-j2d6p"},"value":"...", ”create_revision”:2005,"mod_revision":4067,...] } E 7 Response Body: {{"type":"MODIFIED","object":{"apiVersion":"v1","kind":"Pod","metadata": {“name”:”vcsim-7c578468cc-j2d6p”,"resourceVersion":”4067",...},"spec": {...}} A 8 

33 (c) Michael Gasch 2023 @embano1 • Can a Controller miss Events e.g., during Downtime? • How to handle duplicate Events, such as when re-LISTING due to transient Network Errors? • How to reconcile Changes on external Resources that don’t support a WATCH mechanism? Considera&ons State Watch Pa-ern ! 

Recap 

35 (c) Michael Gasch 2023 @embano1 • Kubernetes is a Distributed System • Building and operabng Distributed Systems is hard • Pagerns decompose complex Problems into understandable and reusable Solubons Recap 1/2 

36 (c) Michael Gasch 2023 @embano1 • Closed Feedback (Control) Loops provide Boundaries • Choreography and event-driven Integrabon unlock Extensibility and Autonomy • At the Cost of added Developer Complexity (Asynchrony and Eventual Consistency) • Consistent (replicated) Core, State Watch and Versioned Values for Durability, Availability, Scalability, and Consistency Recap 2/2 

37 (c) Michael Gasch 2023 @embano1 THANK YOU !" ! @embano1 " www.mgasch.com