Share
Embed
Design and Patterns of Distributed Systems
Talk for a introductory lecture to a computer science class, using Kubernetes to link theory with a practical example.
Michael Gasch
June 08, 2023
Transcript
37 slides
-
Design and Pa+erns of Distributed Systems Using Kubernetes as a
real-world Example Michael Gasch (June 2023) -
2 (c) Michael Gasch 2023 @embano1 about://me • First Computer
1992 (C-64) • IHK Applied Computer Science/Max-Planck Society (-2007) • Systems Engineer Dell (-2015) • Engineer and Research VMware Office of the CTO (-2022) • Product Manager AWS EventBridge (2022-) • Self-taught Developer (Golang) • Public Speaking and Blogging (@embano1 www.mgasch.com) • ! and " -
4 (c) Michael Gasch 2023 @embano1 In 2013 Docker changed
the IT World Credit: docker.com -
5 (c) Michael Gasch 2023 @embano1 Kubernetes won the Container
OrchestraAon War -
6 (c) Michael Gasch 2023 @embano1 Overview Kubernetes Architecture Control
Plane API Server etcd Controller Manager Scheduler … Access REST API SDKs Web UI kubectl Workers Kubelet Kubelet Kubelet Pod Pod Pod -
8 (c) Michael Gasch 2023 @embano1 Design Considerations Kubernetes Architecture
Control Plane API Server etcd Controller Manager Scheduler … Access REST API SDKs Web UI kubectl Workers Kubelet Kubelet Kubelet Pod Pod Pod Availability Scalability Resiliency Security Flow Control Extensibility Observability Durability Consistency Deployment Versioning Responsiveness Programming Model Open Source Recovery Communication -
11 (c) Michael Gasch 2023 @embano1 • Common Definition: “A
Solution to a Problem in a Context” • Applicable in lots of different Situations • Patterns are Solutions • Why as well as how • Code Examples Defini&on Patterns Source: https://martinfowler.com/articles/writingPatterns.html -
12 (c) Michael Gasch 2023 @embano1 • Consistent Core •
Control (Feedback) Loops* • Idempotent Receiver • Leader-Follower • Quorum • Replicated Log • State Watch • Versioned Value Examples Patterns Source: h;ps://mar<nfowler.com/ar<cles/pa;erns-of-distributed-systems/ -
14 (c) Michael Gasch 2023 @embano1 Kubernetes Architecture Control Plane
API Server etcd Controller Manager Scheduler … Access REST API SDKs Web UI kubectl Workers Kubelet Kubelet Kubelet Pod Pod Pod -
15 (c) Michael Gasch 2023 @embano1 Control Loops everywhere Kubernetes
Architecture Control Plane API Server Workers = Control Loop -
16 (c) Michael Gasch 2023 @embano1 Commands vs Events Kubernetes
Architecture Commands Events • Requests (intent) to do something • Named in the impera)ve, e.g. “CREATE” • Can be rejected • Higher coupling between sender and owner • Typically used in synchronous 1-to-1 request/response communicaEon • Something that has happened (a fact) • Named in past tense, e.g. “CREATED” • Cannot (semantically) be rejected by receiver • Lowest coupling between sender and owner • Asynchronous 1-to-many communication, e.g. publish/subscribe -
17 (c) Michael Gasch 2023 @embano1 Request Flow Kubernetes Architecture
API Server REST REST Decoding Conversion & Defaulting Admission Persistency (etcd) … WATCH POST $ kubectl create –f my_replicaset.yaml apiVersion: extensions/v1beta1 kind: ReplicaSet spec: replicas: 2 Commands Events EVENT -
18 (c) Michael Gasch 2023 @embano1 Inside the Control Loop
Kubernetes Architecture Observe Analyze Act apiVersion: extensions/v1beta1 kind: ReplicaSet spec: replicas: 2 desired := getDesiredState() current := getCurrentState() diff := desired – current if diff < 0 { deletePods() } if diff > 0 { createPods() } Command Event (Edge-Triggered) Event (Level-Triggered) -
21 (c) Michael Gasch 2023 @embano1 Asynchronous IntegraGon Kubernetes Architecture
API Server CREATE apiVersion: extensions/v1beta1 kind: ReplicaSet spec: replicas: 2 CREATE Pod ReplicaSet CREATED ReplicaSet Controller BIND Pod Pod CREATED Scheduler Kubelet Pod BOUND UPDATE Pod (“running”) Time Command Event -
23 • Single Responsible Principle • Decoupling via event-driven Messaging
• No central Coordinator A different Mindset WriAng Controllers Autonomous Processes -
24 • Eventual consistent by Design • Don’t rely on
(assume) Order • Single Responsible Principle • Decoupling via event-driven Messaging • No central Coordinator A different Mindset Writing Controllers Autonomous Processes Concurrency & Asynchrony -
25 • API server (etcd) is the Source of Truth*
• In-memory Cache via Reconciliation • Eventual consistent by Design • Don’t rely on (assume) Order • Single Responsible Principle • Decoupling via event-driven Messaging • No central Coordinator A different Mindset Writing Controllers Autonomous Processes Concurrency & Asynchrony Stateless over Stateful -
26 • Things will go wrong (crash) • No shared
(wall) Clock • Anticipate Effects on the Rest of the System • API server (etcd) is the Source of Truth* • In-memory Cache via Reconciliation • Eventual consistent by Design • Don’t rely on (assume) Order • Single Responsible Principle • Decoupling via event-driven Messaging • No central Coordinator A different Mindset Writing Controllers Autonomous Processes Concurrency & Asynchrony Stateless over Stateful Defensive Programming -
27 • Delivery and Processing Guarantees only within Kubernetes •
Things will go wrong (crash) • No shared (wall) Clock • Anticipate Effects on the Rest of the System • API server (etcd) is the Source of Truth* • In-memory Cache via Reconciliation • Eventual consistent by Design • Don’t rely on (assume) Order • Single Responsible Principle • Decoupling via event-driven Messaging • No central Coordinator A different Mindset Writing Controllers Autonomous Processes Concurrency & Asynchrony Stateless over Stateful Side Effects Defensive Programming -
29 (c) Michael Gasch 2023 @embano1 Clients are interested in
changes to the specific values on the server. It's difficult for clients to structure their logic if they need to poll the server conbnuously to look for changes. If clients open too many connecbons to the server for watching changes, it can overwhelm the server. Problem State Watch Pattern -
30 (c) Michael Gasch 2023 @embano1 Allow clients to register
their interest with the server for specific state changes. The server nobfies the interested clients when state changes happen. The client maintains a Single Socket Channel with the server. The server sends state change nobficabons on this channel. Solution State Watch Pattern -
31 (c) Michael Gasch 2023 @embano1 Kubernetes ListerWatcher: LIST Phase
State Watch Pattern Controller API SRV etcd GET https://127.0.0.1:65048/api/v1/namespaces/default/pods?limit=500 C 1 Get(“/registry/pods/default”).WithPrefix() A 2 RangeResponse { "header": { "cluster_id": 14358680983224840000, "member_id": 1033796535975940100, "revision": 3788, "raU_term": 2 }, "kvs": [...] } E 3 Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"resourceVersion":"3788"},"items":[...] A 4 -
32 (c) Michael Gasch 2023 @embano1 Kubernetes ListerWatcher: WATCH Phase
State Watch Pattern Controller API SRV etcd GET h]ps://127.0.0.1:65048/api/v1/namespaces/default/pods?resourceVersion=3788&watch=true C 5 Watch(“/registry/pods/default”).WithPrefix().WithRev(3788+1) A 6 WatchResponse { "Header": { "cluster_id": 14358680983224840000, "member_id": 1033796535975940100, "revision": 4067, "raft_term": 2 }, "Events": [{type: 0, kv: {"key":"/registry/pods/default/vcsim-7c578468cc-j2d6p"},"value":"...", ”create_revision”:2005,"mod_revision":4067,...] } E 7 Response Body: {{"type":"MODIFIED","object":{"apiVersion":"v1","kind":"Pod","metadata": {“name”:”vcsim-7c578468cc-j2d6p”,"resourceVersion":”4067",...},"spec": {...}} A 8 -
33 (c) Michael Gasch 2023 @embano1 • Can a Controller
miss Events e.g., during Downtime? • How to handle duplicate Events, such as when re-LISTING due to transient Network Errors? • How to reconcile Changes on external Resources that don’t support a WATCH mechanism? Considera&ons State Watch Pa-ern ! -
35 (c) Michael Gasch 2023 @embano1 • Kubernetes is a
Distributed System • Building and operabng Distributed Systems is hard • Pagerns decompose complex Problems into understandable and reusable Solubons Recap 1/2 -
36 (c) Michael Gasch 2023 @embano1 • Closed Feedback (Control)
Loops provide Boundaries • Choreography and event-driven Integrabon unlock Extensibility and Autonomy • At the Cost of added Developer Complexity (Asynchrony and Eventual Consistency) • Consistent (replicated) Core, State Watch and Versioned Values for Durability, Availability, Scalability, and Consistency Recap 2/2 -
37 (c) Michael Gasch 2023 @embano1 THANK YOU !" !
@embano1 " www.mgasch.com