Tweet
Tweet

Slide 1

Slide 1 text

© 2023 Lucas Käldström 1 Managing scalable database clusters with the TiDB Operator Lucas Käldström – CNCF Ambassador Mountain View – September 21, 2023

Slide 2

Slide 2 text

© 2023 Lucas Käldström 2 Cloud Native Philosophy: Why Do We Now Design Software the Way We Do? Lucas Käldström – CNCF Ambassador Mountain View – September 21, 2023 or similarly,

Slide 3

Slide 3 text

© 2023 Lucas Käldström 3 $ whoami Lucas Käldström, 1st-year MSc student at Aalto University, Finland CNCF Ambassador, Certified Kubernetes Administrator and Emeritus Kubernetes WG/SIG Lead KubeCon Speaker in Berlin, Austin, Copenhagen, Shanghai, Seattle, San Diego & Valencia KubeCon Keynote Speaker in Barcelona Former Kubernetes approver and subproject owner, active in the OSS community for 7+ years. Worked on e.g. SIG Cluster Lifecycle => kubeadm to GA. Former Weaveworks contractor, Weave Ignite & libgitops author Cloud Native Nordics co-founder & meetup organizer Guild of Automation and Systems Technology corporate relations & CFO

Slide 4

Slide 4 text

© 2023 Lucas Käldström 4 Agenda

Slide 5

Slide 5 text

© 2023 Lucas Käldström 5 Agenda - Database Sysadmin Complexities - Kubernetes Design Architecture - A Sysadmin’s Best Friend: The Operator - The TiDB Operator - Demo Screenshots (not enough time for live demo)

Slide 6

Slide 6 text

© 2023 Lucas Käldström 6 Database Sysadmin Complexities

Slide 7

Slide 7 text

© 2023 Lucas Käldström 7 Why are we here?

Slide 8

Slide 8 text

© 2023 Lucas Käldström 8 Why are we here? Want a database for both transactions processing and analytical processing

Slide 9

Slide 9 text

© 2023 Lucas Käldström 9 What does this require?

Slide 10

Slide 10 text

© 2023 Lucas Käldström 10 What does this require? - Failure Tolerance and Capacity Demand => Multiple Replicas - Multiple Replicas => Consistency Control (Paxos / Raft) - Capacity Demands => Sharding - and much more!

Slide 11

Slide 11 text

© 2023 Lucas Käldström 11 - Multiple Nodes => Need scheduling logic - Consensus Algorithms => We need to take care when: - Scaling: Need some kind of “learner mode” - Upgrading: Avoid killing the consensus leader; give a proper handoff first - Sharding => Nodes have varying set of data, one node doesn’t necessarily all the data - Quickly-changing business requirements => Lots of sysadmin work What does this mean?

Slide 12

Slide 12 text

© 2023 Lucas Käldström 12

Slide 13

Slide 13 text

© 2023 Lucas Käldström 13 ⇒ Every non-trivial system requires non-trivial operations

Slide 14

Slide 14 text

© 2023 Lucas Käldström 14 Required sysadmin work grows faster than scale Business scaling requirement Sysadmin work

Slide 15

Slide 15 text

© 2023 Lucas Käldström 15 We want to find a generic solution

Slide 16

Slide 16 text

© 2023 Lucas Käldström 16 Kubernetes Design Philosophy

Slide 17

Slide 17 text

© 2023 Lucas Käldström 17 Kubernetes Primer - Kubernetes is an open source container orchestration system. - Project to solve sysadmin operational challenges of app orchestration - Already decade old (!), the founding project of CNCF, 80000+ contributors - Runs in all environments from own DC to cloud (even on Raspberry Pis!) - Super extensible system, you can configure literally everything

Slide 18

Slide 18 text

© 2023 Lucas Käldström 18 Based on decades of experience

Slide 19

Slide 19 text

© 2023 Lucas Käldström 19 Node Kubernetes Architecture Single source of truth Raft key-value store Stateful Stateless, declarative and extensible REST API stateless controllers Node Node … these controllers “make stuff happen” <- reconcile ->

Slide 20

Slide 20 text

© 2023 Lucas Käldström 20 Kubernetes: A Control Plane for (any) infrastructure

Slide 21

Slide 21 text

© 2023 Lucas Käldström 21 Kubernetes: A Control Plane for (any) infrastructure = A set of automated controllers with operational knowledge of how to control a target system

Slide 22

Slide 22 text

© 2023 Lucas Käldström 22 Kubernetes: A Control Plane for (any) infrastructure = A set of automated controllers with operational knowledge of how to control a target system Around 45 (!) of them in Kubernetes v1.28

Slide 23

Slide 23 text

© 2023 Lucas Käldström 23 Kubernetes: A Control Plane for (any) infrastructure = A set of automated controllers with operational knowledge of how to control a target system “I know how to efficiently schedule workloads to nodes” “I know how to heal applications that were on failed nodes” “I know how to configure dynamic service discovery”

Slide 24

Slide 24 text

© 2023 Lucas Käldström 24 Kubernetes: A Control Plane for (any) infrastructure = A set of automated controllers with operational knowledge of how to control a target system

Slide 25

Slide 25 text

“Control Through Choreography” All user intent is stored in the API server. Business logic split into controllers making user intent a reality

Slide 26

Slide 26 text

© 2023 Lucas Käldström 26 Why a controller-centric model?

Slide 27

Slide 27 text

Chaos is Inevitable

Slide 28

Slide 28 text

Google Finding: “Failure is the Norm”

Slide 29

Slide 29 text

“deliberately leave significant headroom for workload growth, occasional ‘black swan’ events, load spikes, machine failures, hardware upgrades, and large-scale partial failures (e.g., a power supply bus duct)” Source: (Verma et. al., 2015) Google Finding: “Failure is the Norm”

Slide 30

Slide 30 text

© 2023 Lucas Käldström 30 Entropy: Systems become less ordered Time Entropy Order Start Stop Chaos

Slide 31

Slide 31 text

© 2023 Lucas Käldström 31 Entropy: Putting order to chaos Time Entropy Order Start Stop Chaos Reversing, ordering process

Slide 32

Slide 32 text

© 2023 Lucas Käldström 32 Kubernetes: The dishwasher of servers Time Entropy Order Start Stop Chaos Reversing, ordering process

Slide 33

Slide 33 text

© 2023 Lucas Käldström 33 Defining “order” and “chaos”

Slide 34

Slide 34 text

© 2023 Lucas Käldström 34 WHAT

Slide 35

Slide 35 text

© 2023 Lucas Käldström 35 HOW

Slide 36

Slide 36 text

“If you don’t know where you’re going, any road will take you there”

Slide 37

Slide 37 text

© 2023 Lucas Käldström 37 Key Takeaways a) Systems are inevitably becoming less ordered, thus b) need some periodic corrective action to steer the course towards c) some declared desired state of the system.

Slide 38

Slide 38 text

© 2023 Lucas Käldström 38 A sysadmin’s best friend, the operator

Slide 39

Slide 39 text

= Automated reconcile loops with “human-like” operational knowledge Coined in 2016 by Brandon Phillips, back then at CoreOS Operators: Encode human-like knowledge

Slide 40

Slide 40 text

= Automated reconcile loops with “human-like” operational knowledge Coined in 2016 by Brandon Phillips, back then at CoreOS Operators: Encode human-like knowledge Delegate “repetitive human activities that are devoid of lasting value”

Slide 41

Slide 41 text

© 2023 Lucas Käldström 41 What should an operator do? - Keep infrastructure in control: continuously minimizing drift between the desired and actual state, - Resource scalability: codify and automate “repetitive human activities that are devoid of lasting value”, by encoding domain-specific knowledge, - Monitoring scalability: observe application health, metrics and logs, such that configuration can be adaptively tuned and alerts of any abnormal behavior can be sent seldom but with high importance, and - Knowledge scalability: provide a high-level abstraction interface such that the application can be operated by engineers without the domain-specific knowledge otherwise required

Slide 42

Slide 42 text

© 2023 Lucas Käldström 42 Avoids sysadmin management overload

Slide 43

Slide 43 text

Not: Humans Operating Machines

Slide 44

Slide 44 text

Instead: Humans Operating Automation that in turn Operate Machines

Slide 45

Slide 45 text

© 2023 Lucas Käldström 45 The TiDB Operator

Slide 46

Slide 46 text

© 2023 Lucas Käldström 46 TiDB Operator Capabilities The tidb-operator provides you with TiDB as a Service in your own cluster It features features such as: - Multi-Cluster Creation - Online up- and downgrades - Online up- and downscaling of replicas, even automatically - Automatic failover/self-healing - Dynamic monitoring - Re-configuration of the database - Backup and Restore

Slide 47

Slide 47 text

© 2023 Lucas Käldström 47 Operator fulfils the user’s desires Observe and diff Desired State Source Target System 2 1 2, 6: Actual State 1: Desired State

Slide 48

Slide 48 text

© 2023 Lucas Käldström 48 Operator fulfils the user’s desires Observe and diff Act Desired State Source 3 Target System 2 1 2, 6: Actual State 1: Desired State 4: Action 3: Action Plan 4

Slide 49

Slide 49 text

© 2023 Lucas Käldström 49 Operator fulfils the user’s desires Observe and diff Act Desired State Source 3 Report (Actual State Sink) Target System 2 1 7: Requeue 2, 6: Actual State 1: Desired State 4: Action 3: Action Plan 5: Result 4 5 (6)

Slide 50

Slide 50 text

© 2023 Lucas Käldström 50 Operator fulfils the user’s desires Observe and diff Act Desired State Source 3 Report (Actual State Sink) Target System 2 1 7: Requeue 2, 6: Actual State 1: Desired State 4: Action 3: Action Plan 5: Result 4 5 (6) 7

Slide 51

Slide 51 text

© 2023 Lucas Käldström 51 Demo

Slide 52

Slide 52 text

© 2023 Lucas Käldström 52 Hardened tidb-operator setup In this demo, we will initially configure 3 cloud VMs for TiDB, 3 cloud VMs for PD, and 3 cloud VMs for TiKV. Further, we will 1) install the tidb-operator through the CNCF GitOps engine, Flux 2) set up the monitoring stack (Prometheus, Grafana) to watch performance 3) create one TiDBCluster with the operator 4) apply advanced configuration such as topology and upgrade tuning This demo running on UpCloud, thanks for donating cloud credits for this cause!

Slide 53

Slide 53 text

© 2023 Lucas Käldström 53 Upgrading a cluster with a 60k QPS load In this demo, we will: 1) bump the version number from v7.1.0 to v7.1.1 using a GitHub Pull Request, 2) ⇒ operator upgrades the 3*3-TiDB cluster gracefully, 3) while serving 60k requests per second (without any reconnects!), 4) while monitoring TiDB performance This demo running on UpCloud, thanks for donating cloud credits for this cause!

Slide 54

Slide 54 text

© 2023 Lucas Käldström 54 Architecture 1 2 3 tidb-operator 4 k8s controllers Node 5 0

Slide 55

Slide 55 text

© 2023 Lucas Käldström 55 Step 1: Change desired configuration in GitHub bump to v7.1.1

Slide 56

Slide 56 text

© 2023 Lucas Käldström 56 Step 2: Relax and watch the upgrade let the upgrade do the work!

Slide 57

Slide 57 text

© 2023 Lucas Käldström 57 Operator reconciles desired and actual state

Slide 58

Slide 58 text

© 2023 Lucas Käldström 58 We didn’t skip a beat

Slide 59

Slide 59 text

© 2023 Lucas Käldström 59 Recap

Slide 60

Slide 60 text

© 2023 Lucas Käldström 60 - Manual service discovery (for peers, backup and monitoring) - Manual TLS setup - Manual scaling - Manual version upgrades - Manual re-configuration - Manual disaster recovery What do we **not** have to do? real-life footage of sysadmin not having to run 1002 commands to upgrade the database:

Slide 61

Slide 61 text

Not: Humans Operating Machines

Slide 62

Slide 62 text

Instead: Humans Operating Automation that in turn Operate Machines

Slide 63

Slide 63 text

© 2023 Lucas Käldström 63 Further Reading

Slide 64

Slide 64 text

© 2023 Lucas Käldström 64 Check out my thesis for more details! Available openly on Github: https://github.com/luxas/research CC-BY-SA 4.0 licensed Encoding human-like operational knowledge using declarative Kubernetes operator patterns

Slide 65

Slide 65 text

© 2023 Lucas Käldström 65 Control Theory (Vallery Lancery, QCon, 2018) I have another talk on control theory + declarative APIs = Kubernetes Also check out Vallery Lancery’s great talk on the subject.

Slide 66

Slide 66 text

© 2023 Lucas Käldström 66 Thank you!