Declarative Cloud DB Service Using Kubernetes

by Tech-Verse2022

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

Speaker - Role: Develop/Operate DB as a Service - Cloud Service Team WooHyung Han

Slide 3

Slide 3 text

Agenda - What/Why Declarative System for DBaaS? - Technical Details - How it works? - Conclusion

Slide 4

Slide 4 text

Background What/Why Declarative System?

Slide 5

Slide 5 text

DB as a Service at LINE Provided DB Types 3 The number of DBs 8,500+ The number of VMs 37,000+

Slide 6

Slide 6 text

Many VMs, Many Failures Especially exhausting on DBaaS

Slide 7

Slide 7 text

webserver DB Instance Local Storage Storage VM VM VM VM webserver webserver DB Instance Local Storage VM DB Instance Local Storage VM Usual WebService DB DBaaS operating costs

Slide 8

Slide 8 text

Our Solution: Declarative System

Slide 9

Slide 9 text

Imperative vs. Declarative System .BOVBM%SJWJOH "VUPOPNPVT%SJWJOH 1SPWJEFTWBSJPVTDPNNBOET 6TFSEFUFSNJOFTUIFDVSSFOUTUBUFBOEFYFDVUFT %FDMBSFUIFEFTUJOBUJPO 5IFTZTUFNEFUFSNJOFTBOEBDIJFWFTJUTQVSQPTF

Slide 10

Slide 10 text

Imperative vs. Declarative System .BOVBM%SJWJOH "VUPOPNPVT%SJWJOH 1SPWJEFTWBSJPVTDPNNBOET 6TFSEFUFSNJOFTUIFDVSSFOUTUBUFBOEFYFDVUFT %FDMBSFUIFEFTUJOBUJPO 5IFTZTUFNEFUFSNJOFTBOEBDIJFWFTJUTQVSQPTF

Slide 11

Slide 11 text

Imperative vs. Declarative System .BOVBM%SJWJOH "VUPOPNPVT%SJWJOH 1SPWJEFTWBSJPVTDPNNBOET 6TFSEFUFSNJOFTUIFDVSSFOUTUBUFBOEFYFDVUFT %FDMBSFUIFEFTUJOBUJPO 5IFTZTUFNEFUFSNJOFTBOEBDIJFWFTJUTQVSQPTF

Slide 12

Slide 12 text

How to build Declarative System Kubernetes 0QFOTPVSDFQMBUGPSNGPSNBOBHJOHDPOUBJOFSJ[FEXPSLMPBETCZEFDMBSBUJWFDPOpHVSBUJPO

Slide 13

Slide 13 text

Why Virtual Machine, not containers? - multi-cluster is one solution, but too much complexity Kubernetes is SPOF(Single Point of Failure) Container management in Kubernetes is not useful for DBaaS - Can not use pod autoscaling, pod scheduling. - Statefulset has limitations in extension. Already providing VM reliably/on large scale in LINE. - Can be trusted - There are legacy DBs we want to cover without data migration.

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Technical Detail

Slide 16

Slide 16 text

Declarative VM kind: VM spec: vCPU: 2 memory: 8GB process: - dbaas-agent status: dbaas-agent: ok dbaas-agent VM Operator VM VM Custom Resource Create VM Watch

Slide 17

Slide 17 text

Declarative VM kind: VM spec: vCPU: 2 memory: 8GB process: - dbaas-agent status: dbaas-agent: ok dbaas-agent VM Operator VM VM Custom Resource Create VM Watch

Slide 18

Slide 18 text

Declarative VM kind: VM spec: vCPU: 2 memory: 8GB process: - dbaas-agent status: dbaas-agent: ok dbaas-agent VM Operator VM VM Custom Resource Create VM Watch

Slide 19

Slide 19 text

Declarative VM kind: VM spec: vCPU: 2 memory: 8GB process: - dbaas-agent status: dbaas-agent: ok dbaas-agent VM Operator VM VM Custom Resource Create VM Watch If Fail: recreate

Slide 20

Slide 20 text

Declarative DB Instance dbaas-agent DB command DB VM Operator VM VM Custom Resource kind: DB Instance spec: maxclient: 3000 timeout: 5s DB Instance Custom Resource Watch

Slide 21

Slide 21 text

Declarative DB Instance dbaas-agent DB command DB VM Operator VM VM Custom Resource kind: DB Instance spec: maxclient: 3000 timeout: 5s DB Instance Custom Resource Watch

Slide 22

Slide 22 text

Declarative DB Instance dbaas-agent DB command DB VM Operator VM VM Custom Resource kind: DB Instance spec: maxclient: 3000 timeout: 5s DB Instance Custom Resource Watch

Slide 23

Slide 23 text

Declarative DB Instance dbaas-agent DB command DB VM Operator VM VM Custom Resource kind: DB Instance spec: maxclient: 3000 timeout: 5s -> 1s DB Instance Custom Resource Watch Current DB Config maxclient: 3000 timeout: 5s

Slide 24

Slide 24 text

Declarative DB Instance dbaas-agent set timeout as 1s DB VM Operator VM VM Custom Resource kind: DB Instance spec: maxclient: 3000 timeout: 1s DB Instance Custom Resource Watch Current DB Config maxclient: 3000 timeout: 5s -> 1s

Slide 25

Slide 25 text

Declarative DB Instance dbaas-agent DB VM Operator VM VM Custom Resource kind: DB Instance spec: maxclient: 3000 timeout: 1s DB Instance Custom Resource Watch Current DB Config maxclient: 3000 timeout: 1s

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Declarative DB Service 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Overall Architecture 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM No extra DB No Message Queue Kubernetes as a framework

Slide 31

Slide 31 text

Slide 32

Slide 32 text

dbaas-agent DB command DB VM kind: DB Instance spec: maxclient: 3000 timeout: 5s DB Instance Custom Resource Watch 1VM = 1Controller One Controller For Each VM

Slide 33

Slide 33 text

Too Many Controllers for API Server? Kubernetes API Server dbaas-agent (controller) dbaas-agent (controller) dbaas-agent (controller) VM dbaas-agent (controller) dbaas-agent (controller) dbaas-agent (controller) VM dbaas-agent (controller) dbaas-agent (controller) dbaas-agent (controller) VM dbaas-agent (controller) dbaas-agent (controller) dbaas-agent (controller) VM .... DB Service 1 DB Service 2 DB Service 3 DB Service 4 Watch

Slide 34

Slide 34 text

Scaling Goal Large system is important. but... • failures in too large clusters can lead to large data losses. • require too much complexity to create infinitely scalable systems. Kubernetes scaling goal for node count : 5,000 nodes Our Goal: Handle 20,000+ VMs per cluster

Slide 35

Slide 35 text

dbaas-agent VM kind: DB Instance spec: maxclient: 3000 timeout: 5s Watch ETCD Kubernetes API Server Watch Some Optimization kind: DB Instance kind: DB Instance # of VM = 1K Watch Count per VM = 1K Total Watch Count = (# of VM) * (Watch Count per VM) = 1K * 1K = 1M

Slide 36

Slide 36 text

dbaas-agent VM kind: DB Instance spec: maxclient: 3000 timeout: 5s Watch with field selector ETCD Kubernetes API Server Watch Some Optimization kind: DB Instance kind: DB Instance name: my-vm namespace: my-ns

Slide 37

Slide 37 text

dbaas-agent VM kind: DB Instance spec: maxclient: 3000 timeout: 5s Watch with field selector ETCD Kubernetes API Server Watch by kind/namespace Some Optimization kind: DB Instance name: my-vm namespace: my-ns DB Instance/my-ns Don't use one global namespace User Namespace(logical) == Kubernetes Namespace # of Namespaces = 100 # of VM per namespace = 10 Total Watch Count = (# of Namespaces) * (Watch Count per VM) * (# of VM per namespace) = 100 * 10 * 10 = 10K

Slide 38

Slide 38 text

POC Test Setup API Server Count 5 API Server Specification 2vCPU, 8GB RAM The number of controllers 20,000 The number of Custom Resource 100,000 Send a Custom Resource modification request

Slide 39

Slide 39 text

POC Result 10ops/s 20ops/s 40ops/s CPU Usage of Kubernetes API Server

Slide 40

Slide 40 text

Result Features/how it works?

Slide 41

Slide 41 text

Example Features - Scale-out - Change DB Parameters - Node Replacement - Auto Healing

Slide 42

Slide 42 text

Scale-Out

Slide 43

Slide 43 text

Scale-Out DB Service 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3->4 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM User declares vmCount as 4

Slide 44

Slide 44 text

Scale-Out DB Service 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 4 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM DBaaS Manager creates one more CR

Slide 45

Slide 45 text

Slide 46

Slide 46 text

Change DB Parameter

Slide 47

Slide 47 text

Change DB Parameter 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s->1s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s->1s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM User declares timeout as 1s

Slide 48

Slide 48 text

Change DB Parameter 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 1s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s->1s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM DBaaS Manager sync DB config

Slide 49

Slide 49 text

Change DB Parameter 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 1s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 1s … dbaas-agent set config 1s DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM dbbas-agent sends DB commands

Slide 50

Slide 50 text

Manual Node Replacement

Slide 51

Slide 51 text

Node Replacement 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM The administrator deletes the custom resource of the node to be replaced.

Slide 52

Slide 52 text

Node Replacement 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM Declared vmCount (3) != Actual number of VMs (2) DBaaS Manager creates VM CR.

Slide 53

Slide 53 text

Slide 54

Slide 54 text

Auto Healing

Slide 55

Slide 55 text

Auto-Healing 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM VM Maintenance Event Delete VM CR

Slide 56

Slide 56 text

Slide 57

Slide 57 text

Conclusion

Slide 58

Slide 58 text

Summary - Easy to deploy - No Extra component(DB, Message Queue) Using Kubernetes as a framework for building declarative system Declarative DB as a Service - Robustness - Automate manual operations What's Next?: Make All DBaaS as a Declarative Form

Slide 59

Slide 59 text

Thank you