Share
Embed
Declarative Cloud DB Service Using Kubernetes
Woohyung Han (LINE Plus / Cloud Service Team / Software Engineer)
https://tech-verse.me/ja/sessions/67
https://tech-verse.me/en/sessions/67
https://tech-verse.me/ko/sessions/67
Tech-Verse2022
November 17, 2022
Transcript
59 slides
-
Speaker - Role: Develop/Operate DB as a Service - Cloud
Service Team WooHyung Han -
Agenda - What/Why Declarative System for DBaaS? - Technical Details
- How it works? - Conclusion -
DB as a Service at LINE Provided DB Types 3
The number of DBs 8,500+ The number of VMs 37,000+ -
webserver DB Instance Local Storage Storage VM VM VM VM
webserver webserver DB Instance Local Storage VM DB Instance Local Storage VM Usual WebService DB DBaaS operating costs -
Why Virtual Machine, not containers? - multi-cluster is one solution,
but too much complexity Kubernetes is SPOF(Single Point of Failure) Container management in Kubernetes is not useful for DBaaS - Can not use pod autoscaling, pod scheduling. - Statefulset has limitations in extension. Already providing VM reliably/on large scale in LINE. - Can be trusted - There are legacy DBs we want to cover without data migration. -
Why Virtual Machine, not containers? - multi-cluster is one solution,
but too much complexity Kubernetes is SPOF(Single Point of Failure) Container management in Kubernetes is not useful for DBaaS - Can not use pod autoscaling, pod scheduling. - Statefulset has limitations in extension. Already providing VM reliably/on large scale in LINE. - Can be trusted - There are legacy DBs we want to cover without data migration. -
Declarative VM kind: VM spec: vCPU: 2 memory: 8GB process:
- dbaas-agent status: dbaas-agent: ok dbaas-agent VM Operator VM VM Custom Resource Create VM Watch -
Declarative VM kind: VM spec: vCPU: 2 memory: 8GB process:
- dbaas-agent status: dbaas-agent: ok dbaas-agent VM Operator VM VM Custom Resource Create VM Watch -
Declarative VM kind: VM spec: vCPU: 2 memory: 8GB process:
- dbaas-agent status: dbaas-agent: ok dbaas-agent VM Operator VM VM Custom Resource Create VM Watch -
Declarative VM kind: VM spec: vCPU: 2 memory: 8GB process:
- dbaas-agent status: dbaas-agent: ok dbaas-agent VM Operator VM VM Custom Resource Create VM Watch If Fail: recreate -
Declarative DB Instance dbaas-agent DB command DB VM Operator VM
VM Custom Resource kind: DB Instance spec: maxclient: 3000 timeout: 5s DB Instance Custom Resource Watch -
Declarative DB Instance dbaas-agent DB command DB VM Operator VM
VM Custom Resource kind: DB Instance spec: maxclient: 3000 timeout: 5s DB Instance Custom Resource Watch -
Declarative DB Instance dbaas-agent DB command DB VM Operator VM
VM Custom Resource kind: DB Instance spec: maxclient: 3000 timeout: 5s DB Instance Custom Resource Watch -
Declarative DB Instance dbaas-agent DB command DB VM Operator VM
VM Custom Resource kind: DB Instance spec: maxclient: 3000 timeout: 5s -> 1s DB Instance Custom Resource Watch Current DB Config maxclient: 3000 timeout: 5s -
Declarative DB Instance dbaas-agent set timeout as 1s DB VM
Operator VM VM Custom Resource kind: DB Instance spec: maxclient: 3000 timeout: 1s DB Instance Custom Resource Watch Current DB Config maxclient: 3000 timeout: 5s -> 1s -
Declarative DB Instance dbaas-agent DB VM Operator VM VM Custom
Resource kind: DB Instance spec: maxclient: 3000 timeout: 1s DB Instance Custom Resource Watch Current DB Config maxclient: 3000 timeout: 1s -
Declarative DB Instance dbaas-agent DB VM Operator VM VM Custom
Resource kind: DB Instance spec: maxclient: 3000 timeout: 1s DB Instance Custom Resource Watch Current DB Config maxclient: nil timeout: nil set timeout=1s, maxclient=3000 -
Declarative DB Instance dbaas-agent DB VM Operator VM VM Custom
Resource kind: DB Instance spec: maxclient: 3000 timeout: 1s DB Instance Custom Resource Watch Current DB Config maxclient: 3000 timeout: 1s -
Declarative DB Service 2vCPU 8GB RAM … 2vCPU 8GB RAM
… maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM -
Declarative DB Service 2vCPU 8GB RAM … 2vCPU 8GB RAM
… maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM -
Overall Architecture 2vCPU 8GB RAM … 2vCPU 8GB RAM …
maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM No extra DB No Message Queue Kubernetes as a framework -
Overall Architecture 2vCPU 8GB RAM … 2vCPU 8GB RAM …
maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM No extra DB No Message Queue Kubernetes as a framework -
dbaas-agent DB command DB VM kind: DB Instance spec: maxclient:
3000 timeout: 5s DB Instance Custom Resource Watch 1VM = 1Controller One Controller For Each VM -
Too Many Controllers for API Server? Kubernetes API Server dbaas-agent
(controller) dbaas-agent (controller) dbaas-agent (controller) VM dbaas-agent (controller) dbaas-agent (controller) dbaas-agent (controller) VM dbaas-agent (controller) dbaas-agent (controller) dbaas-agent (controller) VM dbaas-agent (controller) dbaas-agent (controller) dbaas-agent (controller) VM .... DB Service 1 DB Service 2 DB Service 3 DB Service 4 Watch -
Scaling Goal Large system is important. but... • failures in
too large clusters can lead to large data losses. • require too much complexity to create infinitely scalable systems. Kubernetes scaling goal for node count : 5,000 nodes Our Goal: Handle 20,000+ VMs per cluster -
dbaas-agent VM kind: DB Instance spec: maxclient: 3000 timeout: 5s
Watch ETCD Kubernetes API Server Watch Some Optimization kind: DB Instance kind: DB Instance # of VM = 1K Watch Count per VM = 1K Total Watch Count = (# of VM) * (Watch Count per VM) = 1K * 1K = 1M -
dbaas-agent VM kind: DB Instance spec: maxclient: 3000 timeout: 5s
Watch with field selector ETCD Kubernetes API Server Watch Some Optimization kind: DB Instance kind: DB Instance name: my-vm namespace: my-ns -
dbaas-agent VM kind: DB Instance spec: maxclient: 3000 timeout: 5s
Watch with field selector ETCD Kubernetes API Server Watch by kind/namespace Some Optimization kind: DB Instance name: my-vm namespace: my-ns DB Instance/my-ns Don't use one global namespace User Namespace(logical) == Kubernetes Namespace # of Namespaces = 100 # of VM per namespace = 10 Total Watch Count = (# of Namespaces) * (Watch Count per VM) * (# of VM per namespace) = 100 * 10 * 10 = 10K -
POC Test Setup API Server Count 5 API Server Specification
2vCPU, 8GB RAM The number of controllers 20,000 The number of Custom Resource 100,000 Send a Custom Resource modification request -
Example Features - Scale-out - Change DB Parameters - Node
Replacement - Auto Healing -
Scale-Out DB Service 2vCPU 8GB RAM … 2vCPU 8GB RAM
… maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3->4 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM User declares vmCount as 4 -
Scale-Out DB Service 2vCPU 8GB RAM … 2vCPU 8GB RAM
… maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 4 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM DBaaS Manager creates one more CR -
Scale-Out DB Service 2vCPU 8GB RAM … 2vCPU 8GB RAM
… maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 4 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM VM Operator and dbaas-agent do their job -
Change DB Parameter 2vCPU 8GB RAM … 2vCPU 8GB RAM
… maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s->1s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s->1s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM User declares timeout as 1s -
Change DB Parameter 2vCPU 8GB RAM … 2vCPU 8GB RAM
… maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 1s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s->1s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM DBaaS Manager sync DB config -
Change DB Parameter 2vCPU 8GB RAM … 2vCPU 8GB RAM
… maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 1s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 1s … dbaas-agent set config 1s DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM dbbas-agent sends DB commands -
Node Replacement 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s
… maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM The administrator deletes the custom resource of the node to be replaced. -
Node Replacement 2vCPU 8GB RAM … 2vCPU 8GB RAM …
maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM Declared vmCount (3) != Actual number of VMs (2) DBaaS Manager creates VM CR. -
Node Replacement 2vCPU 8GB RAM … 2vCPU 8GB RAM …
maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM VM Operator and dbaas-agent do their job -
Auto-Healing 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient:
3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM VM Maintenance Event Delete VM CR -
Auto-Healing 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient:
3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM VM Operator and dbaas-agent do their job DBaaS Manager creates CRs -
Summary - Easy to deploy - No Extra component(DB, Message
Queue) Using Kubernetes as a framework for building declarative system Declarative DB as a Service - Robustness - Automate manual operations What's Next?: Make All DBaaS as a Declarative Form