Declarative Cloud DB Service Using Kubernetes

Woohyung Han (LINE Plus / Cloud Service Team / Software Engineer)

https://tech-verse.me/ja/sessions/67
https://tech-verse.me/en/sessions/67
https://tech-verse.me/ko/sessions/67

Transcript

2. Speaker - Role: Develop/Operate DB as a Service - Cloud Service Team WooHyung Han
3. Agenda - What/Why Declarative System for DBaaS? - Technical Details - How it works? - Conclusion
4. Background What/Why Declarative System?
5. DB as a Service at LINE Provided DB Types 3 The number of DBs 8,500+ The number of VMs 37,000+
6. Many VMs, Many Failures Especially exhausting on DBaaS
7. webserver DB Instance Local Storage Storage VM VM VM VM webserver webserver DB Instance Local Storage VM DB Instance Local Storage VM Usual WebService DB DBaaS operating costs
8. Our Solution: Declarative System
9. Imperative vs. Declarative System .BOVBM
   %SJWJOH "VUPOPNPVT
   %SJWJOH 1SPWJEFT
   WBSJPVT
   DPNNBOET
   6TFS
   EFUFSNJOFT
   UIF
   DVSSFOU
   TUBUF
   BOE
   FYFDVUFT %FDMBSF
   UIF
   EFTUJOBUJPO
   5IF
   TZTUFN
   EFUFSNJOFT
   BOE
   BDIJFWFT
   JUT
   QVSQPTF
10. Imperative vs. Declarative System .BOVBM
    %SJWJOH "VUPOPNPVT
    %SJWJOH 1SPWJEFT
    WBSJPVT
    DPNNBOET
    6TFS
    EFUFSNJOFT
    UIF
    DVSSFOU
    TUBUF
    BOE
    FYFDVUFT %FDMBSF
    UIF
    EFTUJOBUJPO
    5IF
    TZTUFN
    EFUFSNJOFT
    BOE
    BDIJFWFT
    JUT
    QVSQPTF
11. Imperative vs. Declarative System .BOVBM
    %SJWJOH "VUPOPNPVT
    %SJWJOH 1SPWJEFT
    WBSJPVT
    DPNNBOET
    6TFS
    EFUFSNJOFT
    UIF
    DVSSFOU
    TUBUF
    BOE
    FYFDVUFT %FDMBSF
    UIF
    EFTUJOBUJPO
    5IF
    TZTUFN
    EFUFSNJOFT
    BOE
    BDIJFWFT
    JUT
    QVSQPTF
12. How to build Declarative System Kubernetes 0QFO
    TPVSDF
    QMBUGPSN
    GPS
    NBOBHJOH
    DPOUBJOFSJ[FE
    XPSLMPBET
    CZ
    EFDMBSBUJWF
    DPOpHVSBUJPO
13. Why Virtual Machine, not containers? - multi-cluster is one solution, but too much complexity Kubernetes is SPOF(Single Point of Failure) Container management in Kubernetes is not useful for DBaaS - Can not use pod autoscaling, pod scheduling. - Statefulset has limitations in extension. Already providing VM reliably/on large scale in LINE. - Can be trusted - There are legacy DBs we want to cover without data migration.
14. Why Virtual Machine, not containers? - multi-cluster is one solution, but too much complexity Kubernetes is SPOF(Single Point of Failure) Container management in Kubernetes is not useful for DBaaS - Can not use pod autoscaling, pod scheduling. - Statefulset has limitations in extension. Already providing VM reliably/on large scale in LINE. - Can be trusted - There are legacy DBs we want to cover without data migration.
15. Technical Detail
16. Declarative VM kind: VM spec: vCPU: 2 memory: 8GB process: - dbaas-agent status: dbaas-agent: ok dbaas-agent VM Operator VM VM Custom Resource Create VM Watch
17. Declarative VM kind: VM spec: vCPU: 2 memory: 8GB process: - dbaas-agent status: dbaas-agent: ok dbaas-agent VM Operator VM VM Custom Resource Create VM Watch
18. Declarative VM kind: VM spec: vCPU: 2 memory: 8GB process: - dbaas-agent status: dbaas-agent: ok dbaas-agent VM Operator VM VM Custom Resource Create VM Watch
19. Declarative VM kind: VM spec: vCPU: 2 memory: 8GB process: - dbaas-agent status: dbaas-agent: ok dbaas-agent VM Operator VM VM Custom Resource Create VM Watch If Fail: recreate
20. Declarative DB Instance dbaas-agent DB command DB VM Operator VM VM Custom Resource kind: DB Instance spec: maxclient: 3000 timeout: 5s DB Instance Custom Resource Watch
21. Declarative DB Instance dbaas-agent DB command DB VM Operator VM VM Custom Resource kind: DB Instance spec: maxclient: 3000 timeout: 5s DB Instance Custom Resource Watch
22. Declarative DB Instance dbaas-agent DB command DB VM Operator VM VM Custom Resource kind: DB Instance spec: maxclient: 3000 timeout: 5s DB Instance Custom Resource Watch
23. Declarative DB Instance dbaas-agent DB command DB VM Operator VM VM Custom Resource kind: DB Instance spec: maxclient: 3000 timeout: 5s -> 1s DB Instance Custom Resource Watch Current DB Config maxclient: 3000 timeout: 5s
24. Declarative DB Instance dbaas-agent set timeout as 1s DB VM Operator VM VM Custom Resource kind: DB Instance spec: maxclient: 3000 timeout: 1s DB Instance Custom Resource Watch Current DB Config maxclient: 3000 timeout: 5s -> 1s
25. Declarative DB Instance dbaas-agent DB VM Operator VM VM Custom Resource kind: DB Instance spec: maxclient: 3000 timeout: 1s DB Instance Custom Resource Watch Current DB Config maxclient: 3000 timeout: 1s
26. Declarative DB Instance dbaas-agent DB VM Operator VM VM Custom Resource kind: DB Instance spec: maxclient: 3000 timeout: 1s DB Instance Custom Resource Watch Current DB Config maxclient: nil timeout: nil set timeout=1s, maxclient=3000
27. Declarative DB Instance dbaas-agent DB VM Operator VM VM Custom Resource kind: DB Instance spec: maxclient: 3000 timeout: 1s DB Instance Custom Resource Watch Current DB Config maxclient: 3000 timeout: 1s
28. Declarative DB Service 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM
29. Declarative DB Service 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM
30. Overall Architecture 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM No extra DB No Message Queue Kubernetes as a framework
31. Overall Architecture 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM No extra DB No Message Queue Kubernetes as a framework
32. dbaas-agent DB command DB VM kind: DB Instance spec: maxclient: 3000 timeout: 5s DB Instance Custom Resource Watch 1VM = 1Controller One Controller For Each VM
33. Too Many Controllers for API Server? Kubernetes API Server dbaas-agent (controller) dbaas-agent (controller) dbaas-agent (controller) VM dbaas-agent (controller) dbaas-agent (controller) dbaas-agent (controller) VM dbaas-agent (controller) dbaas-agent (controller) dbaas-agent (controller) VM dbaas-agent (controller) dbaas-agent (controller) dbaas-agent (controller) VM .... DB Service 1 DB Service 2 DB Service 3 DB Service 4 Watch
34. Scaling Goal Large system is important. but... • failures in too large clusters can lead to large data losses. • require too much complexity to create infinitely scalable systems. Kubernetes scaling goal for node count : 5,000 nodes Our Goal: Handle 20,000+ VMs per cluster
35. dbaas-agent VM kind: DB Instance spec: maxclient: 3000 timeout: 5s Watch ETCD Kubernetes API Server Watch Some Optimization kind: DB Instance kind: DB Instance # of VM = 1K Watch Count per VM = 1K Total Watch Count = (# of VM) * (Watch Count per VM) = 1K * 1K = 1M
36. dbaas-agent VM kind: DB Instance spec: maxclient: 3000 timeout: 5s Watch with field selector ETCD Kubernetes API Server Watch Some Optimization kind: DB Instance kind: DB Instance name: my-vm namespace: my-ns
37. dbaas-agent VM kind: DB Instance spec: maxclient: 3000 timeout: 5s Watch with field selector ETCD Kubernetes API Server Watch by kind/namespace Some Optimization kind: DB Instance name: my-vm namespace: my-ns DB Instance/my-ns Don't use one global namespace User Namespace(logical) == Kubernetes Namespace # of Namespaces = 100 # of VM per namespace = 10 Total Watch Count = (# of Namespaces) * (Watch Count per VM) * (# of VM per namespace) = 100 * 10 * 10 = 10K
38. POC Test Setup API Server Count 5 API Server Specification 2vCPU, 8GB RAM The number of controllers 20,000 The number of Custom Resource 100,000 Send a Custom Resource modification request
39. POC Result 10ops/s 20ops/s 40ops/s CPU Usage of Kubernetes API Server
40. Result Features/how it works?
41. Example Features - Scale-out - Change DB Parameters - Node Replacement - Auto Healing
42. Scale-Out
43. Scale-Out DB Service 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3->4 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM User declares vmCount as 4
44. Scale-Out DB Service 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 4 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM DBaaS Manager creates one more CR
45. Scale-Out DB Service 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 4 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM VM Operator and dbaas-agent do their job
46. Change DB Parameter
47. Change DB Parameter 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s->1s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s->1s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM User declares timeout as 1s
48. Change DB Parameter 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 1s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s->1s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM DBaaS Manager sync DB config
49. Change DB Parameter 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 1s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 1s … dbaas-agent set config 1s DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM dbbas-agent sends DB commands
50. Manual Node Replacement
51. Node Replacement 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM The administrator deletes the custom resource of the node to be replaced.
52. Node Replacement 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM Declared vmCount (3) != Actual number of VMs (2) DBaaS Manager creates VM CR.
53. Node Replacement 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM VM Operator and dbaas-agent do their job
54. Auto Healing
55. Auto-Healing 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM VM Maintenance Event Delete VM CR
56. Auto-Healing 2vCPU 8GB RAM … 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … maxclient: 3000 timeout: 5s … kind: DB Service spec: vmCount: 3 vmConfig: vCPU: 2 memory: 8GB dbConfig: maxclient: 3000 timeout: 5s User 2vCPU 8GB RAM … maxclient: 3000 timeout: 5s … dbaas-agent DB command DB watch VM Operator DBaaS Manager VM create create VM Custom Resource DB Instance Custom Resource Create VM VM Operator and dbaas-agent do their job DBaaS Manager creates CRs
57. Conclusion
58. Summary - Easy to deploy - No Extra component(DB, Message Queue) Using Kubernetes as a framework for building declarative system Declarative DB as a Service - Robustness - Automate manual operations What's Next?: Make All DBaaS as a Declarative Form
59. Thank you