通往Cloud Native的神奇之旅: 自製 Kubernetes 工具大揭秘

Speaker: Luther Tsai - LINE Pay Taiwan Server Development Team
Event: iThome Kubernetes Summit

Transcript

2. 通往Cloud Native的神奇之旅
3. 01 02 03 04 Flow Decision & Tech Stack Self-Implemented Tool Outcome & Improvement Pain Points 05 Summary AGENDA
4. 01 02 03 04 Flow Decision & Tech Stack Self-Implemented Tool Outcome & Improvement Pain Points 05 Summary AGENDA Why How What Which
6. LINE Pay Services Payment Service Financial Service Points Service Promotion Service Partner Service Donation Service Merchant Service
7. SECTION 01 WHY
8. 01 Pain Points
9. OA Push with High Traffic Volume Pain Points 10x RPS increase
10. OA Push with High Traffic Volume Pain Points 10x requests increase
11. Legacy applications running on VM Application Scale out / in Self-Healing Blue Green Deployment Server Preparation Pain Points
12. Legacy applications running on VM Application Scale out / in Self-Healing Blue Green Deployment Server Preparation Pain Points
13. Legacy applications running on VM Application Scale out / in Self-Healing Blue Green Deployment Server Preparation Pain Points
14. Legacy applications running on VM Application Scale out / in Self-Healing Blue Green Deployment Server Preparation Pain Points
15. SECTION 02 HOW
16. What do we have? Comply with Regulation Pioneer Application Current CI/CD Inspection How do we start? Disaster Recovery
17. Which kind of resource that we could use What we've got on the table? 2020 2021 2022 2023 App Integration HARBOR
18. Pros & Cons • Infra Team support & Gradually support with Verda Resource integration • In 2020, the latest version on Verda VKS is v1.15 • We could choose the resources that we need by our own • Managed all infra layer by our own
19. • Infra Team support & Gradually support with Verda Resource integration • In 2020, the latest version on Verda VKS is v1.15 • We could choose the resources that we need by our own • Managed all infra layer by our own Both of them are baremetal version of Kubernetes. Not like GKE or EKS on AWS. They both need installation on related resources. (eg. Ingress Controller) Pros & Cons
20. What we've got on the table?
21. What do we have? Comply with Regulation Pioneer Application Current CI/CD Inspection How do we start? Disaster Recovery
22. AS-IS CI/CD flow Developer Trigger deployment Build excutable files by branch Deploy to target runtime servers VMs Trigger Build Upload excutable files Code Push
23. What do we have? Comply with Regulation Pioneer Application Current CI/CD Inspection How do we start? Disaster Recovery
24. AS-IS CI/CD flow Developer Trigger deployment Build excutable files by branch Deploy to target runtime servers VMs Trigger Build Upload excutable files Code Push Centralize CI/CD flow control with regulation compliance RESTRICTED ZONE
25. TO-BE CI/CD flow Developer Trigger deployment Build docker image by branch Proxy Agent Trigger Build Upload deploy script & version tag Code Push Pull runtime image Push runtime image Manipulate Cluster
26. TO-BE CI/CD flow Developer Trigger deployment Build docker image by branch Proxy Agent Trigger Build Code Push 1. Run deploy script 2. Generate YAML 3. Apply on our cluster
27. RESTRICTED ZONE TO-BE CI/CD flow Developer Trigger deployment Build docker image by branch Proxy Agent Trigger Build Code Push
28. What do we have? Comply with Regulation Pioneer Application Current CI/CD Inspection How do we start? Disaster Recovery
29. Disaster Recovery (AS-IS) Developer Trigger deployment Build excutable files by branch Deploy to target runtime servers VMs Trigger Build Upload excutable files Code Push
30. Disaster Recovery (AS-IS) Developer Trigger deployment Build excutable files by branch Replacement VMs Trigger Build Upload excutable files Code Push Re-run the deployment jobs (RESUME)
31. Disaster Recovery (AS-IS) Developer Trigger deployment Build excutable files by branch Replacement VMs Trigger Build Upload excutable files Code Push Takes lot of times & different requirements on each applications Re-run the deployment jobs (RESUME)
32. Disaster Recovery (TO-BE) Developer Trigger deployment Build docker image by branch Proxy Agent Trigger Build Code Push
33. Disaster Recovery (TO-BE) Developer Trigger deployment Build docker image by branch Proxy Agent Trigger Build Code Push 1. Setup the cluster with preserved installation YAML 2. Setup the Proxy Agent Config 3. Re-run the deployment jobs (RESUME)
34. What do we have? Comply with Regulation Pioneer Application Current CI/CD Inspection How do we start? Disaster Recovery
35. OA Push with High Traffic Volume Pain Points 10x requests increase
36. Designed CI/CD flow Developer Trigger deployment Build docker image by branch Proxy Agent Trigger Build Code Push 1. Run deploy script 2. Generate YAML 3. Apply on our cluster
37. How we made our own deployment integration? First Try – Customize target project with following mechanism Deployment YAML template Build docker image script App deploy script – Generate YAML & Apply YAML Tool for creating ConfigMap YAML
38. How we made our own deployment integration? First Try – Make a common deployment YAML
39. How we made our own deployment integration? First Try - Making our own ConfigMap YAML generator with python
40. How we made our own deployment integration? First Try – Write our own deploy script with the tool above 1. Generating ConfigMap YAML 2. Replace the YAML template with each environment & deployment version 3. Print output YAML on each run 4. Apply YAML to cluster
41. We’ve made it!! First application running on our own cluster
42. How we made our own deployment integration? First Try Proxy Agent Generate Deployment YAML Apply Deployment YAML Upload by nDeploy
43. How we made our own deployment integration? First Try Generate Deployment YAML Apply Deployment YAML Upload by nDeploy Is it suitable for our team? Proxy Agent
44. How we made our own deployment integration? First Try Proxy Agent Generate Deployment YAML Apply Deployment YAML Upload by nDeploy Raw use on k8s resource YAML Highly Customized by each project / application
45. How we made our own deployment integration? First Try Proxy Agent Generate Deployment YAML Apply Deployment YAML Upload by nDeploy Raw use on k8s resource YAML Highly Customized by each project / application
46. Tech Stack Frontend Backend
47. Tech Stack Backend Share same resource YAML on same tech stack Frontend
48. How we made our own deployment integration? Optimized – Build our own CLI tool
49. How we made our own deployment integration? Optimized – Build our own CLI tool Pay CLI 1. Fetch shared YAML template by project type 2. Generate Deployment YAML 3. Apply Deployment YAML 5. Send deployment notification 4. Store generated YAMLs for app for manifests archive
50. How we made our own deployment integration? Optimized – Build our own CLI tool REMOVED Build docker image script (NO CHANGED) Run command of generating YAML & Run deploy command REMOVED
51. How we made our own deployment integration? Optimized – Build our own CLI tool
52. SECTION 03 WHAT
53. Run deployment with our own CLI tool Integrate k8s deployment easily Upgrade Cluster without downtime Ability to migrate to another cluster Centralized k8s resources Outcome Blue Green Deployment
54. vUser (process/thread) TPS Peak TPS Successful Response Error Response Fail Rate VM 300 (10/30) 47 89 7716 832 9.7 % Kubernetes 62.9 79 10190 0 0 % Kubernetes with HPA (auto-scale 1 -> 3) 126 193 20437 3 0.1 % Kubernetes with HPA (Pre-scale 1 -> 3) 173.2 202 28411 0 0 % Performance Test One of our campaign page (server-side rendering) ※ Test on our BETA env with same CPU/MEM resources 33% Improvement 37% Improvement
55. HPA helps solving the peak issue Pod statistic on Grafana Request statistic on log system Scale out automatically
56. Cluster Usage 70+ Services 200+ Pods 13M+ Requests / day 11M+ Users
57. SECTION 04 WHICH
58. LB Gateway Server Gateway Application Gateway Application Web Server LB API Server Application A Application B Application C Web Static files Dedicate Server with Resource Reserved Hard to scale with request dynamically Which we’ve changed Legacy VM System Infrastructure Web Application
59. LB Gateway Server Gateway Application Gateway Application Web Server LB Which we’ve changed Cluster Based System Infrastructure nodepool A nodepool B nodepool C Ingress Controller
60. Improvement
61. "Stability is the foundation of success."