通往Cloud Native的神奇之旅: 自製 Kubernetes 工具大揭秘

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

通往Cloud Native的神奇之旅

Slide 3

Slide 3 text

01 02 03 04 Flow Decision & Tech Stack Self-Implemented Tool Outcome & Improvement Pain Points 05 Summary AGENDA

Slide 4

Slide 4 text

01 02 03 04 Flow Decision & Tech Stack Self-Implemented Tool Outcome & Improvement Pain Points 05 Summary AGENDA Why How What Which

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

LINE Pay Services Payment Service Financial Service Points Service Promotion Service Partner Service Donation Service Merchant Service

Slide 7

Slide 7 text

SECTION 01 WHY

Slide 8

Slide 8 text

01 Pain Points

Slide 9

Slide 9 text

OA Push with High Traffic Volume Pain Points 10x RPS increase

Slide 10

Slide 10 text

OA Push with High Traffic Volume Pain Points 10x requests increase

Slide 11

Slide 11 text

Legacy applications running on VM Application Scale out / in Self-Healing Blue Green Deployment Server Preparation Pain Points

Slide 12

Slide 12 text

Legacy applications running on VM Application Scale out / in Self-Healing Blue Green Deployment Server Preparation Pain Points

Slide 13

Slide 13 text

Legacy applications running on VM Application Scale out / in Self-Healing Blue Green Deployment Server Preparation Pain Points

Slide 14

Slide 14 text

Legacy applications running on VM Application Scale out / in Self-Healing Blue Green Deployment Server Preparation Pain Points

Slide 15

Slide 15 text

SECTION 02 HOW

Slide 16

Slide 16 text

What do we have? Comply with Regulation Pioneer Application Current CI/CD Inspection How do we start? Disaster Recovery

Slide 17

Slide 17 text

Which kind of resource that we could use What we've got on the table? 2020 2021 2022 2023 App Integration HARBOR

Slide 18

Slide 18 text

Pros & Cons • Infra Team support & Gradually support with Verda Resource integration • In 2020, the latest version on Verda VKS is v1.15 • We could choose the resources that we need by our own • Managed all infra layer by our own

Slide 19

Slide 19 text

• Infra Team support & Gradually support with Verda Resource integration • In 2020, the latest version on Verda VKS is v1.15 • We could choose the resources that we need by our own • Managed all infra layer by our own Both of them are baremetal version of Kubernetes. Not like GKE or EKS on AWS. They both need installation on related resources. (eg. Ingress Controller) Pros & Cons

Slide 20

Slide 20 text

What we've got on the table?

Slide 21

Slide 21 text

What do we have? Comply with Regulation Pioneer Application Current CI/CD Inspection How do we start? Disaster Recovery

Slide 22

Slide 22 text

AS-IS CI/CD flow Developer Trigger deployment Build excutable files by branch Deploy to target runtime servers VMs Trigger Build Upload excutable files Code Push

Slide 23

Slide 23 text

What do we have? Comply with Regulation Pioneer Application Current CI/CD Inspection How do we start? Disaster Recovery

Slide 24

Slide 24 text

AS-IS CI/CD flow Developer Trigger deployment Build excutable files by branch Deploy to target runtime servers VMs Trigger Build Upload excutable files Code Push Centralize CI/CD flow control with regulation compliance RESTRICTED ZONE

Slide 25

Slide 25 text

TO-BE CI/CD flow Developer Trigger deployment Build docker image by branch Proxy Agent Trigger Build Upload deploy script & version tag Code Push Pull runtime image Push runtime image Manipulate Cluster

Slide 26

Slide 26 text

TO-BE CI/CD flow Developer Trigger deployment Build docker image by branch Proxy Agent Trigger Build Code Push 1. Run deploy script 2. Generate YAML 3. Apply on our cluster

Slide 27

Slide 27 text

RESTRICTED ZONE TO-BE CI/CD flow Developer Trigger deployment Build docker image by branch Proxy Agent Trigger Build Code Push

Slide 28

Slide 28 text

What do we have? Comply with Regulation Pioneer Application Current CI/CD Inspection How do we start? Disaster Recovery

Slide 29

Slide 29 text

Disaster Recovery (AS-IS) Developer Trigger deployment Build excutable files by branch Deploy to target runtime servers VMs Trigger Build Upload excutable files Code Push

Slide 30

Slide 30 text

Disaster Recovery (AS-IS) Developer Trigger deployment Build excutable files by branch Replacement VMs Trigger Build Upload excutable files Code Push Re-run the deployment jobs (RESUME)

Slide 31

Slide 31 text

Disaster Recovery (AS-IS) Developer Trigger deployment Build excutable files by branch Replacement VMs Trigger Build Upload excutable files Code Push Takes lot of times & different requirements on each applications Re-run the deployment jobs (RESUME)

Slide 32

Slide 32 text

Disaster Recovery (TO-BE) Developer Trigger deployment Build docker image by branch Proxy Agent Trigger Build Code Push

Slide 33

Slide 33 text

Disaster Recovery (TO-BE) Developer Trigger deployment Build docker image by branch Proxy Agent Trigger Build Code Push 1. Setup the cluster with preserved installation YAML 2. Setup the Proxy Agent Config 3. Re-run the deployment jobs (RESUME)

Slide 34

Slide 34 text

What do we have? Comply with Regulation Pioneer Application Current CI/CD Inspection How do we start? Disaster Recovery

Slide 35

Slide 35 text

OA Push with High Traffic Volume Pain Points 10x requests increase

Slide 36

Slide 36 text

Designed CI/CD flow Developer Trigger deployment Build docker image by branch Proxy Agent Trigger Build Code Push 1. Run deploy script 2. Generate YAML 3. Apply on our cluster

Slide 37

Slide 37 text

How we made our own deployment integration? First Try – Customize target project with following mechanism Deployment YAML template Build docker image script App deploy script – Generate YAML & Apply YAML Tool for creating ConfigMap YAML

Slide 38

Slide 38 text

How we made our own deployment integration? First Try – Make a common deployment YAML

Slide 39

Slide 39 text

How we made our own deployment integration? First Try - Making our own ConfigMap YAML generator with python

Slide 40

Slide 40 text

How we made our own deployment integration? First Try – Write our own deploy script with the tool above 1. Generating ConfigMap YAML 2. Replace the YAML template with each environment & deployment version 3. Print output YAML on each run 4. Apply YAML to cluster

Slide 41

Slide 41 text

We’ve made it!! First application running on our own cluster

Slide 42

Slide 42 text

How we made our own deployment integration? First Try Proxy Agent Generate Deployment YAML Apply Deployment YAML Upload by nDeploy

Slide 43

Slide 43 text

How we made our own deployment integration? First Try Generate Deployment YAML Apply Deployment YAML Upload by nDeploy Is it suitable for our team? Proxy Agent

Slide 44

Slide 44 text

How we made our own deployment integration? First Try Proxy Agent Generate Deployment YAML Apply Deployment YAML Upload by nDeploy Raw use on k8s resource YAML Highly Customized by each project / application

Slide 45

Slide 45 text

Slide 46

Slide 46 text

Tech Stack Frontend Backend

Slide 47

Slide 47 text

Tech Stack Backend Share same resource YAML on same tech stack Frontend

Slide 48

Slide 48 text

How we made our own deployment integration? Optimized – Build our own CLI tool

Slide 49

Slide 49 text

How we made our own deployment integration? Optimized – Build our own CLI tool Pay CLI 1. Fetch shared YAML template by project type 2. Generate Deployment YAML 3. Apply Deployment YAML 5. Send deployment notification 4. Store generated YAMLs for app for manifests archive

Slide 50

Slide 50 text

How we made our own deployment integration? Optimized – Build our own CLI tool REMOVED Build docker image script (NO CHANGED) Run command of generating YAML & Run deploy command REMOVED

Slide 51

Slide 51 text

How we made our own deployment integration? Optimized – Build our own CLI tool

Slide 52

Slide 52 text

SECTION 03 WHAT

Slide 53

Slide 53 text

Run deployment with our own CLI tool Integrate k8s deployment easily Upgrade Cluster without downtime Ability to migrate to another cluster Centralized k8s resources Outcome Blue Green Deployment

Slide 54

Slide 54 text

vUser (process/thread) TPS Peak TPS Successful Response Error Response Fail Rate VM 300 (10/30) 47 89 7716 832 9.7 % Kubernetes 62.9 79 10190 0 0 % Kubernetes with HPA (auto-scale 1 -> 3) 126 193 20437 3 0.1 % Kubernetes with HPA (Pre-scale 1 -> 3) 173.2 202 28411 0 0 % Performance Test One of our campaign page (server-side rendering) ※ Test on our BETA env with same CPU/MEM resources 33% Improvement 37% Improvement

Slide 55

Slide 55 text

HPA helps solving the peak issue Pod statistic on Grafana Request statistic on log system Scale out automatically

Slide 56

Slide 56 text

Cluster Usage 70+ Services 200+ Pods 13M+ Requests / day 11M+ Users

Slide 57

Slide 57 text

SECTION 04 WHICH

Slide 58

Slide 58 text

LB Gateway Server Gateway Application Gateway Application Web Server LB API Server Application A Application B Application C Web Static files Dedicate Server with Resource Reserved Hard to scale with request dynamically Which we’ve changed Legacy VM System Infrastructure Web Application

Slide 59

Slide 59 text

LB Gateway Server Gateway Application Gateway Application Web Server LB Which we’ve changed Cluster Based System Infrastructure nodepool A nodepool B nodepool C Ingress Controller