No content

Current Adoption Status October 2020 Projects 20+ App Configs 130+ K8s Clusters 50+

Projects Adopted LINE SPOT LINE Shopping LINE Travel LINE HUB LINE MUSIC LINE TODAY

Infrastructure Modernization

Cloud-Native Applications Microservices Containers DevOps

Why Modernize Infrastructure? Scalability Developer On-boarding Portability

Developer On-boarding Why Modernize Infrastructure? BEFORE. Install dependencies, setup environments, lots of moving parts AFTER. Self-contained, ready-to-go Docker images can be run anywhere with Docker Scalability Portability

Developer On-boarding Why Modernize Infrastructure? BEFORE. Take days if not weeks to get a development environment setup AFTER. The entire stack is only a single kubectl apply command away Scalability Portability

Developer On-boarding Why Modernize Infrastructure? BEFORE. Scaling with VMs, high overhead, slower startup time AFTER. Seconds to start, run more applications or replicas on a node, higher utilization Scalability Portability

› Started at beginning of 2018 › "One cluster for all" approach › Separate clusters for dev, staging and production environments › K8s provisioned and managed by Rancher Kubernetesization at LINE Taiwan

Shared K8s Cluster at LINE Taiwan December 2019 Namespaces 35 Running Pods 3500+ Nodes 100+

Challenges of Adopting Kubernetes

Needs for Developer Tooling Limited K8s Knowledge Lack of Awareness of K8s Challenges of Adopting Kubernetes Config & Manifests Management Arbitrary Cluster Manipulations Lack of Best Practices

› Most teams thought of Rancher as a sort of "PaaS" › Configure and deploy workloads directly from Rancher UI › No awareness of what's "underneath" Rancher Lack of Awareness of Kubernetes

› Most teams thought of Rancher as a sort of "PaaS" › Configure and deploy workloads directly from Rancher UI › No awareness of what's "underneath" Rancher Lack of Awareness of Kubernetes

› Limited Kubernetes knowledge due to the ease-of-use of Rancher UI › Hard to communicate when encountering issues › Guess and check approach to troubleshooting Limited Kubernetes Knowledge

› VKS is our in-house managed Kubernetes service › Need for more developer-friendly tooling after VKS migration › Rancher UI was used by developers, QAs and even non-technical people › Importing VKS clusters to Rancher is not possible Needs for Developer Tooling

› Where to store YAML files? › How to handle configuration changes? › Permission & access control of files Configuration & Manifests Management

› Easily obtained kubeconfigs › Direct cluster manipulation thru kubectl › Difficult to track changes made to Kubernetes objects Arbitrary Cluster Manipulations

› Many ways of implementing and exposing services › Choice of ingress controllers › Resource & capacity planning › Observability of services & applications Lack of Best Practices

GitOps

Benefits of GitOps Single Source of Truth Minimal Direct Manipulations Developer-Friendly

› Single Git repository to store configuration for all environments › Declarative configuration with Kubernetes manifests in YAML format Single Source of Truth

› Manage Kubernetes clusters with familiar Git workflows › Git history becomes the change log of Kubernetes objects and cluster states Developer- Friendly

› Provide a safer manner for changing cluster states › Minimize the need to manipulate Kubernetes objects manually › All changes could be verified thru code reviews › Live cluster states can be synced with the changes automatically Minimal Direct Manipulations

Developer Config Repository Pull Request Developers Create Code Review Merge Sync Agent Pull Sync K8s Cluster

GitOps at LINE Taiwan

How We Implement GitOps ArgoCD Standardized Apps & Practices Kustomize

ArgoCD › Declarative Continuous Delivery for Kubernetes › The controller/sync agent for config repository and Kubernetes clusters › Web UI for live comparison between desired state and live state

ArgoCD Dashboard

ArgoCD Application View

ArgoCD App Diff

Developer Config Repository Pull Request Developers Create Code Review Merge Sync Agent Pull Sync K8s Cluster

Developer Config Repository Pull Request Developers Create Code Review Merge ArgoCD Webhook Sync K8s Cluster

Kustomize › Kubernetes native configuration management › Plain, template-free YAMLs › Supported natively by ArgoCD › Encourage using with GitOps

Manifests , kustomization.yaml Reference BASE Patches , kustomization.yaml Reference OVERLAY Reference Kustomize Build Final Manifests

base overlay kustomization

Generated YAML Output

Developer Config Repository Pull Request Developers Create Code Review Merge ArgoCD Webhook Sync K8s Cluster

Developer Config Repository Pull Request Developers Create Code Review Merge ArgoCD Webhook Sync K8s Cluster Kustomize Build

› All clusters need to have common infrastructure setup › Ingress controller › Observability agents › Collect all readily deployable application in a single repository Standardized Common Apps

Kustomization with Remote Base

K8s + GitOps Package applications in Docker images Write base manifests, use standardized apps if necessary Prepare overlays for different environments Sync manifests to live clusters Base Config Overlays Sync Containerize

Developer Config Repository Pull Request Developers Create Code Review Merge ArgoCD Webhook Sync K8s Cluster Kustomize Build

› Create separate read-only accounts for daily use › Create usage specific accounts for manual operation › Setup different roles by using Kubernetes RBAC Other Common Practices

Solving Challenges of Adopting K8s › Migrate from shared cluster with Rancher to VKS Lack of Awareness of K8s › From shared cluster to project-owned clusters › Introducing GitOps and related tooling Limited K8s Knowledge Needs for Developer Tooling › Use Git as the single source of truth for manifests › Kustomize for manifest customization Config & Manifests Management › Sync cluster states with ArgoCD › Avoid direct usage of kubectl › Standardized common apps › Implement industry-wide best practices Arbitrary Cluster Manipulations Lack of Best Practices

Next Steps › Automate new cluster on-boarding process › YAML validation › Kubernetes object validation › Manifest policy checks

Thank you