Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How GitOps Helps Kubernetes Adoption

How GitOps Helps Kubernetes Adoption

Chih Chiang Tsai
LINE Taiwan Site Reliability Engineering Software Engineer
https://linedevday.linecorp.com/2020/ja/sessions/9156
https://linedevday.linecorp.com/2020/en/sessions/9156

Eebedc2ee7ff95ffb9d9102c6d4a065c?s=128

LINE DevDay 2020

November 25, 2020
Tweet

Transcript

  1. None
  2. Current Adoption Status October 2020 Projects 20+ App Configs 130+

    K8s Clusters 50+
  3. Projects Adopted LINE SPOT LINE Shopping LINE Travel LINE HUB

    LINE MUSIC LINE TODAY
  4. Infrastructure Modernization

  5. Cloud-Native Applications Microservices Containers DevOps

  6. Why Modernize Infrastructure? Scalability Developer On-boarding Portability

  7. Developer On-boarding Why Modernize Infrastructure? BEFORE. Install dependencies, setup environments,

    lots of moving parts AFTER. Self-contained, ready-to-go Docker images can be run anywhere with Docker Scalability Portability
  8. Developer On-boarding Why Modernize Infrastructure? BEFORE. Take days if not

    weeks to get a development environment setup AFTER. The entire stack is only a single kubectl apply command away Scalability Portability
  9. Developer On-boarding Why Modernize Infrastructure? BEFORE. Scaling with VMs, high

    overhead, slower startup time AFTER. Seconds to start, run more applications or replicas on a node, higher utilization Scalability Portability
  10. › Started at beginning of 2018 › "One cluster for

    all" approach › Separate clusters for dev, staging and production environments › K8s provisioned and managed by Rancher Kubernetesization at LINE Taiwan
  11. Shared K8s Cluster at LINE Taiwan December 2019 Namespaces 35

    Running Pods 3500+ Nodes 100+
  12. Challenges of Adopting Kubernetes

  13. Needs for Developer Tooling Limited K8s Knowledge Lack of Awareness

    of K8s Challenges of Adopting Kubernetes Config & Manifests Management Arbitrary Cluster Manipulations Lack of Best Practices
  14. › Most teams thought of Rancher as a sort of

    "PaaS" › Configure and deploy workloads directly from Rancher UI › No awareness of what's "underneath" Rancher Lack of Awareness of Kubernetes
  15. › Most teams thought of Rancher as a sort of

    "PaaS" › Configure and deploy workloads directly from Rancher UI › No awareness of what's "underneath" Rancher Lack of Awareness of Kubernetes
  16. › Limited Kubernetes knowledge due to the ease-of-use of Rancher

    UI › Hard to communicate when encountering issues › Guess and check approach to troubleshooting Limited Kubernetes Knowledge
  17. › VKS is our in-house managed Kubernetes service › Need

    for more developer-friendly tooling after VKS migration › Rancher UI was used by developers, QAs and even non-technical people › Importing VKS clusters to Rancher is not possible Needs for Developer Tooling
  18. › Where to store YAML files? › How to handle

    configuration changes? › Permission & access control of files Configuration & Manifests Management
  19. › Easily obtained kubeconfigs › Direct cluster manipulation thru kubectl

    › Difficult to track changes made to Kubernetes objects Arbitrary Cluster Manipulations
  20. › Many ways of implementing and exposing services › Choice

    of ingress controllers › Resource & capacity planning › Observability of services & applications Lack of Best Practices
  21. GitOps

  22. Benefits of GitOps Single Source of Truth Minimal Direct Manipulations

    Developer-Friendly
  23. › Single Git repository to store configuration for all environments

    › Declarative configuration with Kubernetes manifests in YAML format Single Source of Truth
  24. › Manage Kubernetes clusters with familiar Git workflows › Git

    history becomes the change log of Kubernetes objects and cluster states Developer- Friendly
  25. › Provide a safer manner for changing cluster states ›

    Minimize the need to manipulate Kubernetes objects manually › All changes could be verified thru code reviews › Live cluster states can be synced with the changes automatically Minimal Direct Manipulations
  26. Developer Config Repository Pull Request Developers Create Code Review Merge

    Sync Agent Pull Sync K8s Cluster
  27. GitOps at LINE Taiwan

  28. How We Implement GitOps ArgoCD Standardized Apps & Practices Kustomize

  29. ArgoCD › Declarative Continuous Delivery for Kubernetes › The controller/sync

    agent for config repository and Kubernetes clusters › Web UI for live comparison between desired state and live state
  30. ArgoCD Dashboard

  31. ArgoCD Application View

  32. ArgoCD App Diff

  33. Developer Config Repository Pull Request Developers Create Code Review Merge

    Sync Agent Pull Sync K8s Cluster
  34. Developer Config Repository Pull Request Developers Create Code Review Merge

    ArgoCD Webhook Sync K8s Cluster
  35. Kustomize › Kubernetes native configuration management › Plain, template-free YAMLs

    › Supported natively by ArgoCD › Encourage using with GitOps
  36. Manifests , kustomization.yaml Reference BASE Patches , kustomization.yaml Reference OVERLAY

    Reference Kustomize Build Final Manifests
  37. base overlay kustomization

  38. Generated YAML Output

  39. Developer Config Repository Pull Request Developers Create Code Review Merge

    ArgoCD Webhook Sync K8s Cluster
  40. Developer Config Repository Pull Request Developers Create Code Review Merge

    ArgoCD Webhook Sync K8s Cluster Kustomize Build
  41. › All clusters need to have common infrastructure setup ›

    Ingress controller › Observability agents › Collect all readily deployable application in a single repository Standardized Common Apps
  42. Kustomization with Remote Base

  43. K8s + GitOps Package applications in Docker images Write base

    manifests, use standardized apps if necessary Prepare overlays for different environments Sync manifests to live clusters Base Config Overlays Sync Containerize
  44. Developer Config Repository Pull Request Developers Create Code Review Merge

    ArgoCD Webhook Sync K8s Cluster Kustomize Build
  45. › Create separate read-only accounts for daily use › Create

    usage specific accounts for manual operation › Setup different roles by using Kubernetes RBAC Other Common Practices
  46. Solving Challenges of Adopting K8s › Migrate from shared cluster

    with Rancher to VKS Lack of Awareness of K8s › From shared cluster to project-owned clusters › Introducing GitOps and related tooling Limited K8s Knowledge Needs for Developer Tooling › Use Git as the single source of truth for manifests › Kustomize for manifest customization Config & Manifests Management › Sync cluster states with ArgoCD › Avoid direct usage of kubectl › Standardized common apps › Implement industry-wide best practices Arbitrary Cluster Manipulations Lack of Best Practices
  47. Next Steps › Automate new cluster on-boarding process › YAML

    validation › Kubernetes object validation › Manifest policy checks
  48. Thank you