Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Lessons learned using GitOps to deploy thousands of Kubernetes pods

Lessons learned using GitOps to deploy thousands of Kubernetes pods

A talk about GitOps, what it is and what benefits it brings and how Vinted utilises GitOps to deploy thousands of Kubernetes pods over more than 2k times per day.

Edgaras Apšega

November 17, 2022
Tweet

More Decks by Edgaras Apšega

Other Decks in Technology

Transcript

  1. Lessons learned using GitOps to deploy thousands of Kubernetes pods

    Edgaras Apšega Site Reliability Engineer @apsega
  2. Vinted infrastructure 3x growth over 2 years Kubernetes production stats

    • All services running on Kubernetes* • 2000+ deployments per day • 6000+ running pods • 300+ physical nodes (30k CPU cores; 120TB memory) *Almost
  3. What’s GitOps, anyway?

  4. GitOps Coined in August 2017 by Weaveworks CEO Alexis Richardson

    GitOps is a DevOps process characterized by: Best practices of deployment, management and monitoring of containerized applications Experience for managing applications with fully automated pipelines/workflows using Git for development and operations Use of Git revision control system to track and approve changes to the infrastructure
  5. Why GitOps? Infrastructure as a Code (IaaC) Repeatability Reliability Efficiency

    Visibility Self-service Code reviews
  6. Imperative vs. Declarative Imperative Declarative $ kubectl run busybox --image=busybox:1.29

    --restart=Never --command -- sleep 3600 apiVersion: v1 kind: Pod metadata: labels: run: busybox name: busybox spec: containers: - command: - sleep - "3600" image: busybox:1.29 imagePullPolicy: Always name: busybox
  7. Build Docker push Test Git commit and push Git clone

    config repo Update manifests kubectl apply Git clone config repo Discover manifests GitOps continuous integration GitOps continuous deployment
  8. Git strategies Single branch (multiple directories) Multiple branches Do use

    directories for GitOps environments kubernetes-deployments ├── demo-app │ ├── development │ ├── production │ └── staging └── guestbook ├── development ├── production └── staging Don’t use long-running branches for GitOps environments
  9. Configuration management Helm Kustomize Package manager Go templating language Environments

    per values files No parameters and templates - as close as you can get to Kubernetes manifests No parameters and templates - limiting in edge cases Overlays per environment More a language than a tool JSON with comments and templating Not Kubernetes specific Jsonnet
  10. Vinted case study: GitOps

  11. ArgoCD App of Apps Pattern App of Apps manifest ArgoCD

    custom plugins
  12. Kubernetes deployments Kubernetes deployments directory Application manifests kubernetes-deployments ├── guestbook

    │ ├── frontend │ │ ├── development-values.yaml │ │ ├── development-application.yaml │ │ ├── staging-values.yaml │ │ ├── staging-application.yaml │ │ ├── production-values.yaml │ │ └── production-application.yaml │ ├── backend │ │ ├── development-values.yaml │ │ ├── development-application.yaml │ │ ├── staging-values.yaml │ │ ├── staging-application.yaml │ │ ├── production-values.yaml │ │ └── production-application.yaml │ └── common-values.yaml └── demo-app ├── development-values.yaml ├── development-application.yaml ├── staging-values.yaml ├── staging-application.yaml ├── production-values.yaml └── production-application.yaml
  13. Helm charts Centralized Helm charts Helm chart values files

  14. Helm charts (2) Centralized Helm charts Helm chart values files

    in Kubernetes deployments repository
  15. ArgoCD application view

  16. Change image tag in deployments repo Jenkinsfile in code repo

    git clone yq write \ --inplace "${params.ENVIRONMENT}-values.yaml" \ --tag '!!str' image.tag \ "${params.IMAGE_TAG}" git push stage('Build') { when { branch 'master' } steps { echo 'Build and push Docker image' DockerImageBuildAndPublish('frontend') } } stage('Deploy App to Kubernetes') { when { branch 'master' } steps { KubernetesDeploymentsApply('frontend', 'production') } } Jenkins update image tag action
  17. ArgoCD Sync waves Supports application dependencies and defines deployments order

    Supports only definitions within same defined application Prolongs deployments Use when doing frequent deployments argocd app sync $(APP_NAME) \ --revision ${ARGOCD_APP_REVISION} Sync wave example Sync waves
  18. GitOps is awesome Use directories for GitOps environments Separate code

    and configuration repositories Use common values for repetitive configuration Avoid using cross application dependencies
  19. Thank you! @apsega