Lessons learned using GitOps to deploy thousands of Kubernetes pods Edgaras Apšega Site Reliability Engineer @apsega

Vinted infrastructure 3x growth over 2 years Kubernetes production stats ● All services running on Kubernetes* ● 2000+ deployments per day ● 6000+ running pods ● 300+ physical nodes (30k CPU cores; 120TB memory) *Almost

What’s GitOps, anyway?

GitOps Coined in August 2017 by Weaveworks CEO Alexis Richardson GitOps is a DevOps process characterized by: Best practices of deployment, management and monitoring of containerized applications Experience for managing applications with fully automated pipelines/workflows using Git for development and operations Use of Git revision control system to track and approve changes to the infrastructure

Why GitOps? Infrastructure as a Code (IaaC) Repeatability Reliability Efficiency Visibility Self-service Code reviews

Imperative vs. Declarative Imperative Declarative $ kubectl run busybox --image=busybox:1.29 --restart=Never --command -- sleep 3600 apiVersion: v1 kind: Pod metadata: labels: run: busybox name: busybox spec: containers: - command: - sleep - "3600" image: busybox:1.29 imagePullPolicy: Always name: busybox

Build Docker push Test Git commit and push Git clone config repo Update manifests kubectl apply Git clone config repo Discover manifests GitOps continuous integration GitOps continuous deployment

Git strategies Single branch (multiple directories) Multiple branches Do use directories for GitOps environments kubernetes-deployments ├── demo-app │ ├── development │ ├── production │ └── staging └── guestbook ├── development ├── production └── staging Don’t use long-running branches for GitOps environments

Configuration management Helm Kustomize Package manager Go templating language Environments per values files No parameters and templates - as close as you can get to Kubernetes manifests No parameters and templates - limiting in edge cases Overlays per environment More a language than a tool JSON with comments and templating Not Kubernetes specific Jsonnet

Vinted case study: GitOps

ArgoCD App of Apps Pattern App of Apps manifest ArgoCD custom plugins

Kubernetes deployments Kubernetes deployments directory Application manifests kubernetes-deployments ├── guestbook │ ├── frontend │ │ ├── development-values.yaml │ │ ├── development-application.yaml │ │ ├── staging-values.yaml │ │ ├── staging-application.yaml │ │ ├── production-values.yaml │ │ └── production-application.yaml │ ├── backend │ │ ├── development-values.yaml │ │ ├── development-application.yaml │ │ ├── staging-values.yaml │ │ ├── staging-application.yaml │ │ ├── production-values.yaml │ │ └── production-application.yaml │ └── common-values.yaml └── demo-app ├── development-values.yaml ├── development-application.yaml ├── staging-values.yaml ├── staging-application.yaml ├── production-values.yaml └── production-application.yaml

Helm charts Centralized Helm charts Helm chart values files

Helm charts (2) Centralized Helm charts Helm chart values files in Kubernetes deployments repository

ArgoCD application view

Change image tag in deployments repo Jenkinsfile in code repo git clone yq write \ --inplace "${params.ENVIRONMENT}-values.yaml" \ --tag '!!str' image.tag \ "${params.IMAGE_TAG}" git push stage('Build') { when { branch 'master' } steps { echo 'Build and push Docker image' DockerImageBuildAndPublish('frontend') } } stage('Deploy App to Kubernetes') { when { branch 'master' } steps { KubernetesDeploymentsApply('frontend', 'production') } } Jenkins update image tag action

ArgoCD Sync waves Supports application dependencies and defines deployments order Supports only definitions within same defined application Prolongs deployments Use when doing frequent deployments argocd app sync $(APP_NAME) \ --revision ${ARGOCD_APP_REVISION} Sync wave example Sync waves

GitOps is awesome Use directories for GitOps environments Separate code and configuration repositories Use common values for repetitive configuration Avoid using cross application dependencies

Thank you! @apsega