Lessons learned using GitOps to deploy thousands of Kubernetes pods

A talk about GitOps, what it is and what benefits it brings and how Vinted utilises GitOps to deploy thousands of Kubernetes pods over more than 2k times per day.

Transcript

1. Lessons learned using GitOps to deploy thousands of Kubernetes pods Edgaras Apšega Site Reliability Engineer @apsega
2. Vinted infrastructure 3x growth over 2 years Kubernetes production stats • All services running on Kubernetes* • 2000+ deployments per day • 6000+ running pods • 300+ physical nodes (30k CPU cores; 120TB memory) *Almost
3. What’s GitOps, anyway?
4. GitOps Coined in August 2017 by Weaveworks CEO Alexis Richardson GitOps is a DevOps process characterized by: Best practices of deployment, management and monitoring of containerized applications Experience for managing applications with fully automated pipelines/workflows using Git for development and operations Use of Git revision control system to track and approve changes to the infrastructure
5. Why GitOps? Infrastructure as a Code (IaaC) Repeatability Reliability Efficiency Visibility Self-service Code reviews
6. Imperative vs. Declarative Imperative Declarative $ kubectl run busybox --image=busybox:1.29 --restart=Never --command -- sleep 3600 apiVersion: v1 kind: Pod metadata: labels: run: busybox name: busybox spec: containers: - command: - sleep - "3600" image: busybox:1.29 imagePullPolicy: Always name: busybox
7. Build Docker push Test Git commit and push Git clone config repo Update manifests kubectl apply Git clone config repo Discover manifests GitOps continuous integration GitOps continuous deployment
8. Git strategies Single branch (multiple directories) Multiple branches Do use directories for GitOps environments kubernetes-deployments ├── demo-app │ ├── development │ ├── production │ └── staging └── guestbook ├── development ├── production └── staging Don’t use long-running branches for GitOps environments
9. Configuration management Helm Kustomize Package manager Go templating language Environments per values files No parameters and templates - as close as you can get to Kubernetes manifests No parameters and templates - limiting in edge cases Overlays per environment More a language than a tool JSON with comments and templating Not Kubernetes specific Jsonnet
10. Vinted case study: GitOps
11. ArgoCD App of Apps Pattern App of Apps manifest ArgoCD custom plugins
12. Kubernetes deployments Kubernetes deployments directory Application manifests kubernetes-deployments ├── guestbook │ ├── frontend │ │ ├── development-values.yaml │ │ ├── development-application.yaml │ │ ├── staging-values.yaml │ │ ├── staging-application.yaml │ │ ├── production-values.yaml │ │ └── production-application.yaml │ ├── backend │ │ ├── development-values.yaml │ │ ├── development-application.yaml │ │ ├── staging-values.yaml │ │ ├── staging-application.yaml │ │ ├── production-values.yaml │ │ └── production-application.yaml │ └── common-values.yaml └── demo-app ├── development-values.yaml ├── development-application.yaml ├── staging-values.yaml ├── staging-application.yaml ├── production-values.yaml └── production-application.yaml
13. Helm charts Centralized Helm charts Helm chart values files
14. Helm charts (2) Centralized Helm charts Helm chart values files in Kubernetes deployments repository
15. ArgoCD application view
16. Change image tag in deployments repo Jenkinsfile in code repo git clone yq write \ --inplace "${params.ENVIRONMENT}-values.yaml" \ --tag '!!str' image.tag \ "${params.IMAGE_TAG}" git push stage('Build') { when { branch 'master' } steps { echo 'Build and push Docker image' DockerImageBuildAndPublish('frontend') } } stage('Deploy App to Kubernetes') { when { branch 'master' } steps { KubernetesDeploymentsApply('frontend', 'production') } } Jenkins update image tag action
17. ArgoCD Sync waves Supports application dependencies and defines deployments order Supports only definitions within same defined application Prolongs deployments Use when doing frequent deployments argocd app sync $(APP_NAME) \ --revision ${ARGOCD_APP_REVISION} Sync wave example Sync waves
18. GitOps is awesome Use directories for GitOps environments Separate code and configuration repositories Use common values for repetitive configuration Avoid using cross application dependencies
19. Thank you! @apsega