Lessons learned using GitOps to deploy thousands of Kubernetes pods

A talk about GitOps, what it is and what benefits it brings and how Vinted utilises GitOps to deploy thousands of Kubernetes pods over more than 2k times per day.

Transcript

1. Lessons learned using
   GitOps to deploy thousands
   of Kubernetes pods
   Edgaras Apšega
   Site Reliability Engineer
   @apsega
   

   View Slide

2. Vinted infrastructure
   3x growth over 2 years Kubernetes production stats
   ● All services running on Kubernetes*
   ● 2000+ deployments per day
   ● 6000+ running pods
   ● 300+ physical nodes (30k CPU cores; 120TB memory)
   *Almost
   

   View Slide

3. What’s GitOps, anyway?
   

   View Slide

4. GitOps
   Coined in August 2017 by Weaveworks CEO Alexis Richardson
   GitOps is a DevOps process characterized by:
   Best practices of deployment, management and monitoring of containerized
   applications
   Experience for managing applications with fully automated pipelines/workflows using
   Git for development and operations
   Use of Git revision control system to track and approve changes to the infrastructure
   

   View Slide

5. Why GitOps?
   Infrastructure as a Code (IaaC)
   Repeatability
   Reliability
   Efficiency
   Visibility
   Self-service
   Code reviews
   

   View Slide

6. Imperative vs. Declarative
   Imperative Declarative
   $ kubectl run busybox --image=busybox:1.29
   --restart=Never --command -- sleep 3600
   apiVersion: v1
   kind: Pod
   metadata:
   labels:
   run: busybox
   name: busybox
   spec:
   containers:
   - command:
   - sleep
   - "3600"
   image: busybox:1.29
   imagePullPolicy: Always
   name: busybox
   

   View Slide

7. Build
   Docker
   push
   Test
   Git commit
   and push
   Git clone
   config repo
   Update
   manifests
   kubectl
   apply
   Git clone
   config repo
   Discover
   manifests
   GitOps continuous integration
   GitOps continuous deployment
   

   View Slide

8. Git strategies
   Single branch (multiple directories) Multiple branches
   Do use directories for GitOps environments
   kubernetes-deployments
   ├── demo-app
   │ ├── development
   │ ├── production
   │ └── staging
   └── guestbook
   ├── development
   ├── production
   └── staging
   Don’t use long-running branches for GitOps
   environments
   

   View Slide

9. Configuration management
   Helm Kustomize
   Package manager
   Go templating language
   Environments per values files
   No parameters and templates - as
   close as you can get to Kubernetes
   manifests
   No parameters and templates -
   limiting in edge cases
   Overlays per environment
   More a language than a tool
   JSON with comments and
   templating
   Not Kubernetes specific
   Jsonnet
   

   View Slide

10. Vinted case study:
    GitOps
    

    View Slide

11. ArgoCD App of Apps Pattern
    App of Apps manifest ArgoCD custom plugins
    

    View Slide

12. Kubernetes deployments
    Kubernetes deployments directory Application manifests
    kubernetes-deployments
    ├── guestbook
    │ ├── frontend
    │ │ ├── development-values.yaml
    │ │ ├── development-application.yaml
    │ │ ├── staging-values.yaml
    │ │ ├── staging-application.yaml
    │ │ ├── production-values.yaml
    │ │ └── production-application.yaml
    │ ├── backend
    │ │ ├── development-values.yaml
    │ │ ├── development-application.yaml
    │ │ ├── staging-values.yaml
    │ │ ├── staging-application.yaml
    │ │ ├── production-values.yaml
    │ │ └── production-application.yaml
    │ └── common-values.yaml
    └── demo-app
    ├── development-values.yaml
    ├── development-application.yaml
    ├── staging-values.yaml
    ├── staging-application.yaml
    ├── production-values.yaml
    └── production-application.yaml
    

    View Slide

13. Helm charts
    Centralized Helm charts Helm chart values files
    

    View Slide

14. Helm charts (2)
    Centralized Helm charts
    Helm chart values files in Kubernetes
    deployments repository
    

    View Slide

15. ArgoCD application view
    

    View Slide

16. Change image tag in deployments repo
    Jenkinsfile in code repo
    git clone
    yq write \
    --inplace "${params.ENVIRONMENT}-values.yaml" \
    --tag '!!str' image.tag \ "${params.IMAGE_TAG}"
    git push
    stage('Build') {
    when {
    branch 'master'
    }
    steps {
    echo 'Build and push Docker image'
    DockerImageBuildAndPublish('frontend')
    }
    }
    stage('Deploy App to Kubernetes') {
    when {
    branch 'master'
    }
    steps {
    KubernetesDeploymentsApply('frontend',
    'production')
    }
    }
    Jenkins update image tag action
    

    View Slide

17. ArgoCD Sync waves
    Supports application dependencies and defines
    deployments order
    Supports only definitions within same defined application
    Prolongs deployments
    Use when doing frequent deployments
    argocd app sync $(APP_NAME) \
    --revision ${ARGOCD_APP_REVISION}
    Sync wave example
    Sync waves
    

    View Slide

18. GitOps is awesome
    Use directories for GitOps environments
    Separate code and configuration repositories
    Use common values for repetitive configuration
    Avoid using cross application dependencies
    

    View Slide

19. Thank you!
    @apsega
    

    View Slide