GitOps: Introduction to Continuous Operations with Kubernetes - Mastering Kubernetes 2021

Für viele ist GitOps die Weiterentwicklung klassischer CI/CD-Prozesse. Es klingt simpel und bietet Vorteile, z.B. das konsequente deklarative Beschreiben von Infrastructure as Code. Auch die Security verbessert sich, weder Mensch noch CI-Server brauchen Zugriff auf die Betriebsumgebung. Im Detail sind dann aber viele Fragen zu beantworten:

Welche Tools gibt es? Wie lassen sich Stages realisieren, Fehler bemerken, Helm Charts deployen, Ressourcen löschen oder Secrets sicher ablegen? Wie viele Repositories ergeben Sinn und wie strukturiert man sie? Auf diese und andere Fragen geht der Vortrag ein. Der Fokus wird hierbei auf den Betrieb von Anwendungen auf Kubernetes-Clustern liegen.
Dabei fließen Praxistipps aus meinen Erfahrungen aus knapp zwei Jahren GitOps in Produktion aus verschiedenen Umfeldern ein.

Transcript

1. 
   
   // GITOPS -
   
   CONTINUOUS OPERATIONS
   WITH KUBERNETES
   Johannes Schnatterer, Cloudogu GmbH
   Version: 202107141319-8052d90
   
   @jschnatterer
   1
   .
   1
   

   View Slide

2. Agenda
   • What is GitOps?
   • Where can it be used?
   • How can it be used?
   • What challenges arise?
   1
   .
   2
   

   View Slide

3. What is GitOps?
   2
   .
   1
   

   View Slide

4. • Operating model
   • Origin: blog post by Weaveworks, August 2017
   Use developer tooling to drive operations
   weave.works/blog/gitops-operations-by-pull-request
   2
   .
   2
   

   View Slide

5. "Classic" Continuous Delivery ("CIOps")
   Developer
   Git Repo CI Server K8s Cluster
   push pull deploy
   GitOps
   
   K8s Cluster
   Developer
   Git Repo
   CI Server
   GitOps
   operator
   push
   pull
   pull
   deploy
   2
   .
   3
   

   View Slide

6. GitOps Principles
   1 The principle of declarative desired state
   2 The principle of immutable desired state versions
   3 The principle of continuous state reconciliation
   4 The principle of operations through declaration
   github.com/open-gitops/documents/blob/main/PRINCIPLES.md
   2
   .
   4
   

   View Slide

7. GitOps vs DevOps
   • DevOps is about collaboration of formerly separate groups (mindset)
   • GitOps focuses on ops (operations model)
   • GitOps can be used with or without DevOps
   2
   .
   5
   

   View Slide

8. K8s Cluster
   Developer
   Git Repo
   CI Server
   GitOps
   operator
   push
   pull
   pull
   deploy
   Advantages of GitOps
   • (Almost) no access to cluster from outside
   • No credentials on CI server
   • Forces 100% declarative description
   • auditable
   • automatic sync of cluster and git
   • Enterprise: Accessing git is simpler
   
   (no new firewall rules)
   2
   .
   6
   

   View Slide

9. What can GitOps be used for?
   3
   .
   1
   

   View Slide

10. GitOps History in a nutshell
    • grew up operating applications on Kubernetes,
    • is now rising above it, operating clusters
    and other (cloud) infrastructure
    3
    .
    2
    

    View Slide

11. A GitOps Vision
    Physical Layer
    Cloud Infra
    Cluster
    K8s Clusters
    Cloud Infra
    Physical Layer
    Servers Switches
    ...
    IAM DNS
    ...
    Pods Services
    ...
    K8s Resources
    3
    .
    3
    

    View Slide

12. GitOps reality
    Physical Layer
    Cloud Infra
    Cluster
    K8s Clusters
    Cloud Infra
    Physical Layer
    Servers Switches
    ...
    IAM DNS
    ...
    Pods Services
    ...
    K8s Resources
    GitOps tool
    m a t u r i t y
    GitOps tool
    maturity
    3
    .
    4
    

    View Slide

13. How can GitOps be used?
    4
    .
    1
    

    View Slide

14. Physical Layer
    Cloud Infra
    Cluster
    K8s Clusters
    Cloud Infra
    Physical Layer
    Servers Switches
    ...
    IAM DNS
    ...
    Pods Services
    ...
    K8s Resources
    GitOps tool
    m a t u r i t y
    GitOps tool
    maturity
    Categories
    • Tools for Kubernetes AppOps
    • Tools for Kubernetes ClusterOps
    • Supplementary GitOps tools
    4
    .
    2
    

    View Slide

15. GitOps Tools for Kubernetes AppOps
    
    
    
    
    
    
    4
    .
    3
    

    View Slide

16. Operate Kubernetes with Kubernetes
    GitOps Repo Management Cluster Target Cluster
    Target Cluster
    Target Cluster
    4
    .
    4
    

    View Slide

17. | -
    GitOps Tools for Kubernetes ClusterOps
    
    
    
    
    
    
    +
    Cloud or Operator
    •
    •
    • rancher/terraform-controller
    4
    .
    5
    

    View Slide

18. Supplementary GitOps tools
    Secrets
    •
    •
    • + K8s integration
    • Operators for Key Management Systems
    bitnami-labs/sealed-secrets
    Soluto/kamus
    mozilla/sops
    4
    .
    6
    

    View Slide

19. Others
    • Backups
    • Deployment Strategies - Progressive Delivery
    
    
    • ...
    4
    .
    7
    

    View Slide

20. See also
    ( )
    • General tool comparison,
    • tips on criteria for tool selection,
    • comparison of ArgoCD v1 and Flux v2
    cloudogu.com/blog/gitops-tools iX 4/2021
    4
    .
    8
    

    View Slide

21. What challenges arise with GitOps?
    5
    .
    1
    

    View Slide

22. More Infra ...
    • GitOps Operator: One or more custom controllers
    • Helm, Kustomize Controllers
    • Operators for Supplementary tools (secrets, etc.)
    • Monitoring/Alerting systems
    • ...
    5
    .
    2
    

    View Slide

23. ... higher cost
    • Maintenance/patching (vendor dependency)
    • Resource consumption
    • Error handling
    • failing late and silently
    • monitoring/alerting required
    • reason might be difficult to pinpoint
    • operators cause alerts (OOM errors, on Git/API server down, etc.)
    5
    .
    3
    

    View Slide

24. Day two questions
    • POC is simple
    • Operations in prod has its challenges
    • How to realize staging?
    • Role of CI server?
    • How to structure repos?
    • How to delete resources?
    • How to realize local dev env?
    • ...
    5
    .
    4
    

    View Slide

25. Implementing stages
    Idea 1: Staging Branches
    • Develop Staging
    • Main Production
    
    
    
    
    Logic for branching complicated and error prone (merges)
    5
    .
    5
    

    View Slide

26. Idea 2: Staging folders
    • On the same branch: One folder per stage
    • Process:
    • Commit to staging folder only,
    • create short lived branches and pull requests for prod
    • Risky, but can be automized
    
    
    • Logic for branching simpler
    • Supports arbitrary number of stages
    5
    .
    6
    

    View Slide

27. Role of CI server
    K8s Cluster
    Developer
    Git Repo
    CI Server
    GitOps
    operator
    OCI Registry
    push
    pull
    push
    pull
    pull
    deploy
    5
    .
    7
    

    View Slide

28. Number of repositories: application vs GitOps repo
    • Good pratice: Keeping everything in app repo (code, docs, infra)
    • GitOps: Put infra in separate repo!
    • Advantage: All cluster infra in one repo
    • Disadvantages:
    • Separated maintenance & versioning of app and infra code
    • Review spans across multiple repos
    • Local dev more difficult
    
    Can't we have both?
    5
    .
    8
    

    View Slide

29. Yes, we can! Using a CI-Server
    K8s Cluster
    Developer
    App Repo GitOps Repo
    CI Server
    GitOps
    operator
    OCI Registry
    push
    pull push
    push
    pull
    pull
    deploy
    5
    .
    9
    

    View Slide

30. Disadvantages
    • Complexity in CI pipelines efforts for development
    • A lot can go wrong. Examples
    • Git Conflicts caused by concurrency
    • Danger of inconsistencies
    Recommendation: Use a plugin or library
    Example: cloudogu/gitops-build-lib
    5
    .
    10
    

    View Slide

31. Advantages
    • Fail early: static code analysis + policy check on CI server,
    e.g. yamlint, kubeval, helm lint, conftest
    • Automated staging (e.g. PR creation, namespaces)
    • Use IaC for local dev
    • Write config files not inline YAML
    Automatically converted to configMap
    • Simplify review by adding info to PRs
    
    5
    .
    11
    

    View Slide

32. How to delete resources?
    • "garbage collection" (Flux) / "resource pruning" (ArgoCD)
    disabled by default
    • Enable from beginning avoid manual interaction
    5
    .
    12
    

    View Slide

33. Local development
    • Option 1: Deploy GitOps operator and Git server on local cluster
    complicated
    • Option 2: Just carry on without GitOps.
    Easy, when IaC remains in app repo
    5
    .
    13
    

    View Slide

34. Demo
    Your Host
    K3d Container
    SCM-Manager
    Docker Daemon
    GitOps-Operator
    Jenkins Controller Agent Pods
    OCI Registry API-Server
    kubelet
    containerd daemon
    App Repo GitOps Repo
    run
    pull push push run containers
    manage
    run containers
    build images
    pull apply resources
    pull watch
    cloudogu/gitops-playground
    6
    

    View Slide

35. 7
    .
    1
    

    View Slide

36. Personal Conclusion
    After migrating to and operating with GitOps in production for > 1
    year
    • Smoother CI/CD,
    • everything declarative
    • faster deployment
    • But: security advantages only when finished migration
    7
    .
    2
    

    View Slide

37. GitOps experience distilled
    Has advantages, once established
    Mileage for getting there may vary
    7
    .
    3
    

    View Slide

38. Adopt GitOps?
    • Greenfield
    • AppOps: Definitely
    • ClusterOps: Depends
    • Brownfield: Depends
    7
    .
    4
    

    View Slide

39. Johannes Schnatterer, Cloudogu GmbH
    
    • GitOps Resources (intro, tool comparison, etc.)
    • Links to GitOps Playground and Build Lib
    • Discussions
    • Trainings
    cloudogu.com/gitops
    7
    .
    5
    

    View Slide

40. Image sources
    • What is GitOps?
    
    • What can GitOps be used for?
    
    • How can GitOps be used? Tools:
    • What challenges arise with GitOps?
    https://pixabay.com/illustrations/question-mark-
    important-sign-1872665/
    https://pixabay.com/photos/hammer-
    nails-wood-board-tool-work-1629587/
    https://pixabay.com/photos/tools-
    knives-wrenches-drills-1845426/
    https://unsplash.com/photos/bJhT_8nbUA0
    7
    .
    6
    

    View Slide