GitOps: Introduction to Continuous Operations with Kubernetes - Mastering Kubernetes 2021

Slide 1

Slide 1 text

// GITOPS - CONTINUOUS OPERATIONS WITH KUBERNETES Johannes Schnatterer, Cloudogu GmbH Version: 202107141319-8052d90 @jschnatterer 1 . 1

Slide 2

Slide 2 text

Agenda • What is GitOps? • Where can it be used? • How can it be used? • What challenges arise? 1 . 2

Slide 3

Slide 3 text

What is GitOps? 2 . 1

Slide 4

Slide 4 text

• Operating model • Origin: blog post by Weaveworks, August 2017 Use developer tooling to drive operations weave.works/blog/gitops-operations-by-pull-request 2 . 2

Slide 5

Slide 5 text

"Classic" Continuous Delivery ("CIOps") Developer Git Repo CI Server K8s Cluster push pull deploy GitOps K8s Cluster Developer Git Repo CI Server GitOps operator push pull pull deploy 2 . 3

Slide 6

Slide 6 text

GitOps Principles 1 The principle of declarative desired state 2 The principle of immutable desired state versions 3 The principle of continuous state reconciliation 4 The principle of operations through declaration github.com/open-gitops/documents/blob/main/PRINCIPLES.md 2 . 4

Slide 7

Slide 7 text

GitOps vs DevOps • DevOps is about collaboration of formerly separate groups (mindset) • GitOps focuses on ops (operations model) • GitOps can be used with or without DevOps 2 . 5

Slide 8

Slide 8 text

K8s Cluster Developer Git Repo CI Server GitOps operator push pull pull deploy Advantages of GitOps • (Almost) no access to cluster from outside • No credentials on CI server • Forces 100% declarative description • auditable • automatic sync of cluster and git • Enterprise: Accessing git is simpler (no new firewall rules) 2 . 6

Slide 9

Slide 9 text

What can GitOps be used for? 3 . 1

Slide 10

Slide 10 text

GitOps History in a nutshell • grew up operating applications on Kubernetes, • is now rising above it, operating clusters and other (cloud) infrastructure 3 . 2

Slide 11

Slide 11 text

A GitOps Vision Physical Layer Cloud Infra Cluster K8s Clusters Cloud Infra Physical Layer Servers Switches ... IAM DNS ... Pods Services ... K8s Resources 3 . 3

Slide 12

Slide 12 text

GitOps reality Physical Layer Cloud Infra Cluster K8s Clusters Cloud Infra Physical Layer Servers Switches ... IAM DNS ... Pods Services ... K8s Resources GitOps tool m a t u r i t y GitOps tool maturity 3 . 4

Slide 13

Slide 13 text

How can GitOps be used? 4 . 1

Slide 14

Slide 14 text

Physical Layer Cloud Infra Cluster K8s Clusters Cloud Infra Physical Layer Servers Switches ... IAM DNS ... Pods Services ... K8s Resources GitOps tool m a t u r i t y GitOps tool maturity Categories • Tools for Kubernetes AppOps • Tools for Kubernetes ClusterOps • Supplementary GitOps tools 4 . 2

Slide 15

Slide 15 text

GitOps Tools for Kubernetes AppOps 4 . 3

Slide 16

Slide 16 text

Operate Kubernetes with Kubernetes GitOps Repo Management Cluster Target Cluster Target Cluster Target Cluster 4 . 4

Slide 17

Slide 17 text

| - GitOps Tools for Kubernetes ClusterOps + Cloud or Operator • • • rancher/terraform-controller 4 . 5

Slide 18

Slide 18 text

Supplementary GitOps tools Secrets • • • + K8s integration • Operators for Key Management Systems bitnami-labs/sealed-secrets Soluto/kamus mozilla/sops 4 . 6

Slide 19

Slide 19 text

Others • Backups • Deployment Strategies - Progressive Delivery • ... 4 . 7

Slide 20

Slide 20 text

See also ( ) • General tool comparison, • tips on criteria for tool selection, • comparison of ArgoCD v1 and Flux v2 cloudogu.com/blog/gitops-tools iX 4/2021 4 . 8

Slide 21

Slide 21 text

What challenges arise with GitOps? 5 . 1

Slide 22

Slide 22 text

More Infra ... • GitOps Operator: One or more custom controllers • Helm, Kustomize Controllers • Operators for Supplementary tools (secrets, etc.) • Monitoring/Alerting systems • ... 5 . 2

Slide 23

Slide 23 text

... higher cost • Maintenance/patching (vendor dependency) • Resource consumption • Error handling • failing late and silently • monitoring/alerting required • reason might be difficult to pinpoint • operators cause alerts (OOM errors, on Git/API server down, etc.) 5 . 3

Slide 24

Slide 24 text

Day two questions • POC is simple • Operations in prod has its challenges • How to realize staging? • Role of CI server? • How to structure repos? • How to delete resources? • How to realize local dev env? • ... 5 . 4

Slide 25

Slide 25 text

Implementing stages Idea 1: Staging Branches • Develop Staging • Main Production Logic for branching complicated and error prone (merges) 5 . 5

Slide 26

Slide 26 text

Idea 2: Staging folders • On the same branch: One folder per stage • Process: • Commit to staging folder only, • create short lived branches and pull requests for prod • Risky, but can be automized • Logic for branching simpler • Supports arbitrary number of stages 5 . 6

Slide 27

Slide 27 text

Role of CI server K8s Cluster Developer Git Repo CI Server GitOps operator OCI Registry push pull push pull pull deploy 5 . 7

Slide 28

Slide 28 text

Number of repositories: application vs GitOps repo • Good pratice: Keeping everything in app repo (code, docs, infra) • GitOps: Put infra in separate repo! • Advantage: All cluster infra in one repo • Disadvantages: • Separated maintenance & versioning of app and infra code • Review spans across multiple repos • Local dev more difficult Can't we have both? 5 . 8

Slide 29

Slide 29 text

Yes, we can! Using a CI-Server K8s Cluster Developer App Repo GitOps Repo CI Server GitOps operator OCI Registry push pull push push pull pull deploy 5 . 9

Slide 30

Slide 30 text

Disadvantages • Complexity in CI pipelines efforts for development • A lot can go wrong. Examples • Git Conflicts caused by concurrency • Danger of inconsistencies Recommendation: Use a plugin or library Example: cloudogu/gitops-build-lib 5 . 10

Slide 31

Slide 31 text

Advantages • Fail early: static code analysis + policy check on CI server, e.g. yamlint, kubeval, helm lint, conftest • Automated staging (e.g. PR creation, namespaces) • Use IaC for local dev • Write config files not inline YAML Automatically converted to configMap • Simplify review by adding info to PRs 5 . 11

Slide 32

Slide 32 text

How to delete resources? • "garbage collection" (Flux) / "resource pruning" (ArgoCD) disabled by default • Enable from beginning avoid manual interaction 5 . 12

Slide 33

Slide 33 text

Local development • Option 1: Deploy GitOps operator and Git server on local cluster complicated • Option 2: Just carry on without GitOps. Easy, when IaC remains in app repo 5 . 13

Slide 34

Slide 34 text

Demo Your Host K3d Container SCM-Manager Docker Daemon GitOps-Operator Jenkins Controller Agent Pods OCI Registry API-Server kubelet containerd daemon App Repo GitOps Repo run pull push push run containers manage run containers build images pull apply resources pull watch cloudogu/gitops-playground 6

Slide 35

Slide 35 text

7 . 1

Slide 36

Slide 36 text

Personal Conclusion After migrating to and operating with GitOps in production for > 1 year • Smoother CI/CD, • everything declarative • faster deployment • But: security advantages only when finished migration 7 . 2

Slide 37

Slide 37 text

GitOps experience distilled Has advantages, once established Mileage for getting there may vary 7 . 3

Slide 38

Slide 38 text

Adopt GitOps? • Greenfield • AppOps: Definitely • ClusterOps: Depends • Brownfield: Depends 7 . 4

Slide 39

Slide 39 text

Johannes Schnatterer, Cloudogu GmbH • GitOps Resources (intro, tool comparison, etc.) • Links to GitOps Playground and Build Lib • Discussions • Trainings cloudogu.com/gitops 7 . 5

Slide 40

Slide 40 text

Image sources • What is GitOps? • What can GitOps be used for? • How can GitOps be used? Tools: • What challenges arise with GitOps? https://pixabay.com/illustrations/question-mark- important-sign-1872665/ https://pixabay.com/photos/hammer- nails-wood-board-tool-work-1629587/ https://pixabay.com/photos/tools- knives-wrenches-drills-1845426/ https://unsplash.com/photos/bJhT_8nbUA0 7 . 6