GitOps Directory Structures Lots of opinions means lots of options GitOps Guide to the Galaxy 1 Christian Hernandez Hybrid Platforms BU

So you’ve decided to GitOps GitOps Guide to the Galaxy ● GitOps principals state that you have a single source of truth. ● So you dumped all your YAML into a Git Repository. ● Now you’re running into issues ○ How do I apply this to multiple clusters? ○ How do I apply this to multiple environments?

Many have solved this problem GitOps Guide to the Galaxy ● Tools built around GitOps faced similar questions ● Many have decided on an opinionated approach. ● Argo Autopilot ○ Repo Layout ● GitOps Application Manager (KAM) ○ Repo Layout ● Flux CD ○ Isn’t opinionated but has recommendations. ● Gerald Nunn

There is NO magic bullet GitOps Guide to the Galaxy ● No one is “right” in this regard. ● Everyone does what makes sense for them. ○ The answer everyone hates: “It depends” ● However there are some best practices and tips to follow ● Grab a pencil and take some notes!

Best Practice: DRY Don’t Repeat Yourself GitOps Guide to the Galaxy 5 Save it once and then forget about it!

Avoiding Duplication GitOps Guide to the Galaxy GitOps enables deployment across multiple clusters, awesome! Wait, how do we manage configuration without copying and pasting yaml everywhere?

Tip: Use Kustomize! GitOps Guide to the Galaxy Kustomize lets you customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is. ● Kustomize is a patching framework ● Enables environment specific changes to be introduced without duplicating yaml unnecessarily ● Unlike templating frameworks, all yaml can be directly applied ● Kustomize is included in kubectl/oc (you can even run: oc kustomize) kubectl apply -k apps/myapp/overlays/dev

Kustomize - Organization Kustomize is organized in a hierarchical directory structure of bases and overlays. ● A base is a directory with a kustomization.yaml file that contains a set of resources and associated customization. ○ A base has no knowledge of an overlay and can be used in multiple overlays. ● An overlay is a directory with a kustomization.yaml file that refers to other kustomization directories as its bases ○ An overlay may have multiple bases and it composes all resources from bases and may also have customization on top of them. GitOps Guide to the Galaxy

Using Kustomize └── apps └── myapp ├── base │ ├── kustomization.yaml │ ├── service.yaml │ ├── route.yaml │ └── deployment.yaml │ └── overlays ├── dev │ ├── patch-route.yaml │ ├── namespace.yaml │ └── kustomization.yaml └── test ├── patch-route.yaml ├── namespace.yaml └── kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - service.yaml - route.yaml - deployment.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization Namespace: dev bases: - ../../base resources: - namespace.yaml patchesStrategicMerge: - patch-route.yaml

Why Kustomize? ● Eliminates needless duplication of yaml ● Enables reuse through customization (patching) ● Hierarchical structure provides flexibility ○ Overlays can leverage other bases and overlays ○ Overlays can reference remote repositories ● Included with kubernetes ● Validates yaml before deployment ● Agnostic (works with: Argo CD, Flux, ACM, etc) GitOps Guide to the Galaxy

11 Helm is a package manager for Kubernetes applications define, install and update applications

12 How does Helm work? OPENSHIFT NAMESPACE Image Repository Releases Helm Chart (templates) Values (configs) Helm CLI

Helm Templates apiVersion: build.openshift.io/v1 kind: BuildConfig . . . spec: . . . source: type: Git git: uri: {{ .Values.build.uri }} contextDir: {{ .Values.build.contextDir }} {{- if and .Values.build.native.useDefaultDockerfile (eq .Values.build.mode “native”) }} dockerfile: |- FROM registry.redhat.com/quarkus/mandrel-20-rhel8 AS builder . . . {{- end }} strategy: {{- if eq .Values.build.mode “jvm” }} type: Source sourceStrategy: . . . apiVersion: apps/v1 kind: Deployment metadata: . . . spec: replicas: {{ .Values.deploy.replicas }} . . . template: spec: containers: - name: web image: {{ .Release.Name }}:{{ .Values.image.tag }} {{- if .Values.deploy.resources }} resources: {{- toYaml .Values.deploy.resources | nindent 12 }} {{- end }} . . .

Configuration using “Values” build: uri: https://github.com/deweya/quarkus-quickstarts contextDir: getting-started mode: jvm deploy: resources: limits: cpu: 500m memory: 2Gi Create a values.yaml file $ helm install quarkus-app redhat-charts/quarkus --values values.yaml Install the Helm chart BuildConfig ImageStream

Things to look out for! ● Helm ○ There are two ways to use Helm with Argo CD ○ Remember that GitOps is declarative ○ Keep a single source of Truth ● Kustomize ○ There will be a LOT of kustomization.yaml files ○ It’s not an “either or” with Kustomize/Helm Using Kustomize will dictate your GitOps repo structure! GitOps Guide to the Galaxy

Best Practice: Repo Structure Okay, now what? GitOps Guide to the Galaxy 16 Save it once and then forget about it!

GitOps Repo Structure It’s not all unicorns and rainbows ● Repo structure for manifests: ○ Monorepo? ○ Separate repos for environments? ○ Repo for specific clusters? ● Argo CD has scale issues with Monorepo ● Order dependent deployments and non-declarative requirements ○ Syncwaves/Hooks ● The way your Organization is laid out will come into play.

Approach 1: Multiple repositories /taxi-config-stage.git /taxi-config-test.git /taxi-config-prod.git /taxi-config-dev.git /taxi.git

Approach 2 : Single Repository ├── apps │ └── app-1 │ ├── base │ │ └── kustomization.yaml │ └── dev │ ├── deployment.yaml │ └── kustomization.yaml ├── envs │ ├── base │ │ ├── 205-serviceaccount.yaml │ │ └── kustomization.yaml │ └── dev │ └── kustomization.yaml └── services └── service-a ├── base │ ├── config │ │ ├── 300-deployment.yaml │ │ ├── 310-service.yaml │ │ └── kustomization.yaml │ └── kustomization.yaml └── dev ├── dev-deployment.yaml ├── dev-service.yaml └── kustomization.yaml ├── base │ ├── deployment.yaml │ ├── kustomization.yaml │ └── service.yaml └── overlays ├── development │ ├── kustomization.yaml │ └── replicas.yaml ├── production │ ├── kustomization.yaml │ ├── replicas.yaml │ └── volumes.yaml └── staging ├── kustomization.yaml └── volumes.yaml ├── 00-tekton │ ├── release.notags.yaml │ └── release.yaml ├── 01-namespaces │ ├── cicd-environment.yaml │ ├── dev-environment.yaml │ └── stage-environment.yaml ├── 02-serviceaccount │ ├── demo-sa-admin-dev.rolebinding.yaml │ ├── demo-sa-admin-stage.rolebinding.yaml │ ├── role-binding.yaml │ ├── role.yaml │ └── serviceaccount.yaml ├── 03-tasks │ ├── buildah-task.yaml │ ├── create-github-status-task.yaml │ ├── deploy-from-source-task.yaml │ └── deploy-using-kubectl-task.yaml ├── 04-templatesandbindings │ ├── dev-cd-deploy-from-master-binding.yaml │ ├── dev-cd-deploy-from-master-template.yaml │ ├── dev-ci-build-from-pr-binding.yaml │ ├── dev-ci-build-from-pr-template.yaml │ ├── stage-cd-deploy-from-push-binding.yaml │ ├── stage-cd-deploy-from-push-template.yaml │ ├── stage-ci-dryrun-from-pr-binding.yaml │ └── stage-ci-dryrun-from-pr-template.yaml ├── 05-ci │ ├── dev-ci-pipeline.yaml │ └── stage-ci-pipeline.yaml ├── 06-cd │ ├── dev-cd-pipeline.yaml │ └── stage-cd-pipeline.yaml ├── 07-eventlisteners │ └── cicd-event-listener.yaml └── 08-routes └── github-webhook-event-listener.yaml Rember: Argo CD has issues with Monorepos

Tips and Things to Ask yourself ● Tips ○ Utilize kustomize to refer to other Git repos ○ Make sure it’s easily repeatable ○ Keep application repos separate from deployment repos ● Ask yourself… ○ Do we have environment specific configurations? ○ Do we have cluster specific configurations? ○ Do we have a mixture of both? GitOps Guide to the Galaxy ☝

Fan Out/Cascade Approach GitOps Guide to the Galaxy Base Configuration (Every cluster gets this) DEV Stage PROD Cluster 1 Cluster 2

Questions? Comments? Concerns? GitOps Guide to the Galaxy 22 Be safe out there!