GitOps on AWS: Codifying Multi-cloud Operations

Mahmoud Saada Customer Reliability Engineer @saadazzz @saada

● Roles: SE, TL, SRE, CRE ● Startups and Enterprises ● Industries: HR, AI, Fin-tech, Infrastructure ● Companies: TargetCW, ADP, Agolo, Weaveworks ● Meetups: Docker NYC, Uber Tracing NYC ● Open Source: grafana, prometheus, terraform, helm, helm operator, eksctl, elasticsearch, … ● Certified Kubernetes Administrator Background

● Gitops ● Operator pattern ● Cluster Management ● Demo ● Q&A Agenda

● Gitops ● Operator pattern ● Cluster Management ● Demo ● Q&A Agenda

Definition

GitOps is the practice of using Git to declaratively define desired state and Continuous Delivery agents (Flux) to automate the reconciliation of current state to desired/intended state; effectively decoupling CI and CD GitOps Definition

Weaveworks’s Gitops Tools ● Flux is an async-pull continuous delivery agent that runs as a controller inside its target Kubernetes cluster. Flux pulls git changes and converges them with a Kubernetes state. It was created at Weaveworks. ● Helm Operator is a Kubernetes operator that watches for a CRD called HelmRelease and renders it into k8s resources

Weaveworks: The GitOps Company Weaveworks created the GitOps methodology and tooling to solve our own Kubernetes management, scalability, and reliability requirements Weaveworks is a key partner with all the major infrastructure and Kubernetes vendors Weaveworks is deeply committed to the Open Source Community Weaveworks is backed by solid investors

1 The entire system is described declaratively 2 The canonical desired system state is versioned (with Git) 3 Approved changes to the desired state are automatically applied to the system 4 Software agents ensure correctness and alert on divergence GitOps Principles 10

GitOps – An Operating Model for Cloud Native Deployment (clusters, apps) Monitoring Logging (Observability) Management (operations) Build GIT Test IDE Unifying Deployment, Monitoring and Management. Git as the single source of truth of a system’s desired state ALL intended operations are committed by pull request ALL diffs between intended and observed state with automatic convergence ALL changes are observable, verifiable, and auditable “Immutability Firewall” Kubernetes GitOps Continuous Integration

Development Experience -

GitOps Patterns

One git path/directory per namespace Namespace Strategy /namespaces ./team1 ./team2 ./team3 Team1 namespace Team2 namespace Team3 namespace

One git path/directory per environment Directory Path Strategy /env ./develop ./staging ./production Develop cluster staging cluster Production cluster

One branch per environment Branching Strategy develop staging production Develop cluster staging cluster Production cluster

One git path/directory per environment, one subdirectory per namespace Mixing Strategies together /env ./develop ././team1 ././team2 ./staging ././team1 ././team2 ./production ././team1 ././team2 Develop cluster broken down by namespace Staging cluster broken down by namespace Production cluster broken down by namespace

Benefits

● Increased productivity ● Enhanced developer experience ● Improved stability ● Higher reliability ● Consistency and standardization ● Stronger security guarantees Benefits of adopting GitOps

Common Concerns

● Not designed for programmatic updates → feature branches + test environments ● Proliferation of Git repositories → Consolidate gitops repos + GoTK ● Lack of visibility → commit messages, flux logs, GoTK grafana dashboard ● Doesn’t solve centralized secret management → Sealed Secrets, Hashicorp Vault, AWS SecretsManager ● Auditing isn’t as great as it sounds like → commit messages, flux logs, GoTK grafana dashboard ● Lack of input validation → CI linting Common Concerns

● Gitops ● Operator pattern ● Cluster Management ● Demo ● Q&A Agenda

The Operator Pattern Current State Watch Desired State Reconcile Controller

Current State Watch Desired State Reconcile Controller Current State (etcd) Watch Desired State (etcd) Reconcile K8S Current State (git) Watch Desired State (etcd) Reconcile GitOps Controllers Everywhere

Reconciliation Loops Current State Desired State

● Gitops ● Operator pattern ● Cluster Management ● Demo ● Q&A Agenda

Helm Operator Custom Operator WKP / ClusterAPI state

➔ Kubernetes application platform ➔ Management of cluster and applications ➔ Builds on GitOps and adds enterprise features ➔ Define clusters and components using a model based system ➔ Deploy new clusters using those definitions: multiple back-ends ➔ Alerting and operations built-in Weave Kubernetes Platform

● Gitops ● Operator pattern ● Cluster Management ● Demo ● Q&A Agenda

eksctl create cluster

kind create cluster

Let’s see it in action EC2 (CAPI Managed) EC2 (CAPI Managed) Kind (on-prem Unmanaged) EKS (management cluster)

Demo

Gitops Workshops (Module 7) https://weaveworks-gitops.awsworkshop.io/ Do try this at home!

Future

Future ● Gitops Toolkit ○ FluxV2, Helm Controller, Kustomize Controller, Notifications, Prometheus metrics, Grafana dashboard, ... ○ Major release expected later this year ○ You can find documentation and quickstart examples here: https://toolkit.fluxcd.io ○ Helm Controller will be the successor for Helm Operator ○ Community-driven. If interested in getting involved, go to https://github.com/fluxcd/toolkit/discussions

● Gitops ● Operator pattern ● Cluster Management ● Demo ● Q&A Agenda

Questions?

Thank you! Demo repo: https://github.com/saada/gitops-cluster-management/ Workshop (Module 7): https://weaveworks-gitops.awsworkshop.io/ EKS Control: https://eksctl.io Flux: https://docs.fluxcd.io/ Helm Operator: https://docs.fluxcd.io/projects/helm-operator/ The Art of Modern Ops (podcast): https://bit.ly/weave-podcast FluxV2 (coming soon): https://toolkit.fluxcd.io/ Mahmoud Saada @saada @saadazzz weave-community.slack.com