AWS MENA Community Day - GitOps on AWS_ Codifying multi-cloud operations

AWS MENA Community Day - GitOps on AWS_ Codifying multi-cloud operations By Mahmoud Saada
@saadazzz

Transcript

1. GitOps on AWS: Codifying Multi-cloud Operations
2. Mahmoud Saada Customer Reliability Engineer @saadazzz @saada
3. • Roles: SE, TL, SRE, CRE • Startups and Enterprises • Industries: HR, AI, Fin-tech, Infrastructure • Companies: TargetCW, ADP, Agolo, Weaveworks • Meetups: Docker NYC, Uber Tracing NYC • Open Source: grafana, prometheus, terraform, helm, helm operator, eksctl, elasticsearch, … • Certified Kubernetes Administrator Background
4. • Gitops • Operator pattern • Cluster Management • Demo • Q&A Agenda
5. • Gitops • Operator pattern • Cluster Management • Demo • Q&A Agenda
6. Definition
7. GitOps is the practice of using Git to declaratively define desired state and Continuous Delivery agents (Flux) to automate the reconciliation of current state to desired/intended state; effectively decoupling CI and CD GitOps Definition
8. Weaveworks’s Gitops Tools • Flux is an async-pull continuous delivery agent that runs as a controller inside its target Kubernetes cluster. Flux pulls git changes and converges them with a Kubernetes state. It was created at Weaveworks. • Helm Operator is a Kubernetes operator that watches for a CRD called HelmRelease and renders it into k8s resources
9. Weaveworks: The GitOps Company Weaveworks created the GitOps methodology and tooling to solve our own Kubernetes management, scalability, and reliability requirements Weaveworks is a key partner with all the major infrastructure and Kubernetes vendors Weaveworks is deeply committed to the Open Source Community Weaveworks is backed by solid investors
10. 1 The entire system is described declaratively 2 The canonical desired system state is versioned (with Git) 3 Approved changes to the desired state are automatically applied to the system 4 Software agents ensure correctness and alert on divergence GitOps Principles 10
11. GitOps – An Operating Model for Cloud Native Deployment (clusters, apps) Monitoring Logging (Observability) Management (operations) Build GIT Test IDE Unifying Deployment, Monitoring and Management. Git as the single source of truth of a system’s desired state ALL intended operations are committed by pull request ALL diffs between intended and observed state with automatic convergence ALL changes are observable, verifiable, and auditable “Immutability Firewall” Kubernetes GitOps Continuous Integration
12. Development Experience -
13. GitOps Patterns
14. One git path/directory per namespace Namespace Strategy /namespaces ./team1 ./team2 ./team3 Team1 namespace Team2 namespace Team3 namespace
15. One git path/directory per environment Directory Path Strategy /env ./develop ./staging ./production Develop cluster staging cluster Production cluster
16. One branch per environment Branching Strategy develop staging production Develop cluster staging cluster Production cluster
17. One git path/directory per environment, one subdirectory per namespace Mixing Strategies together /env ./develop ././team1 ././team2 ./staging ././team1 ././team2 ./production ././team1 ././team2 Develop cluster broken down by namespace Staging cluster broken down by namespace Production cluster broken down by namespace
18. Benefits
19. • Increased productivity • Enhanced developer experience • Improved stability • Higher reliability • Consistency and standardization • Stronger security guarantees Benefits of adopting GitOps
20. Common Concerns
21. • Not designed for programmatic updates → feature branches + test environments • Proliferation of Git repositories → Consolidate gitops repos + GoTK • Lack of visibility → commit messages, flux logs, GoTK grafana dashboard • Doesn’t solve centralized secret management → Sealed Secrets, Hashicorp Vault, AWS SecretsManager • Auditing isn’t as great as it sounds like → commit messages, flux logs, GoTK grafana dashboard • Lack of input validation → CI linting Common Concerns
22. • Gitops • Operator pattern • Cluster Management • Demo • Q&A Agenda
23. The Operator Pattern Current State Watch Desired State Reconcile Controller
24. Current State Watch Desired State Reconcile Controller Current State (etcd) Watch Desired State (etcd) Reconcile K8S Current State (git) Watch Desired State (etcd) Reconcile GitOps Controllers Everywhere
25. Reconciliation Loops Current State Desired State
26. • Gitops • Operator pattern • Cluster Management • Demo • Q&A Agenda
27. Helm Operator Custom Operator WKP / ClusterAPI state
28. ➔ Kubernetes application platform ➔ Management of cluster and applications ➔ Builds on GitOps and adds enterprise features ➔ Define clusters and components using a model based system ➔ Deploy new clusters using those definitions: multiple back-ends ➔ Alerting and operations built-in Weave Kubernetes Platform
29. • Gitops • Operator pattern • Cluster Management • Demo • Q&A Agenda
30. eksctl create cluster
31. kind create cluster
32. Let’s see it in action EC2 (CAPI Managed) EC2 (CAPI Managed) Kind (on-prem Unmanaged) EKS (management cluster)
33. Demo
34. Gitops Workshops (Module 7) https://weaveworks-gitops.awsworkshop.io/ Do try this at home!
35. Future
36. Future • Gitops Toolkit ◦ FluxV2, Helm Controller, Kustomize Controller, Notifications, Prometheus metrics, Grafana dashboard, ... ◦ Major release expected later this year ◦ You can find documentation and quickstart examples here: https://toolkit.fluxcd.io ◦ Helm Controller will be the successor for Helm Operator ◦ Community-driven. If interested in getting involved, go to https://github.com/fluxcd/toolkit/discussions
37. • Gitops • Operator pattern • Cluster Management • Demo • Q&A Agenda
38. Questions?
39. Thank you! Demo repo: https://github.com/saada/gitops-cluster-management/ Workshop (Module 7): https://weaveworks-gitops.awsworkshop.io/ EKS Control: https://eksctl.io Flux: https://docs.fluxcd.io/ Helm Operator: https://docs.fluxcd.io/projects/helm-operator/ The Art of Modern Ops (podcast): https://bit.ly/weave-podcast FluxV2 (coming soon): https://toolkit.fluxcd.io/ Mahmoud Saada @saada @saadazzz weave-community.slack.com