Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Who Owns the Upgrade? Automating Fleet-Wide Hel...

Who Owns the Upgrade? Automating Fleet-Wide Helm Rollouts Without Breaking Prod

Many Platform Engineering teams maintain a set of abstractions which development teams use to deploy their applications - often in the shape of a Helm chart.
Even after the chart is mature, the Platform team needs to introduce bug fixes, security patches, the occasional improvement.
For an app team to onboard each of this changes, they need to read the changelog, decide if it's safe, bump a version, run a deploy.

In Banking Circle, we had around 200 applications across 25 teams. With an average of 3 chart releases a month and 30 min needed to onboard each release on any app, we were wasting hundreds of engineer-hours a month. Moreover, as you can imagine, not every update reached every app, so fleets quietly drifted out of date. It also left us non-compliant, because security wanted periodic redeploys nobody had time to run.

So we flipped ownership. This talk is about the Blueprint Update Orchestrator (BUO): a GitOps-native system that makes the Platform team own chart versions, and pushes non-breaking upgrades to hundreds of apps automatically, without ever touching a team's `values.yaml`, and without opening a PR.

BUO stages rollouts through test apps, dev, and regional prod with change-approval gates, and every change lands as a signed Git commit with a full audit trail. Only minor/patch versions auto-apply, breaking changes still go through an explicit migration path, and any team can opt out.

Pushing chart upgrades to production on a schedule sounds reckless. It turned out to be safer. By upgrading continuously instead of in rare big jumps, we surfaced bugs minutes after they were introduced.
We'll share the skepticism we hit, how we hardened testing before flipping the switch, and how reluctant developers came around once the system kept quietly working.

Juan Herreros

October 28, 2026

Transcript

43 slides
  1. 28 October 2026 Who Owns the Upgrade? Automating Fleet-Wide Helm

    Rollouts Without Breaking Prod NEXT GEN BANKING. POWERED BY THE LATEST TECHNOLOGY.
  2. A bank built for payments. Infrastructure built for scale. The

    scale, reach and regulatory foundation to support your global operations. €1.5 440+ 45+ 10+ Payment volumes processed annually Payments processed in 2026 Virtual accounts issued Clients deposits held Regulated by ISO27001 Certified Trillion 950+ Financial institution clients Million 14 Offices Worldwide Including Luxembourg, London, Denmark, Australia, Czech Republic, Germany, Japan (representative office), Norway, Poland, Singapore (subsidiary payment institution), Sweden, UK and Liechtenstein Million CSSF, FCA, PRA, BaFin, FI, Finanstilsynet, MAS, ASIC, AUSTRAC, APRA, CNB, KNF, MNB, and FMA Liechtenstein Billion
  3. Our Office Locations NORWAY SWEDEN CANADA DENMARK POLAND GERMANY USA

    UNITED KINGDOM JAPAN CZECH REPUBLIC LIECHTENSTEIN INDIA HONG KONG SINGAPORE AUSTRALIA
  4. Our unique tech platform allows us to be the leading

    payments infrastructure player globally Single, multi-tenant tech stack Holistic, API-enabled value proposition Own core banking infrastructure with all clearing connections integrated into a single, multi-tenant, cloud-based technology platform Comprehensive set of market-leading financial infrastructure solutions, including payments, accounts and FX Built for real-time transactions at scale and liquidity management Seamless application API integration
  5. Atlas A Platform to Run a Bank On • Many

    different teams: UI, API, Core Banking, Clearing, FX, Data… • • Cloud Native, Multi-region Compute, Messaging, State, Configuration, Observability, API management
  6. Atlas Compute (Kubernetes) A Platform to Run a Bank On

    • Based on AKS • We install different “add-ons” on top: Crossplane, Kyverno, Grafana, ArgoCD, etc • Multi-tenant: Teams provision and manage their own tenants, via self-service • We provide a “Blueprint” for teams to deploy their applications.
  7. Atlas Compute (Kubernetes) A Platform to Run a Bank On

    • Multi-tenant: Teams provision and manage their own tenants, via self-service • We provide a “Blueprint” for teams to deploy their applications.
  8. The Blueprint • Templates used to deploy any application •

    Workload • Networking • Identity • Volumes •…
  9. Helm “Helm Charts describe even the most complex apps, provide

    repeatable application installation, and serve as a single point of authority.” “Helm Charts are easy to version, share, and host on public or private servers.” Ref: helm.sh
  10. Helm blueprint | simplified APPLICATION Deployment / Rollout Clients Ingress

    / HTTPRoute ConfigMaps / Secrets Service PVC / PV Pods Workload identity PDB / egress policy
  11. Helm blueprint | exhaustive Helm Controller-created Existing In-pod AZURE &

    EXTERNAL Clients Traffic Manager Routing PrivateDNS Managed identity PrivateDNS TrafficManager ARecord Profile TrafficManager TrafficManager Regional ExternalEndpoint Endpoint NEU / WEU load balancer XFederationV2 XAppGroupMemberV2 Deployment Rollout ServiceAccount HTTPRoute ReplicaSet stable BackendConfig Policy Pods ExternalSecrets Secrets ConfigMap Secret Oracle Wallet Oracle Wallet ConfigMaps / Secrets Projected volume existing references Oracle Wallet tcpdump / uploader PersistentVolumes PersistentVolume Claims app / capture tmp / capture cache ClusterRole extraObjects PodDisruptionBudget individual / shared TrafficPolicy Service Canary Ingress SecretStores app / env / volumes / capture ConfigMap metadata Container Registry Configuration & storage XIdentityV2 ConfigMap TLS Secrets Azure Files / SMB Identity & workload CNAMERecord Ingress Key Vault Alloy Service CiliumNetwork canary Policy app / capture emptyDir PLATFORM, API ACCESS & EXTENSIONS Gateway NGINX Crossplane / TMO provider configs External Secrets CSI drivers StorageClass Role RoleBindings x 2 lease-access
  12. Release process Commit to cluster Azure DevOps pipeline commit Developer

    shared templates commit manifests trigger App repo Build scan.yml push.yml helm.yml GitOps repo code + values test + image Trivy + Copacetic push + sign tag + render desired manifests read desired state signed image sync + wait Kubernetes sync Image registry Azure ACR Argo CD reconcile pull image Ref: The Rendered Manifests Pattern – Akuity Blog Application running pods
  13. Release process Commit to cluster Azure DevOps pipeline commit Developer

    shared templates commit manifests trigger App repo Build scan.yml push.yml helm.yml GitOps repo code + values test + image Trivy + Copacetic push + sign tag + render desired manifests read desired state signed image sync + wait Kubernetes sync Image registry Azure ACR Argo CD reconcile pull image Ref: The Rendered Manifests Pattern – Akuity Blog Application running pods
  14. Updating takes time • 200 applications • 3 updates a

    month • 30 min to adopt an update = 300 hours a month
  15. Requisites • Push & pull upgrades • Opt-out • Keep

    values.yaml as interface • Keep rendered manifests as source of truth
  16. Alternatives kro Ref: GitOps and the Manifest Dilemma: Helm, Kustomize,

    Crossplane, Kro, and Beyond - Dag Bjerre Andersen, KubeCon NA 2025
  17. Inside BUO Values files Policies repo Blueprint repo write version

    Local files chart render-manifest BUO CLI Policy engine render-manifest resolve version Chart provider Helm SDK Manifests render locally split + save push apps + values + GitOps source metadata GitOps repo auto-upgrade BUO CLI Connect Scan namespaces auto-upgrade AKS API foundation=v3 Discover apps ConfigMaps + AppSets Blueprint Update Orchestrator query cluster Kubernetes AKS
  18. Inside BUO Values files Policies repo Blueprint repo write version

    Local files chart render-manifest BUO CLI Policy engine render-manifest resolve version Chart provider Helm SDK Manifests render locally split + save push Blueprint Update Orchestrator GitOps repo auto-upgrade
  19. Inside BUO Values files Policies repo Blueprint repo write version

    Local files chart render-manifest BUO CLI Policy engine render-manifest resolve version Chart provider Helm SDK Manifests render locally split + save push apps + values + GitOps source metadata GitOps repo auto-upgrade BUO CLI Connect Scan namespaces auto-upgrade AKS API foundation=v3 Discover apps ConfigMaps + AppSets Blueprint Update Orchestrator query cluster Kubernetes AKS
  20. BUO: user-initiated Blueprint Policies Kubernetes deploy commit Developer App repo

    code + values commit manifests BUO sync GitOps repo render-manifest Application Argo CD running pods Application pipeline sync + wait
  21. BUO: platform-initiated Deployed values Blueprint Kubernetes publish policy change Platform

    Policies repo targetVersion commit manifests BUO sync GitOps repo auto-upgrade Application Argo CD running pods Policy pipeline watch health
  22. Upgrades (As of Oct 28th) • 25 rollouts • 12,975

    applications updated • 2,841 in production • 1,420 hours or ~9 months of a Full-Time Employee
  23. Thank you! FEEDBACK NEXT GEN BANKING. POWERED BY THE LATEST

    TECHNOLOGY. LinkedIn JuanHerreros.com GitHub

More Decks by Juan Herreros

Other Decks in Programming