The 3 Layers of our Platform and Helmping Ourselves before helping others

A look into the structure of Banking Circle's Internal Developer Platform, and the open source tooling we have built for it.

Transcript

1. The 3 Layers of our Platform Helmping Ourselves before helping others Cloud Native Copenhagen, May 21st 2024
2. WHO ARE WE? Banking Circle 2
3. WHO ARE WE? Banking Circle 3
4. WHO ARE WE? As Platform Engineers, we support teams (~200 people) that build: • Client-facing APIs • Core Banking Systems • Internal reporting tools • Data • Integrations with other companies • … In the cloud (Azure), using different languages, architectures, development processes, etc Banking Circle 4
5. HISTORY Banking Circle 5
6. HISTORY • At some point, using only services from a public cloud is not feasible, due to several reasons: • Orchestration and scalability • Security and compliance • Observability • Cost • To avoid a “tennis match”, we need: • Developer Experience • Maintainability Banking Circle 6 Kubernetes Platform
7. GOALS • Reliability • Scale • Compliance • Security • Developer Experience • Standardization • Maintainability Banking Circle 7
8. 8 What’s in a Platform? Banking Circle 8
9. WHAT’S IN A PLATFORM? Banking Circle 9
10. 10 Layers approach Banking Circle 10
11. LAYER APPROACH Banking Circle 11
12. 12 Layers approach Tooling Banking Circle 12
13. TOOLING Banking Circle 13
14. TOOLING Banking Circle 14
15. TOOLING Banking Circle 15
16. TOOLING – INSTALLATION PROCESS Installation process • Terraform, ArgoCD, Helm, Helmper Banking Circle 16
17. 17 Layers approach Foundation Banking Circle 17
18. FOUNDATION Collection of resources needed to build on top – “Starter pack” • Multi-tenancy • Observability • Security - Guardrails • Networking Banking Circle 18
19. FOUNDATION All major Cloud providers (GCP, AWS, Azure) have their own definition • Points in common: IAM, Networking, Security, resource organization • Some also mention: Billing, governance, automation, disaster recovery, observability ChatGPT condensed and summarized their 3 definitions as: “A landing zone is a well-designed, scalable, and secure cloud environment that serves as a foundation for cloud operations, offering modular and subscription-based resource management.” Banking Circle 19 Landing Zone
20. FOUNDATION Banking Circle 20
21. FOUNDATION Each team gets: • Resources on Kubernetes • A namespace • An ArgoCD project • A Cilium network policy • A service account Banking Circle 21 • Resources on Azure, Azure DevOps, Entra Id • Credentials • Role assignments • Identities • DNS records • Etc
22. FOUNDATION - INTERFACE Banking Circle 22
23. FOUNDATION “New generation of tools that go beyond the traditional platform-as-a-service (PaaS) model and offer published contracts between developers and platform teams. The contract might involve provisioning cloud environments, databases, monitoring, authentication and more in a different environment. These tools enforce organizational standards while granting developers self-service access to variations through configuration.” Banking Circle 23 Platform Orchestrator Ref: Platform orchestration | Technology Radar | Thoughtworks
24. FOUNDATION Our Foundation is a “Platform Orchestrator” that creates “Landing Zones” • Namespaces • Crossplane • Terraform Banking Circle 24
25. QUICK RECAP As a Developer, after the Foundation layer, I have: • My code • A Foundation to build on Banking Circle 25
26. 26 Layers approach Delivery Banking Circle 26
27. DELIVERY This layer is optional, some developers know (and have opinions about) how to build and deploy their software For those who don’t, we need to provide a way of getting their code onto our clusters • Pipeline templates • Application blueprints Banking Circle 27
28. DELIVERY - PIPELINES Banking Circle 28
29. DELIVERY - BLUEPRINTS • Teams define their app in a few lines of YAML • We provide blueprints for the most common application topologies (e.g. a simple API, a cronjob, an app with persistent storage) Banking Circle 29
30. DELIVERY - BLUEPRINTS Banking Circle 30
31. SUMMARY Layer Development team input Mandatory Delivery No Foundation Yes Tooling None Yes Banking Circle 31
32. TECHNOLOGIES Banking Circle 32 Layer At Banking Circle we use… Alternatives Delivery Pipeline templates, Shuttle, ArgoCD OpenFunction Foundation Namespaces, Crossplane, Terraform vCluster, kcp, Kratix Tooling Kubernetes, ArgoCD, Trivy, Copa, Kyverno, Velero, Grafana, Prometheus, Cilium… Install through Helm, terraform, Helmper CNCF landscape CNOE reference implementation Install through Porter or a script
33. 33 Wait a moment… Where’s the UI? Banking Circle
34. ACCESS LAYER Banking Circle 34
35. ACCESS LAYER • Developer Portal • Single pane of glass At Banking Circle we use: No IDP (yet), Argo & Dynatrace Other Options: Backstage Banking Circle 35
36. 36 Conclusions Banking Circle
37. BENEFITS • Once the layers are clearly established, they are easier to maintain • It’s easy to understand the split of responsibilities between Development and Operations • Different teams can work on different layers Banking Circle 37
38. Helmper DevOps Demo, May 23th 2024
39. WHERE TO FIND US • devops.bankingcircle.com • DevOps Demos • Kubernetes Platform on Confluence Banking Circle Confidential 39
40. 40 Introduction to Helmper In 15 minutes, or less Banking Circle
41. WHAT’S IN IT FOR YOU • A new tool • Ensure good practice around storing dependencies for charts used in production • Easy and repeatable import of Container Images in Helm Charts • See an example of how to build Golang projects leveraging popular tools • Learn about recent CNCF projects in the domain of vulnerability detection and patching Banking Circle 41
42. AGENDA • The problem • The solution • The problem with the solution • The solution, continued • Demo Banking Circle 42
43. DEPLOYMENTS 1. Internal Applications o Shuttle o Bring-your-own 2. External Applications o Helm Charts Banking Circle 43
44. THE PROBLEM 1/3 • The platform team uses a significant number of Helm Charts to build the Kubernetes Platform in Banking Circle. Banking Circle 44
45. THE PROBLEM 2/3 • Multiple consequences of regulatory requirements to handle when operating in a bank: o No access to public internet in production o OS level vulnerabilities must be patched according to risk policy Banking Circle 45
46. THE PROBLEM 3/3 • The Platform Team would like to enable teams to use the best tool available for the job and to take ownership of operational aspects. • Apache Airflow for Data Engineers • Ollama • Etc.. Making developers understand Helm Charts including dependencies, security patching, image signing etc. is not feasible. Banking Circle 46
47. THE REQUIREMENTS • Store Helm Charts in OCI-compliant registries • Scan and Patch container images Menti: How do you solve this in your company? Banking Circle 47
48. HOW WE APPROACHED IT • Go program • Helm SDK • Azure and Azure DevOps integration • Azure DevOps pipeline to facilitate scanning, patching and distribution Banking Circle 48
49. THE SOLUTION Banking Circle 49
50. THE GOOD • Pipelines provide a convenient way to store artifacts for compliance • Selfhosted agents can run from virtual network for registry access • Solution was used to create images going into production for about a year Banking Circle 50
51. THE PROBLEM WITH THE SOLUTION • Built on pipelines and scripting • Difficult to test and develop • One pipeline run per image import does not scale with manual approvals before pushing to production • Huge resource overhead Banking Circle 51
52. BACK TO THE DRAWING BOARD… • Create one solution to the problem • Build a robust solution with wide compatibility and applicability • What we changed • Replaced dependency on Docker with Oras for OCI • Removed Azure specific code, instead relying on Helm/Docker authentication • Integrated all tools into Helmper • Rewrote the Helmper parsing logic • Goroutines – errgroups, slog etc. • Helmper got a new look • Convinced management to release the code to the community! Banking Circle 52
53. Banking Circle 53
55. WHAT 1. Complete backups of Helm Charts in your own registries to ensure availability 2. Analyze Helm Charts for OCI artifacts 3. Check status of images in registries 4. Secure container images by patching OS level vulnerabilities 5. Signing images 6. Everything above defined with a declarative spec Banking Circle 55
56. HOW • Helmper Core: Extracting image references from Helm Charts • Standing on the shoulders of giants • Integrate with all the tools through Go libraries Banking Circle 56
57. HOW IT WORKS (CORE) Banking Circle 57 1) Pull helm charts from remote registries 2) Analyze charts for image references 3) Check status of images in registries 4) Sync images
58. HOW IT WORKS Banking Circle 58 1) Pull charts 2) Analyses charts for image references 3) Check status of images in registries 4) Scan external 5) Patch 6) Scan 7) Push 8) Sign
59. DEMO: CORE • Interacting with Helm repositories through Helm SDK • Analyze artifacts with Helmper • Check image availability in registries with Oras • Push charts and images with Oras Banking Circle 59
61. DEMO: EXTENDED • Everything in previous demo, and … • Vulnerability detection with Trivy • Vulnerability patching with Copacetic • Signing with Cosign Banking Circle 61
63. NEXT STEPS • Grow userbase oDetermine if we are solving a problem oGet more companies involved • Roadmap oKubernetes Operator oSBOM oOpenTelemetry • CNCF Sandbox Application Banking Circle 63 https://github.com/christoffernissen/helmper Menti: Quick evaluation of Helmper
64. © Banking Circle, 2020 bankingcircle.com bankingcircle.com Thank you