Building a Bank from Scratch in the Cloud on Kubernetes

Speaker: Hans Kristian Flaatten

Description:
We have built the next generation mobile bank for Norway in the cloud from scratch! This is the story of how we build Bulder Bank from the ground up in 10 months.
Early on it was decided that we had to be open for radical new ideas if we was to succeed. Important criteria was speed, modern and automated processes and future-proofing. In addition we wanted to make something that was cool and motivated everyone on the team to work on.
The choice landed on Google Kubernetes Engine and Google Firebase and the possibilities this gives us are literally endless! And it gives us the opportunity to move fast while having a more or less fully managed environment that is always up and running!
We have decided to model payments entirely event based and asynchronously where multiple microservices written in Go each do their part before handing over the transaction to the next service over a message queue.

Bio:
Hans Kristian is DevOps Practice Lead at EVRY where he work with some of EVRY's largest customers within financial, insurance, and public sector to deliver higher quality software, faster, using agile methodologies and DevOps practices. He specialises in test- and deployment-automation, large scale container systems using Kubernetes, and distributed software architecture.
Apart from his day job, Hans Kristian is an avid Open Source and Open Data evangelist. He is an acting member of the Node.js Foundation where he manages test and release of official Node.js versions and the official Docker Image for Node.js.
Hans Kristian previously worked as a Systems Developer / Systems Architect at the Norwegian Trekking Association (DNT) where he lead thetransition to a microservice oriented architecture on immutable infrastructure and Docker-containers.

Transcript

1. #CloudNativeNordics Starefossen 1 Building a Bank from scratch* HANS KRISTIAN FLAATTEN CLOUD NATIVE COPENHAGEN, JANUARY 2020
2. #CloudNativeNordics Starefossen 2 Hans Kristian Flaatten • From Norway ! • Married, two boys (0b011 && 0b101) • DevOps Practice Lead @ TietoEVRY • Node.js Foundation Member • Open Source Contributor ❤
3. #CloudNativeNordics Starefossen 3 $ go get github.com/bulderbank/ndc-2019 $ go run agenda.go • Bulder, what?! • Architecture • Build and Deploy • Runtime • Operations $
4. #CloudNativeNordics Starefossen 4 BULDER BANK
5. #CloudNativeNordics Starefossen 5 BULDER BANK Awesome mobile experience Competitive mortgages with automatic rent adjustments Value adds and integrations   
6. #CloudNativeNordics Starefossen 6 Minimum Viable Product Onboarding Transactions My Cards Push notifications Payments Invoice scanning Subscriptions       Transactions 2.0   My Economy 
7. #CloudNativeNordics Starefossen 7 The Team Cloud Engineers Backend iOS Android CTO Frontend Design / UX
8. #CloudNativeNordics Starefossen 8 Open Banking @ Sparebanken Vest
9. #CloudNativeNordics Starefossen Deliver Value Continuously Experiment & Learn Rapidly Make Safety a Prerequisite 9 Key Principles Make People Awesome
10. #CloudNativeNordics Starefossen 10 Timeline aug jan jun 2018 2019 sep (beta) (public)
11. #CloudNativeNordics Starefossen 11 Timeline 10 MONTHS
12. #CloudNativeNordics Starefossen 12
13. #CloudNativeNordics Starefossen 13 GKE Firebase
14. #CloudNativeNordics Starefossen 14 Bulder Bank GKE App Firebase Open Banking
15. #CloudNativeNordics Starefossen 15 ! ! ! ! ! ! ! ! ! Firestore Functions Authentication Crashlytics Performance Monitoring Test Lab Messaging Analytics Remote Config Firebase
16. #CloudNativeNordics Starefossen 16 Firestore
17. #CloudNativeNordics Starefossen 17 Firestore
18. #CloudNativeNordics Starefossen 18 No Schema – No Problems ?
19. #CloudNativeNordics Starefossen 19 No Schema – No Problems new
20. #CloudNativeNordics Starefossen 20 OpenAPI Write Verify Deploy definitions: user: type: object properties: created: type: string format: date … userInfo: $ref: '#/definitions/contact' contact: type: object properties: …
21. #CloudNativeNordics Starefossen 21 OpenAPI definitions: user: type: object properties: created: type: string format: date … userInfo: $ref: '#/definitions/contact' contact: type: object properties: … Account v1 Card v1 v2 Payment v1
22. #CloudNativeNordics Starefossen 22 OpenAPI act' Account v1 Card v1 v2 Payment v1 ! ! ! !
23. #CloudNativeNordics Starefossen • Native app written in Kotlin • Gradle for dependencies • Android JetPack (AndroidX) • Coroutines • CircleCI → Fastlane → Google Play 23 • Native app written in Swift 5 • CocoaPods for dependencies • Model–view–viewmodel (MVVM) • Storyboard-reference • CircleCI → Fastlane → TestFlight ! !
24. #CloudNativeNordics Starefossen 24 Kubernetes
25. #CloudNativeNordics Starefossen 25 Kubernetes Greek for "Helmsman" • Container orchestrator • Based on Google's internal systems • Runs on cloud or bare-metal • Supports multiple container runtimes
26. #CloudNativeNordics Starefossen 26 API
27. #CloudNativeNordics Starefossen 27 API
28. #CloudNativeNordics Starefossen 28 API
29. #CloudNativeNordics Starefossen 29 API
30. #CloudNativeNordics Starefossen 30 API
31. #CloudNativeNordics Starefossen 31 One-Click Managed Kubernetes Amazon EKS Microsoft AKS Google GKE
32. #CloudNativeNordics Starefossen 32 Kubernetes in a Nutshell API First Modular Self Healing  
33. #CloudNativeNordics Starefossen 33 Bulder Bank GKE Open Banking App Firebase GCP Project GCP Project
34. #CloudNativeNordics Starefossen PubSub 34 Bulder Bank Functions GKE App NATS
35. #CloudNativeNordics Starefossen 35 NATS • High performance messaging system • Publish-subscribe • Atleast once-delivery • Static binary written in Go • First release in 2013 • Official CNCF Project
36. #CloudNativeNordics Starefossen 36 Kubernetes Operators • Building Kubernetes native applications • Extends the Kubernetes API • Runs inside Kubernetes • Built for a specific application • Runs inside Kubernetes • Lifecycle; scaling, upgrading, and recovery • Well suited for complex applications apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: prometheus spec: serviceAccountName: prometheus serviceMonitorSelector: matchLabels: team: frontend resources: requests: memory: 400Mi enableAdminAPI: false
37. #CloudNativeNordics Starefossen NATS Operator Payment Backend 37 NATS Operator NATS Operator Deployment NATS Clustere Deployment NATS Cluster Deployment Payment Backend Deployment Payment SA ServiceAccount  Payment Role NatsServiceRole NATS Token Secret  NATS Cluster NATS Clustere Deployment NATS Cluster Deployment NATS Cluster Deployment NATS Cluster NatsCluster NATS Cluster Service NATS CRD CustomResource 
38. #CloudNativeNordics Starefossen 38 Cloud Functions → NATS VPC Firebase Functions GKE
39. #CloudNativeNordics Starefossen 39 Cloud Functions → NATS VPC Serverless VPC Accessor connector Firebase Functions GKE
40. #CloudNativeNordics Starefossen 40 Cloud Functions → NATS VPC-1 Serverless GW Firebase Functions GKE europe-west1 europe-north1 VPC-2 VPC Network Peering NATS
41. #CloudNativeNordics Starefossen 41 Cloud Functions → NATS Firebase Functions GKE europe-west1 europe-north1 NATS Load Balancer  TLS v1.3 X.509 Token 
42. #CloudNativeNordics Starefossen 42 What if we could run Serverless Functions inside Kubernetes?
43. #CloudNativeNordics Starefossen 43 Knative • Serverless Containers on Kubernetes • Built by Google, IBM, RedHat++ • Runs on top of Istio • Build – source-to-container • Serving – run serverless containers • Eventing – universal subscriptions • Managed Knative with Google Cloud Run
44. #CloudNativeNordics Starefossen 44 Backend Services • Go v1.12 • APIs and functions • gRPC (and some REST) • No framework! • Node.js v10 LTS • BulderBank.no • Administration ! !
45. #CloudNativeNordics Starefossen 45 Software Delivery Pipeline • All changes must go through Git! • CircleCI is automatically starts on push • Audit and traceability for all changes Deploy Push Build Test
46. #CloudNativeNordics Starefossen 46 Software Delivery Pipeline • Fetch and install dependencies • Automated and reproducible! Deploy Push Build Test
47. #CloudNativeNordics Starefossen 47 Software Delivery Pipeline • Syntax linting (go fmt) • Unit testing, integration testing • Source code scanning (SonarCloud) • Dependency scanning (Snyk) Deploy Push Build Test
48. #CloudNativeNordics Starefossen 48 Software Delivery Pipeline • Build and sign images • Deploy and promote (Helm) dev → test → prod • No cuts, No buts, No coconuts Deploy Push Build Test
49. #CloudNativeNordics Starefossen 49 Add new features #123 Open ! Developer Dave wants to merge 1 commit into master from new-feature Commits Files changed Conversation 1 This pull request adds all the new features to our awesome application Now we are ready to rock our competitors Developer Dave commented 23 minutes ago Reviewers x Security Sarha requested changes 17 minutes ago Security Sarha x Add new backend functionality af231ce ! Update roles based on feedback from Sarha ad77b09 ! ! ! 2 ! 5 Security Sarha approved these changes 6 minutes ago ! ! 3 Teamlead Tom approved these changes 4 minutes ago ! Teamlead Tom ! Add new security roles de0230f ! 2 3 4 !
50. #CloudNativeNordics Starefossen 50 Add new features #123 Open ! Developer Dave wants to merge 1 commit into master from new-feature Commits Files changed Conversation 1 This Pull Request adds all the new features to our awesome application Now we are ready to rock our competitors Developer Dave commented 23 minutes ago Reviewers x Security Bot requested changes 22 minutes ago Security Bot Add new backend functionality af231ce ! Update roles based on feedback from Security Bot ad77b09 ! ! ! 2 ! 5 Security Bot approved these changes 6 minutes ago ! ! 3 Teamlead Tom approved these changes 4 minutes ago ! Teamlead Tom ! Add new security roles de0230f ! 2 3 4 !
51. #CloudNativeNordics Starefossen 51 255 Build Hours Month Android Go Functions iOS Web TF
52. #CloudNativeNordics Starefossen 52 Applications are 'assembled' • Not all are created equal, some are healthy and some are not • All go bad over time, they age like milk, not like wine • Enterprises consumes on average 229.000 software components annually, of which 17.000 had a known security vulnerability. Libraries 80 % Source Code 20 %
53. #CloudNativeNordics Starefossen 53 Bulder Bank Functions GKE App NATS
54. #CloudNativeNordics Starefossen 54 Bulder Bank Cloud APIs Endpoints
55. #CloudNativeNordics Starefossen 55 Bulder Bank Cloud APIs Istio
56. #CloudNativeNordics Starefossen 56 Istio • Service mesh control plane • Kubernetes native • Builds on top of Envoy proxy • Created by Google, IBM, RedHat, Lyft • Managed version in GKE
57. #CloudNativeNordics Starefossen 57 Why Istio? Microservices must deal with • Client side load balancing • Timeouts and retries • Monitoring and tracing • Traffic encryption • Traffic policies
58. #CloudNativeNordics Starefossen 58 What is a service mesh? Control Plane API Service A Proxy Service B Proxy HTTP/1.1, HTTP/2, gRPC or TCP -- with or without mTLS Policy checks, telemetry Config data to proxies TLS certs to proxies Pilot Mixer Citadel
59. #CloudNativeNordics Starefossen 59 Istio Observability
60. #CloudNativeNordics Starefossen 60 Site Reliability Engineering
61. #CloudNativeNordics Starefossen 61 Ben Treynor – Founder of Google's SRE Team “SRE is what happens when a software engineer is tasked with what used to be called operations”
62. #CloudNativeNordics Starefossen 62 Observability Logs Metrics Tracing
63. #CloudNativeNordics Starefossen 63 Observability Prometheus Grafana
64. #CloudNativeNordics Starefossen 64 Prometheus Monitoring system and TSDB: • Instrumentation • Metrics collection and storage • Querying, alerting, dashboarding • For all levels of the stack Auto discovery of new applications Database Prometheus Exporter kubernetes Discover services kubelet Grafana Service A Service B Query metrics Pull metrics Alertmanager Push alerts
65. #CloudNativeNordics Starefossen 65
66. #CloudNativeNordics Starefossen 66 Observability Stackdriver Prometheus OpenTelemetry Grafana
67. #CloudNativeNordics Starefossen 67
68. #CloudNativeNordics Starefossen 68 CALMS Culture Automation Metrics Sharing Lean
69. #CloudNativeNordics Starefossen 69 vs. Pets vs. Cattle
70. #CloudNativeNordics Starefossen 70 Infrastructure as Code resource "container_cluster" "apps" { name = "apps-cluster" location = "europe1-north" initial_node_clount = 1 } resource "container_node_pool" "nodes" { name = "apps-nodes" location = "europe1-north" cluster = "${container_cluster.apps.name}" node_config { machine_type = "n1-standard-1" } } apps-nodes … apps-cluster
71. #CloudNativeNordics Starefossen 71 Infrastructure as Code Workflow Code Commit Review Validate Deploy
72. #CloudNativeNordics Starefossen 72 Infrastructure Templates Make the right way the easy way! • Shared templates • Shared knowledge • Brest practices • Conventions • Security
73. #CloudNativeNordics Starefossen 73 Security Benefits of IaC • Change history and audit trail • Review and collaborate • Statical security analysis • Inventory tracking • Security knowledge sharing by transparency • Consistency and reproducibility
74. #CloudNativeNordics Starefossen 74 $ go get github.com/bulderbank/ndc-2019 $ go run sumary.go • From idea to production in 10 months • Building on the shoulders of giants • Kubernetes and Firebase $
75. #CloudNativeNordics Starefossen 75 Questions?
76. #CloudNativeNordics Starefossen 76 Thank you! Hans Kristian Flaatten hans.flaatten@evry.com @Starefossen