How to Build a Digital Bank Using AWS

Monzo has built a bank for over 3.5 million customers and growing fast. It’s done so by leveraging open-source cloud native technology, all running on top of AWS. With over 100 engineers deploying 100s of changes to over 1500 microservices every day, Monzo has built an organisation which is able to ship and iterate fast.

From working with the banking regulators, to providing highly scalable, secure infrastructure primitives, AWS has been an obvious choice for the foundation of the bank. Services like EC2, S3, ECR have been in-place since the start, but as we’ve grown and matured, services like IAM, GuardDuty and AWS Shield have allowed to move both fast and in control.

We’ll explain how it all comes together to process payment when you pay with your Monzo card. Processing payments need to be done under tight latency and reliability tolerances. Nobody wants to wait for a long period of time while their bank determines whether to let their transaction through. We’ll talk about what it takes to run our own minimal physical infrastructure and how we’ve hooked that up into AWS using Direct Connect to securely and reliably transmit payments back and forth between us and the wider payment network.

Transcript

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. How to Build a Digital Bank Using AWS How Monzoprocesses payments and operates a bank in the cloud Chris Evans Platform Team Lead Suhail Patel Platform Engineer
6. Platfrom Team Lead @evnsio Senior Platform Engineer @suhailpatel We’re part of the Platform Team at Monzo Bank in the UK
7. Use your card at a store or online One of our DCs gets a process message Messages get passed through AWS Direct Connect VPC AWS Cloud Auto Scaling group K8s Worker Instances Auto Scaling group Cassandra Cluster Auto Scaling group Messages go through dozens of microservices running on Kubernetes Services use Cassandra for persistent data storage Services leverage etcd for distributed locking and coordination NSQ is used for unordered queuing and event publishing Prometheus is used for monitoring and alerting etcd Cluster Auto Scaling group Prometheus Cluster Auto Scaling group NSQ Cluster Kafka is used in microservices for ordered queuing
8. Gateways to payment networks Storing the bank’s data in Cassandra Distributed locks with etcd Asynchronous Processing with NSQ/Kafka Our microservice platform on Kubernetes Metrics and alerting with Prometheus
11. Encrypted Messages are transferred via AWS Direct Connect Pods running in Kubernetes on EC2 receive messages and process them Mastercard Monzo VPC AWS Cloud Auto Scaling group K8s Worker Instance K8s Worker Instance K8s Worker Instance
12. Response is sent back to our servers to relay back to Mastercard Messages are processed through a myriad of microservices before the response is sent back Mastercard Monzo VPC AWS Cloud Auto Scaling group K8s Worker Instance K8s Worker Instance K8s Worker Instance
13. Gateways to payment networks Storing the bank’s data in Cassandra Distributed locks with etcd Asynchronous Processing with NSQ/Kafka Our microservice platform on Kubernetes Metrics and alerting with Prometheus
14. VPC AWS Cloud K8s Worker Auto Scaling group K8s Worker Instance K8s Worker Instance K8s Worker Instance K8s Worker Instance K8s Worker Instance K8s Worker Instance K8s Worker Instance K8s Worker Instance K8s Worker Instance K8s Worker Instance
17. Kubernetes Cluster Engineers trigger a deployment using a local tool called Shipper Deployment Service validates code, does static analysis and builds a container image Rolling deployments are invoked and completed via Kubernetes
19. Service Communication Kubernetes Worker on EC2 service.transaction service.account Kubernetes Worker on EC2 service.account
20. service.account service.account Service Discovery service.transaction Kubernetes Worker on EC2 service.balance service.pot Kubernetes Worker on EC2 envoy config provider Kubernetes Master on EC2 K8s API Server Kubernetes Worker on EC2 service.account
21. Service Communication service.transaction Kubernetes Worker on EC2 service.account service.account Kubernetes Worker on EC2 service.account service.balance service.pot Service Discovery and Routing Retries / Timeouts / Circuit Breaking Observability
22. Gateways to payment networks Storing the bank’s data in Cassandra Distributed locks with etcd Asynchronous Processing with NSQ/Kafka Our microservice platform on Kubernetes Metrics and alerting with Prometheus
23. Storing Data service.transaction Kubernetes Worker txn_00000123456
24. Storing Data Cassandra Ring service.transaction Kubernetes Worker txn_00000123456 Replication factor: 3 Quorum: Local
25. Storing Data service.transaction Kubernetes Worker txn_00000123456 Replication factor: 3 Quorum: Local
26. Storing Data service.transaction Kubernetes Worker txn_00000123456 Replication factor: 3 Quorum: One
27. Storing Data service.transaction Kubernetes Worker txn_00000123456 Replication factor: 3 Quorum: Local
28. Gateways to payment networks Storing the bank’s data in Cassandra Distributed locks with etcd Asynchronous Processing with NSQ/Kafka Our microservice platform on Kubernetes Metrics and alerting with Prometheus
29. Many things can occur asynchronously rather than a direct blocking RPC. Message queues like NSQ and Kafka provide asynchronous flows with at least once message delivery semantics. Asynchronous Messaging service.transaction Kubernetes Worker service.balance service.pot Kubernetes Worker kafka NSQ Auto-scaling Group NSQ service.transaction Kubernetes Worker service.txn-enrichment
31. Gateways to payment networks Storing the bank’s data in Cassandra Distributed locks with etcd Asynchronous Processing with NSQ/Kafka Our microservice platform on Kubernetes Metrics and alerting with Prometheus
32. Distributed Locking service.transaction txn_00000123456
33. Distributed Locking
34. Gateways to payment networks Storing the bank’s data in Cassandra Distributed locks with etcd Asynchronous Processing with NSQ/Kafka Our microservice platform on Kubernetes Metrics and alerting with Prometheus
35. Prometheus and Thanos Prometheus is a flexible time-series data store and query engine. Thanos allows us to treat many Prometheus servers as one single one, with infinite retention. • RPC Request/Response cycles • CPU / Memory / Network use • Asynchronous processing • C* and Distributed Locking • Cloudwatch Data • Social Media
36. A Global View of Monzo service.account Kubernetes Worker on EC2 Prometheus Services service.account Kubernetes Worker on EC2 Prometheus Infra service.account Kubernetes Worker on EC2 Prometheus Cassandra Kubernetes Worker on EC2 Thanos Query Thanos Store
38. Use your card at a store or online One of our DCs gets a process message Messages get passed through AWS Direct Connect VPC AWS Cloud Auto Scaling group Auto Scaling group Cassandra Cluster Auto Scaling group Messages go through dozens of microservices to process the message etcd Cluster Auto Scaling group Prometheus Cluster Auto Scaling group NSQ Cluster Over a VPN tunnel terminating in our K8s cluster K8s Worker Instances Data will be written to Cassandra to record what’s happened Cassandra will replicate the data to multiple nodes Some services will use distributed locks for exclusive processing
39. VPC AWS Cloud Auto Scaling group Auto Scaling group Cassandra Cluster Auto Scaling group We approve the transaction etcd Cluster Auto Scaling group Prometheus Cluster Auto Scaling group NSQ Cluster K8s Worker Instances We publish an event about the transaction We return the approval message
40. VPC AWS Cloud Auto Scaling group Auto Scaling group Cassandra Cluster Auto Scaling group etcd Cluster Auto Scaling group Prometheus Cluster Auto Scaling group NSQ Cluster K8s Worker Instances The event is consumed A push notification is sent
43. VPC AWS Cloud – eu-west-1 Auto Scaling group K8s Worker Instances Auto Scaling group Cassandra Cluster Auto Scaling group etcd Cluster Auto Scaling group K8s Worker Instances Auto Scaling group Cassandra Cluster Auto Scaling group etcd Cluster VPC AWS Cloud – us-east-2 or us-west-1 (TBD) Auto Scaling group K8s Worker Instances Auto Scaling group Cassandra Cluster Auto Scaling group etcd Cluster Auto Scaling group K8s Worker Instances Auto Scaling group Cassandra Cluster Auto Scaling group etcd Cluster
44. Thank you! © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Thank you! © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Christopher Evans - @evnsio Suhail Patel - @suhailpatel