How the Cloud Helps Nubank Support Millions of Daily Customers [AWS re:Invent 2018]

Presented at AWS re:Invent 2018, with Diogo Beato

Transcript

1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. How the Cloud Helps Nubank Support Millions of Daily Customers Alexandre Cisneiros Software Engineer Diogo Beato Software Engineer
3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Credit Card No-fees, international credit card supported by a fully digital and branchless experience.
4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Rewards A rewards program completely different from all experiences in the Brazilian market. 100% digital, simple and intuitive.
5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Bank Account Our own version of a bank account, the simplest and most intelligent solution yet.
6. “Nubank is the largest digital bank in the world outside of Asia”
7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Cloud Native Banking Teams empowered to execute independently, cradle to grave autonomy Rapidly evolving systems in small increments velocity low upfront cost and time to market strategy Bank needs reputation security
8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Growing quickly in a sensitive domain Unique applications 18M+ Countries 198 Customers 5M+ Millions of customers Credit Card 0.0 1.3 2.5 3.8 5.0 Set-14 Jan-15 Mai-15 Set-15 Jan-16 Mai-16 Set-16 Jan-17 Mai-17 Set-17 Jan-18 May-18
9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Growing quickly in a sensitive domain Daily HTTP requests 500M+ Daily Kafka messages 200M+ Deploys per day 50+ Microservices 190+ Engineers 150+ Millions of customers Credit Card 0.0 1.3 2.5 3.8 5.0 Set-14 Jan-15 Mai-15 Set-15 Jan-16 Mai-16 Set-16 Jan-17 Mai-17 Set-17 Jan-18 May-18
10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Scalability Bottlenecks 1. Database writes throughput 2. Batch Message Lags impacting UX Millions of customers Credit Card 0.0 1.3 2.5 3.8 5.0 Set-14 Jan-15 Mai-15 Set-15 Jan-16 Mai-16 Set-16 Jan-17 Mai-17 Set-17 Jan-18 May-18
11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Scalability Plan • Split the workload • Each service own a different type of data from customer • Low level of interactions between customers • Safe split by customers
12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Option #1: Split Services Database • Database writes throughput are the worst bottleneck • Split service database • Change services to route queries and writes to the right shard db shard s0 db shard s1 db shard s2 backend service
13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Option #1: Known Issues • Huge effort to change all services • Solves only DB bottleneck issues • High risk of mixing business logic with infrastructure code • High risk of introducing new bugs db shard s0 db shard s1 db shard s2 backend service
14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Option #2: Scalability Units SERVICE 1 shard S0 SERVICE 3 SERVICE 2 SERVICE 1 shard S1 SERVICE 3 SERVICE 2 SERVICE 1 shard S2 SERVICE 3 SERVICE 2
15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Cloud Option #2: Scalability Units + Global Routing SERVICE 1 shard S0 SERVICE 3 SERVICE 2 SERVICE 1 shard S1 SERVICE 3 SERVICE 2 SERVICE 1 shard S2 SERVICE 3 SERVICE 2 SERVICE 1 global routing SERVICE 3 SERVICE 2 purchase
16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Cloud Easy to scale S0 S1 global routing S2 …
17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Cloud Reduced Blast Radius S0 S1 global routing S2 … X
18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Under the Hood We develop our own technology exploring functional programming and open source projects
19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Stack JVM Lisp Language Functional, Immutable structures Simple and easy to learn Quick feedback cycle using REPL Gradual typing (schemas) Clojure
20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Stack Immutable, no update-in-place ACID transactions preserve informations about what and when changed Query using data structures (Datalog) Cloud-native using DynamoDB as storage Datomic for data
21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Stack Messages are Immutable and persistent Decoupling layer between micro services Async workload Fault Tolerance and Resilience (circuit-breakers and dead-letters queues) External batch jobs are transformed into internal stream processors Kafka
22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Stack Container orchestrator Ready for microservices architecture Enforces immutability on infrastructure Faster iterations Self-healing capabilities Fast and dynamic auto-scaling Kubernetes
23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. All running on AWS
24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Automation Automation is the Contingency Plan that protects software systems from human failure
25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Infrastructure as code definition edn map containing the definition of all deployable units deploy Internal Clojure project wrapping cloud APIs
26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Infrastructure as code Everything must have a declarative definition
27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Infrastructure as code - Definition Environments have specific configurations for runtime {:name :staging :shards {:global [:global] :sharded [:s0 :s1] :defaults {:workload [:burst :small]}} {:name :prod :shards {:global [:global] :sharded [:s0 :s1 :s2 :s3 :s4 :s5 :s6] :defaults {:workload [:generic :large] :scaling {:min-size 2} :jvm {:flags [“-Xfuture"]}}}
28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Infrastructure as code - Definition Each service belongs to environments {:name :billing :squad :bills :envs {:prod :sharded :staging sharded} :jvm {:strategy :g1} :scaling {:min-size 8}} {:name :auth :squad :infosec :envs {:prod :global :staging :global} :workload [:nitro :2x-large] :scaling-polices [:cpu_high_alarm :cpu_low_alarm]}
29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Infrastructure as code - Deploy deploy Internal Clojure project wrapping cloud APIs Provisioning EC2 IAM roles Security groups Load balancers … definition edn map containing the definition of all deployable units CloudFormation Environment services deploy
30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Immutable Infrastructure Mutable state is the root of all evil
31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Immutability It’s taking a step forward with the possibility of taking a step back if things go wrong
32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Immutable Infrastructure (why?) • Zero-Day vulnerability • OS update • Upgrade kafka version • Change systemd unit
33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Sad Engineer SSH into all instances? • Inconsistent states • High urgency • Painful debugs
34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Cloud Immutable Infrastructure global routing S0 S1 S2 V 2.0 S0 global routing V 1.0 S1 S2
35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. • Short feedback loop • Easy to test • Quick to recover • Safety when making big changes Immutable Infrastructure
36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Moving to Kubernetes
37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Kubernetes Container orchestrator Ready for microservices architecture Enforces immutability on infrastructure Faster iterations Self-healing capabilities Fast and dynamic auto-scaling
38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. CloudFormation Services deploy Internal Clojure project wrapping cloud APIs Provisioning EC2 IAM roles Security groups Load balancers … definition edn map containing the definition of all deployable units CloudFormation Environment services deploy
39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Kubernetes on AWS deploy Internal Clojure project wrapping cloud APIs Provisioning definition edn map containing the definition of all deployable units Kubernetes Kube Master Kube Nodes … services deploy Kubernetes CF cluster
40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Cloud Migrating shards AWS CF Shard 0 AWS CF Shard 1 AWS CF Shard 2
41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Cloud Migrating shards AWS CF Shard 0 Kubernetes Shard 0 AWS CF Shard 1 AWS CF Shard 2
42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Cloud Migrating shards Kubernetes Shard 0 AWS CF Shard 1 AWS CF Shard 2
43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Cloud Migrating shards Kubernetes Shard 0 AWS CF Shard 1 Kubernetes Shard 1 AWS CF Shard 2 Kubernetes Shard 2
44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Cloud Migrating shards Kubernetes Shard 0 Kubernetes Shard 1 Kubernetes Shard 2
45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Learned the cloud is as elastic as our architecture scalability we are as fast as our automation velocity architecture planned for partial failures resilience infrastructure as code empowers teams autonomy
46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Interested in joining our revolution? sou.nu/jobs-at-nubank
47. Thank you! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
48. Please complete the session survey in the mobile app. ! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.