Share
Embed
How the Cloud Helps Nubank Support Millions of Daily Customers [AWS re:Invent 2018]
Presented at AWS re:Invent 2018, with Diogo Beato
Alexandre Cisneiros
November 29, 2018
Transcript
48 slides
-
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. How the Cloud Helps Nubank Support Millions of Daily Customers Alexandre Cisneiros Software Engineer Diogo Beato Software Engineer -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Credit Card No-fees, international credit card supported by a fully digital and branchless experience. -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Rewards A rewards program completely different from all experiences in the Brazilian market. 100% digital, simple and intuitive. -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Bank Account Our own version of a bank account, the simplest and most intelligent solution yet. -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Cloud Native Banking Teams empowered to execute independently, cradle to grave autonomy Rapidly evolving systems in small increments velocity low upfront cost and time to market strategy Bank needs reputation security -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Growing quickly in a sensitive domain Unique applications 18M+ Countries 198 Customers 5M+ Millions of customers Credit Card 0.0 1.3 2.5 3.8 5.0 Set-14 Jan-15 Mai-15 Set-15 Jan-16 Mai-16 Set-16 Jan-17 Mai-17 Set-17 Jan-18 May-18 -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Growing quickly in a sensitive domain Daily HTTP requests 500M+ Daily Kafka messages 200M+ Deploys per day 50+ Microservices 190+ Engineers 150+ Millions of customers Credit Card 0.0 1.3 2.5 3.8 5.0 Set-14 Jan-15 Mai-15 Set-15 Jan-16 Mai-16 Set-16 Jan-17 Mai-17 Set-17 Jan-18 May-18 -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Scalability Bottlenecks 1. Database writes throughput 2. Batch Message Lags impacting UX Millions of customers Credit Card 0.0 1.3 2.5 3.8 5.0 Set-14 Jan-15 Mai-15 Set-15 Jan-16 Mai-16 Set-16 Jan-17 Mai-17 Set-17 Jan-18 May-18 -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Scalability Plan • Split the workload • Each service own a different type of data from customer • Low level of interactions between customers • Safe split by customers -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Option #1: Split Services Database • Database writes throughput are the worst bottleneck • Split service database • Change services to route queries and writes to the right shard db shard s0 db shard s1 db shard s2 backend service -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Option #1: Known Issues • Huge effort to change all services • Solves only DB bottleneck issues • High risk of mixing business logic with infrastructure code • High risk of introducing new bugs db shard s0 db shard s1 db shard s2 backend service -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Option #2: Scalability Units SERVICE 1 shard S0 SERVICE 3 SERVICE 2 SERVICE 1 shard S1 SERVICE 3 SERVICE 2 SERVICE 1 shard S2 SERVICE 3 SERVICE 2 -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Nubank Cloud Option #2: Scalability Units + Global Routing SERVICE 1 shard S0 SERVICE 3 SERVICE 2 SERVICE 1 shard S1 SERVICE 3 SERVICE 2 SERVICE 1 shard S2 SERVICE 3 SERVICE 2 SERVICE 1 global routing SERVICE 3 SERVICE 2 purchase -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Nubank Cloud Easy to scale S0 S1 global routing S2 … -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Nubank Cloud Reduced Blast Radius S0 S1 global routing S2 … X -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Nubank Under the Hood We develop our own technology exploring functional programming and open source projects -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Nubank Stack JVM Lisp Language Functional, Immutable structures Simple and easy to learn Quick feedback cycle using REPL Gradual typing (schemas) Clojure -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Nubank Stack Immutable, no update-in-place ACID transactions preserve informations about what and when changed Query using data structures (Datalog) Cloud-native using DynamoDB as storage Datomic for data -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Nubank Stack Messages are Immutable and persistent Decoupling layer between micro services Async workload Fault Tolerance and Resilience (circuit-breakers and dead-letters queues) External batch jobs are transformed into internal stream processors Kafka -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Nubank Stack Container orchestrator Ready for microservices architecture Enforces immutability on infrastructure Faster iterations Self-healing capabilities Fast and dynamic auto-scaling Kubernetes -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. All running on AWS -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Automation Automation is the Contingency Plan that protects software systems from human failure -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Infrastructure as code definition edn map containing the definition of all deployable units deploy Internal Clojure project wrapping cloud APIs -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Infrastructure as code Everything must have a declarative definition -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Infrastructure as code - Definition Environments have specific configurations for runtime {:name :staging :shards {:global [:global] :sharded [:s0 :s1] :defaults {:workload [:burst :small]}} {:name :prod :shards {:global [:global] :sharded [:s0 :s1 :s2 :s3 :s4 :s5 :s6] :defaults {:workload [:generic :large] :scaling {:min-size 2} :jvm {:flags [“-Xfuture"]}}} -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Infrastructure as code - Definition Each service belongs to environments {:name :billing :squad :bills :envs {:prod :sharded :staging sharded} :jvm {:strategy :g1} :scaling {:min-size 8}} {:name :auth :squad :infosec :envs {:prod :global :staging :global} :workload [:nitro :2x-large] :scaling-polices [:cpu_high_alarm :cpu_low_alarm]} -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Infrastructure as code - Deploy deploy Internal Clojure project wrapping cloud APIs Provisioning EC2 IAM roles Security groups Load balancers … definition edn map containing the definition of all deployable units CloudFormation Environment services deploy -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Immutable Infrastructure Mutable state is the root of all evil -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Immutability It’s taking a step forward with the possibility of taking a step back if things go wrong -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Immutable Infrastructure (why?) • Zero-Day vulnerability • OS update • Upgrade kafka version • Change systemd unit -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Sad Engineer SSH into all instances? • Inconsistent states • High urgency • Painful debugs -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Nubank Cloud Immutable Infrastructure global routing S0 S1 S2 V 2.0 S0 global routing V 1.0 S1 S2 -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. • Short feedback loop • Easy to test • Quick to recover • Safety when making big changes Immutable Infrastructure -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Moving to Kubernetes -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Kubernetes Container orchestrator Ready for microservices architecture Enforces immutability on infrastructure Faster iterations Self-healing capabilities Fast and dynamic auto-scaling -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. CloudFormation Services deploy Internal Clojure project wrapping cloud APIs Provisioning EC2 IAM roles Security groups Load balancers … definition edn map containing the definition of all deployable units CloudFormation Environment services deploy -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Kubernetes on AWS deploy Internal Clojure project wrapping cloud APIs Provisioning definition edn map containing the definition of all deployable units Kubernetes Kube Master Kube Nodes … services deploy Kubernetes CF cluster -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Nubank Cloud Migrating shards AWS CF Shard 0 AWS CF Shard 1 AWS CF Shard 2 -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Nubank Cloud Migrating shards AWS CF Shard 0 Kubernetes Shard 0 AWS CF Shard 1 AWS CF Shard 2 -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Nubank Cloud Migrating shards Kubernetes Shard 0 AWS CF Shard 1 AWS CF Shard 2 -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Nubank Cloud Migrating shards Kubernetes Shard 0 AWS CF Shard 1 Kubernetes Shard 1 AWS CF Shard 2 Kubernetes Shard 2 -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Nubank Cloud Migrating shards Kubernetes Shard 0 Kubernetes Shard 1 Kubernetes Shard 2 -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Lessons Learned the cloud is as elastic as our architecture scalability we are as fast as our automation velocity architecture planned for partial failures resilience infrastructure as code empowers teams autonomy -
© 2018, Amazon Web Services, Inc. or its affiliates. All
rights reserved. Interested in joining our revolution? sou.nu/jobs-at-nubank -
Thank you! © 2018, Amazon Web Services, Inc. or its
affiliates. All rights reserved. -
Please complete the session survey in the mobile app. !
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.