Tweet
Tweet

Slide 1

Slide 1 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. How the Cloud Helps Nubank Support Millions of Daily Customers Alexandre Cisneiros Software Engineer Diogo Beato Software Engineer

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Credit Card No-fees, international credit card supported by a fully digital and branchless experience.

Slide 4

Slide 4 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Rewards A rewards program completely different from all experiences in the Brazilian market. 100% digital, simple and intuitive.

Slide 5

Slide 5 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Bank Account Our own version of a bank account, the simplest and most intelligent solution yet.

Slide 6

Slide 6 text

“Nubank is the largest digital bank in the world outside of Asia”

Slide 7

Slide 7 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Cloud Native Banking Teams empowered to execute independently, cradle to grave autonomy Rapidly evolving systems in small increments velocity low upfront cost and time to market strategy Bank needs reputation security

Slide 8

Slide 8 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Growing quickly in a sensitive domain Unique applications 18M+ Countries 198 Customers 5M+ Millions of customers Credit Card 0.0 1.3 2.5 3.8 5.0 Set-14 Jan-15 Mai-15 Set-15 Jan-16 Mai-16 Set-16 Jan-17 Mai-17 Set-17 Jan-18 May-18

Slide 9

Slide 9 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Growing quickly in a sensitive domain Daily HTTP requests 500M+ Daily Kafka messages 200M+ Deploys per day 50+ Microservices 190+ Engineers 150+ Millions of customers Credit Card 0.0 1.3 2.5 3.8 5.0 Set-14 Jan-15 Mai-15 Set-15 Jan-16 Mai-16 Set-16 Jan-17 Mai-17 Set-17 Jan-18 May-18

Slide 10

Slide 10 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Scalability Bottlenecks 1. Database writes throughput 2. Batch Message Lags impacting UX Millions of customers Credit Card 0.0 1.3 2.5 3.8 5.0 Set-14 Jan-15 Mai-15 Set-15 Jan-16 Mai-16 Set-16 Jan-17 Mai-17 Set-17 Jan-18 May-18

Slide 11

Slide 11 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Scalability Plan • Split the workload • Each service own a different type of data from customer • Low level of interactions between customers • Safe split by customers

Slide 12

Slide 12 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Option #1: Split Services Database • Database writes throughput are the worst bottleneck • Split service database • Change services to route queries and writes to the right shard db shard s0 db shard s1 db shard s2 backend service

Slide 13

Slide 13 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Option #1: Known Issues • Huge effort to change all services • Solves only DB bottleneck issues • High risk of mixing business logic with infrastructure code • High risk of introducing new bugs db shard s0 db shard s1 db shard s2 backend service

Slide 14

Slide 14 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Option #2: Scalability Units SERVICE 1 shard S0 SERVICE 3 SERVICE 2 SERVICE 1 shard S1 SERVICE 3 SERVICE 2 SERVICE 1 shard S2 SERVICE 3 SERVICE 2

Slide 15

Slide 15 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Cloud Option #2: Scalability Units + Global Routing SERVICE 1 shard S0 SERVICE 3 SERVICE 2 SERVICE 1 shard S1 SERVICE 3 SERVICE 2 SERVICE 1 shard S2 SERVICE 3 SERVICE 2 SERVICE 1 global routing SERVICE 3 SERVICE 2 purchase

Slide 16

Slide 16 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Cloud Easy to scale S0 S1 global routing S2 …

Slide 17

Slide 17 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Cloud Reduced Blast Radius S0 S1 global routing S2 … X

Slide 18

Slide 18 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Under the Hood We develop our own technology exploring functional programming and open source projects

Slide 19

Slide 19 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Stack JVM Lisp Language Functional, Immutable structures Simple and easy to learn Quick feedback cycle using REPL Gradual typing (schemas) Clojure

Slide 20

Slide 20 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Stack Immutable, no update-in-place ACID transactions preserve informations about what and when changed Query using data structures (Datalog) Cloud-native using DynamoDB as storage Datomic for data

Slide 21

Slide 21 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Stack Messages are Immutable and persistent Decoupling layer between micro services Async workload Fault Tolerance and Resilience (circuit-breakers and dead-letters queues) External batch jobs are transformed into internal stream processors Kafka

Slide 22

Slide 22 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Stack Container orchestrator Ready for microservices architecture Enforces immutability on infrastructure Faster iterations Self-healing capabilities Fast and dynamic auto-scaling Kubernetes

Slide 23

Slide 23 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. All running on AWS

Slide 24

Slide 24 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Automation Automation is the Contingency Plan that protects software systems from human failure

Slide 25

Slide 25 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Infrastructure as code definition edn map containing the definition of all deployable units deploy Internal Clojure project wrapping cloud APIs

Slide 26

Slide 26 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Infrastructure as code Everything must have a declarative definition

Slide 27

Slide 27 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Infrastructure as code - Definition Environments have specific configurations for runtime {:name :staging :shards {:global [:global] :sharded [:s0 :s1] :defaults {:workload [:burst :small]}} {:name :prod :shards {:global [:global] :sharded [:s0 :s1 :s2 :s3 :s4 :s5 :s6] :defaults {:workload [:generic :large] :scaling {:min-size 2} :jvm {:flags [“-Xfuture"]}}}

Slide 28

Slide 28 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Infrastructure as code - Definition Each service belongs to environments {:name :billing :squad :bills :envs {:prod :sharded :staging sharded} :jvm {:strategy :g1} :scaling {:min-size 8}} {:name :auth :squad :infosec :envs {:prod :global :staging :global} :workload [:nitro :2x-large] :scaling-polices [:cpu_high_alarm :cpu_low_alarm]}

Slide 29

Slide 29 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Infrastructure as code - Deploy deploy Internal Clojure project wrapping cloud APIs Provisioning EC2 IAM roles Security groups Load balancers … definition edn map containing the definition of all deployable units CloudFormation Environment services deploy

Slide 30

Slide 30 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Immutable Infrastructure Mutable state is the root of all evil

Slide 31

Slide 31 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Immutability It’s taking a step forward with the possibility of taking a step back if things go wrong

Slide 32

Slide 32 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Immutable Infrastructure (why?) • Zero-Day vulnerability • OS update • Upgrade kafka version • Change systemd unit

Slide 33

Slide 33 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Sad Engineer SSH into all instances? • Inconsistent states • High urgency • Painful debugs

Slide 34

Slide 34 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Cloud Immutable Infrastructure global routing S0 S1 S2 V 2.0 S0 global routing V 1.0 S1 S2

Slide 35

Slide 35 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. • Short feedback loop • Easy to test • Quick to recover • Safety when making big changes Immutable Infrastructure

Slide 36

Slide 36 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Moving to Kubernetes

Slide 37

Slide 37 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Kubernetes Container orchestrator Ready for microservices architecture Enforces immutability on infrastructure Faster iterations Self-healing capabilities Fast and dynamic auto-scaling

Slide 38

Slide 38 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. CloudFormation Services deploy Internal Clojure project wrapping cloud APIs Provisioning EC2 IAM roles Security groups Load balancers … definition edn map containing the definition of all deployable units CloudFormation Environment services deploy

Slide 39

Slide 39 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Kubernetes on AWS deploy Internal Clojure project wrapping cloud APIs Provisioning definition edn map containing the definition of all deployable units Kubernetes Kube Master Kube Nodes … services deploy Kubernetes CF cluster

Slide 40

Slide 40 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Cloud Migrating shards AWS CF Shard 0 AWS CF Shard 1 AWS CF Shard 2

Slide 41

Slide 41 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Cloud Migrating shards AWS CF Shard 0 Kubernetes Shard 0 AWS CF Shard 1 AWS CF Shard 2

Slide 42

Slide 42 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Cloud Migrating shards Kubernetes Shard 0 AWS CF Shard 1 AWS CF Shard 2

Slide 43

Slide 43 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Cloud Migrating shards Kubernetes Shard 0 AWS CF Shard 1 Kubernetes Shard 1 AWS CF Shard 2 Kubernetes Shard 2

Slide 44

Slide 44 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Nubank Cloud Migrating shards Kubernetes Shard 0 Kubernetes Shard 1 Kubernetes Shard 2

Slide 45

Slide 45 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lessons Learned the cloud is as elastic as our architecture scalability we are as fast as our automation velocity architecture planned for partial failures resilience infrastructure as code empowers teams autonomy

Slide 46

Slide 46 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Interested in joining our revolution? sou.nu/jobs-at-nubank

Slide 47

Slide 47 text

Thank you! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Slide 48

Slide 48 text

Please complete the session survey in the mobile app. ! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.