Tweet
Tweet

Slide 1

Slide 1 text

Evolving big microservice architectures Nikolay Stoitsev, Engineering Manager @ Uber Engineering

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Dispatch API

Slide 5

Slide 5 text

A monolith? Code was consistently good Easy to reuse code Well tested Hard to deploy

Slide 6

Slide 6 text

Pull Request Pull Request Pull Request Pull Request Pull Request ? ? ? ? ?

Slide 7

Slide 7 text

Pull Request Pull Request Pull Request Pull Request Pull Request ? ? ? ? ? Unknown risk Unknown priority

Slide 8

Slide 8 text

Low Risk High Risk High Risk Med Risk High Risk Build

Slide 9

Slide 9 text

Low Risk High Risk High Risk Med Risk High Risk Build ?

Slide 10

Slide 10 text

Low Risk High Risk High Risk Med Risk High Risk Build

Slide 11

Slide 11 text

Low Risk / P0 High Risk / P1 High Risk / P0 Med Risk / P3 High Risk / P0 Build

Slide 12

Slide 12 text

Low Risk / P0 High Risk / P1 High Risk / P0 Med Risk / P3 High Risk / P0 Build CI / CD

Slide 13

Slide 13 text

Gradual rollout is important

Slide 14

Slide 14 text

Lesson learned: A monolith can scale a lot, but...

Slide 15

Slide 15 text

Dispatch API

Slide 16

Slide 16 text

Dispatch API Users Trips Receipts Payments

Slide 17

Slide 17 text

Dispatch API Users Trips Receipts Payments

Slide 18

Slide 18 text

Dispatch API Users Trips Receipts Payments

Slide 19

Slide 19 text

Dispatch API Users Trips Receipts Payments

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

4000+ microservices 1K+ avg deploys a day

Slide 22

Slide 22 text

Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments Services Receipts Apache Kafka Apache Kafka Apache Kafka Sample architecture

Slide 23

Slide 23 text

We want to introduce Uber Elevate

Slide 24

Slide 24 text

Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments Services Receipts Apache Kafka Apache Kafka Apache Kafka

Slide 25

Slide 25 text

Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments Services Receipts Task #1 Apache Kafka Apache Kafka Apache Kafka

Slide 26

Slide 26 text

Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments Services Receipts Task #1 OK Apache Kafka Apache Kafka Apache Kafka

Slide 27

Slide 27 text

Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments Services Receipts Task #1 OK Task #2 Apache Kafka Apache Kafka Apache Kafka

Slide 28

Slide 28 text

Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments Services Receipts Task #1 OK Task #2 Task #3 Apache Kafka Apache Kafka Apache Kafka

Slide 29

Slide 29 text

Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments Services Receipts Task #1 OK Task #2 Task #3 Task #4 Apache Kafka Apache Kafka Apache Kafka

Slide 30

Slide 30 text

Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments Services Receipts Task #1 OK Task #2 Task #3 Task #4 Task #5 Apache Kafka Apache Kafka Apache Kafka

Slide 31

Slide 31 text

Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments Services Receipts Task #1 OK Task #2 Task #3 Task #4 Task #5 OK Apache Kafka Apache Kafka Apache Kafka

Slide 32

Slide 32 text

Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments Services Receipts Task #1 OK Task #2 Task #3 Task #4 Task #5 OK Task #6 Apache Kafka Apache Kafka Apache Kafka

Slide 33

Slide 33 text

How do we evolve big microservice architectures?

Slide 34

Slide 34 text

You can’t change what you can’t test

Slide 35

Slide 35 text

Staging environment?

Slide 36

Slide 36 text

Integration tests?

Slide 37

Slide 37 text

Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments Services Receipts Change Apache Kafka Apache Kafka Apache Kafka

Slide 38

Slide 38 text

Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments Services Receipts Change MOCK MOCK MOCK MOCK Apache Kafka Apache Kafka Apache Kafka

Slide 39

Slide 39 text

Multitenancy

Slide 40

Slide 40 text

Tenancy - set of data that is isolated from the other sets of data

Slide 41

Slide 41 text

Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments Services Receipts Change Apache Kafka Apache Kafka Apache Kafka

Slide 42

Slide 42 text

Testing tenancy vs. Production tenancy

Slide 43

Slide 43 text

Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments Services Receipts Change Test Data Test Config Test Config Test Data Apache Kafka Apache Kafka Apache Kafka

Slide 44

Slide 44 text

Audit tenancy

Slide 45

Slide 45 text

Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments Services Receipts Audit tenancy traffic Check Test Config Test Config Test Data Apache Kafka Apache Kafka Apache Kafka Check Check Check

Slide 46

Slide 46 text

Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments Services Receipts Task #1 OK Task #2 Task #3 Task #4 Task #5 OK Apache Kafka Apache Kafka Apache Kafka

Slide 47

Slide 47 text

Platforms

Slide 48

Slide 48 text

Services are too concrete so it’s hard to support new use cases

Slide 49

Slide 49 text

Services need to support abstraction

Slide 50

Slide 50 text

AWS is a good example

Slide 51

Slide 51 text

Dispatch Pricing Services Tax Calc Vehicles Trips Services Users Payments Services Receipts Apache Kafka Apache Kafka Apache Kafka

Slide 52

Slide 52 text

Dispatch Pricing Services Tax Calc Vehicles Orders Platform Users Payments Services Receipts Apache Kafka Apache Kafka Apache Kafka

Slide 53

Slide 53 text

Dispatch Pricing Services Tax Calc Vehicles Orders Platform Users Payments Services Receipts Apache Kafka Apache Kafka Apache Kafka

Slide 54

Slide 54 text

Dispatch Pricing Services Tax Calc Vehicles Orders Platform Users Payments Services Receipts Apache Kafka Apache Kafka Apache Kafka

Slide 55

Slide 55 text

Hard shift in mentality in the way you design your services

Slide 56

Slide 56 text

Can you sell it outside of the company?

Slide 57

Slide 57 text

Tax Calc Payments Pricing

Slide 58

Slide 58 text

Tax Calc Payments Pricing 3.14159

Slide 59

Slide 59 text

Tax Calc Payments Pricing 3.14159 “3.14”

Slide 60

Slide 60 text

Tax Calc Payments Pricing 3.14159 “3.14” 314159

Slide 61

Slide 61 text

Common data model

Slide 62

Slide 62 text

No content

Slide 63

Slide 63 text

No content

Slide 64

Slide 64 text

Data extensions = schema + payload

Slide 65

Slide 65 text

How to do it? Schema service Repo Library ...

Slide 66

Slide 66 text

Dispatch Pricing Services Tax Calc Vehicles Orders Platform Users Payments Services Receipts Apache Kafka Apache Kafka Apache Kafka

Slide 67

Slide 67 text

… but there is still business specific logic.

Slide 68

Slide 68 text

Dispatch Pricing Services Tax Calc Vehicles Orders Platform Users Payments Services Receipts Where to put it? Business logic Apache Kafka Apache Kafka Apache Kafka

Slide 69

Slide 69 text

No common understanding of what should go where

Slide 70

Slide 70 text

How do we call code with business logic, presentation logic, and SQL queries all over the place?

Slide 71

Slide 71 text

We fix spaghetti code with separation of concerns

Slide 72

Slide 72 text

Presentation Logic Data

Slide 73

Slide 73 text

Layered architecture

Slide 74

Slide 74 text

Users Vehicles Order Platform Pricing Tax Calc Payments Receipts Rides Elevate Web Services Mobile Backend

Slide 75

Slide 75 text

Why?

Slide 76

Slide 76 text

Common language

Slide 77

Slide 77 text

Clear separation of concerns

Slide 78

Slide 78 text

Consistent patterns

Slide 79

Slide 79 text

API and functionality discoverability

Slide 80

Slide 80 text

Engineering velocity

Slide 81

Slide 81 text

Users Vehicles Order Platform Pricing Tax Calc Payments Receipts Rides Elevate Web Services Mobile Backend Generic Specific

Slide 82

Slide 82 text

Users Vehicles Order Platform Pricing Tax Calc Payments Receipts Rides Elevate Web Services Mobile Backend Breaks everything Breaks specific thing

Slide 83

Slide 83 text

Users Vehicles Order Platform Pricing Tax Calc Payments Receipts Rides Elevate Web Services Mobile Backend Presentation LoB Biz Infra Core

Slide 84

Slide 84 text

https://philcalcado.com/2018/09/24/services_layers.html Users Vehicles Order Platform Pricing Tax Calc Payments Receipts Rides Elevate Web Services Mobile Backend Clay Rocks

Slide 85

Slide 85 text

Users Vehicles Order Platform Pricing Tax Calc Payments Receipts Rides Elevate Web Services Mobile Backend Presentation

Slide 86

Slide 86 text

Users Vehicles Order Platform Pricing Tax Calc Payments Receipts Rides Elevate Web Services Mobile Backend LoB

Slide 87

Slide 87 text

Users Vehicles Order Platform Pricing Tax Calc Payments Receipts Rides Elevate Web Services Mobile Backend Biz Infra

Slide 88

Slide 88 text

Users Vehicles Order Platform Pricing Tax Calc Payments Receipts Rides Elevate Web Services Mobile Backend Core

Slide 89

Slide 89 text

Clear API

Slide 90

Slide 90 text

protobuf Apache Thrift GraphQL

Slide 91

Slide 91 text

End result: Layered multitenant architecture built with abstract platforms and a common data model

Slide 92

Slide 92 text

Users Vehicles Order Platform Pricing Tax Calc Payments Receipts Rides Web Services Mobile Backend

Slide 93

Slide 93 text

Users Vehicles Order Platform Pricing Tax Calc Payments Receipts Rides Elevate Web Services Mobile Backend

Slide 94

Slide 94 text

Q&A Feedback:

Slide 95

Slide 95 text

Proprietary © 2019 Uber Technologies, Inc. All rights reserved. No part of this document may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval systems, without permission in writing from Uber. This document is intended only for the use of the individual or entity to whom it is addressed and contains information that is privileged, confidential or otherwise exempt from disclosure under applicable law. All recipients of this document are notified that the information contained herein includes proprietary and confidential information of Uber, and recipient may not make use of, disseminate, or in any way disclose this document or any of the enclosed information to any person other than employees of addressee to the extent necessary for consultations with authorized personnel of Uber.