ISTA 2019 - Migrating data-intensive microservices from Python to Go

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

Migrating data-intensive microservices from Python to Go Nikolay Stoitsev Engineering Manager @ Uber

Slide 3

Slide 3 text

Early years Dispatch API Storage

Slide 4

Slide 4 text

Early years Dispatch API Storage Python Node.js

Slide 5

Slide 5 text

Invoice Generation Service

Slide 6

Slide 6 text

Background Legal document

Slide 7

Slide 7 text

Background Legal document Vary by country

Slide 8

Slide 8 text

Background Legal document Vary by country Vary by business line

Slide 9

Slide 9 text

Background Legal document Vary by country Vary by business line Triggered after every trip or food delivery

Slide 10

Slide 10 text

Sample architecture Money System Cassandra Kafka Preprocess Render Kafka Consumer Object Store

Slide 11

Slide 11 text

More than 30 upstream systems Large Scale

Slide 12

Slide 12 text

More than 30 upstream systems More than 100 TBs of data stored Large Scale

Slide 13

Slide 13 text

More than 30 upstream systems More than 100 TBs of data stored Running on 400 containers in multiple DCs Large Scale

Slide 14

Slide 14 text

More than 30 upstream systems More than 100 TBs of data stored Running on 400 containers in multiple DCs Running for 5 years Large Scale

Slide 15

Slide 15 text

More than 30 upstream systems More than 100 TBs of data stored Running on 400 containers in multiple DCs Running for 5 years 99.999% availability for last 6 months Large Scale

Slide 16

Slide 16 text

More than 30 upstream systems More than 100 TBs of data stored Running on 400 containers in multiple DCs Running for 5 years 99.999% availability for last 6 months Implemented in Python Large Scale

Slide 17

Slide 17 text

Sample architecture Money System Cassandra Kafka Preprocess Render Kafka Consumer Object Store Web API Hive

Slide 18

Slide 18 text

Building blocks http://ﬂask.pocoo.org

Slide 19

Slide 19 text

Flask Example

Slide 20

Slide 20 text

Flask Usage

Slide 21

Slide 21 text

MVCS

Slide 22

Slide 22 text

MVCS Controller Mapper Service Entities External Services Database

Slide 23

Slide 23 text

Building blocks https://uwsgi-docs.readthedocs.io/

Slide 24

Slide 24 text

uWSGI uwsgi python python python

Slide 25

Slide 25 text

Building blocks http://www.celeryproject.org/

Slide 26

Slide 26 text

Celery celery-worker celery-worker celery-worker kafka consumer Redis

Slide 27

Slide 27 text

“Use the right tool for the job”

Slide 28

Slide 28 text

It hurts velocity at some point

Slide 29

Slide 29 text

What we need for each language? Training / Best practices / Documentation / Experts

Slide 30

Slide 30 text

What we need for each language? Training / Best practices / Documentation / Experts Project template / Bootstrapping

Slide 31

Slide 31 text

What we need for each language? Training / Best practices / Documentation / Experts Project template / Bootstrapping Configuration

Slide 32

Slide 32 text

What we need for each language? Training / Best practices / Documentation / Experts Project template / Bootstrapping Configuration Debuggers

Slide 33

Slide 33 text

What we need for each language? Training / Best practices / Documentation / Experts Project template / Bootstrapping Configuration Debuggers Profilers

Slide 34

Slide 34 text

What we need for each language? Training / Best practices / Documentation / Experts Project template / Bootstrapping Configuration Debuggers Profilers Building, Packaging, Deploying

Slide 35

Slide 35 text

We picked Go and Java

Slide 36

Slide 36 text

Why Go?

Slide 37

Slide 37 text

Broad applicability

Slide 38

Slide 38 text

High performance

Slide 39

Slide 39 text

Static typing

Slide 40

Slide 40 text

Has momentum

Slide 41

Slide 41 text

From Python to Go

Slide 42

Slide 42 text

EAFP versus LBYL

Slide 43

Slide 43 text

EAFP versus LBYL

Slide 44

Slide 44 text

Dependency injection https://github.com/uber-go/fx

Slide 45

Slide 45 text

Cadence instead of Celery https://github.com/uber/cadence

Slide 46

Slide 46 text

Cadence Cadence DB queue Timers invoice service worker worker worker worker

Slide 47

Slide 47 text

MVCS translates nicely

Slide 48

Slide 48 text

How to migrate? Money System Invoice Generation Storage Python

Slide 49

Slide 49 text

Option #1 - Big Bang Rewrite Money System Invoice Generation Storage Python Invoice Generation Go

Slide 50

Slide 50 text

Option #1 - Big Bang Rewrite Money System Storage Invoice Generation Go

Slide 51

Slide 51 text

No visibility on regressions

Slide 52

Slide 52 text

No visibility on performance degradation

Slide 53

Slide 53 text

No visibility on feature parity

Slide 54

Slide 54 text

Option #2 - Do it iteratively

Slide 55

Slide 55 text

Invoice Generation Storage Kafka

Slide 56

Slide 56 text

Storage Kafka Preprocess Render

Slide 57

Slide 57 text

Storage Kafka Preprocess Render Preprocess Go

Slide 58

Slide 58 text

Storage Kafka Preprocess Render Preprocess Go Compare

Slide 59

Slide 59 text

Storage Kafka Preprocess Render Preprocess Go Compare Toggle

Slide 60

Slide 60 text

Volume?

Slide 61

Slide 61 text

m3 DB https://www.m3db.io

Slide 62

Slide 62 text

Tally - stats collection in Go https://github.com/uber-go/tally

Slide 63

Slide 63 text

Tally - stats collection in Go

Slide 64

Slide 64 text

Measure processing time p95, p99

Slide 65

Slide 65 text

Storage Kafka Preprocess Render Preprocess Go Compare m3 Grafana

Slide 66

Slide 66 text

Correctness?

Slide 67

Slide 67 text

Storage Kafka Preprocess Render Preprocess Go Compare Kafka ELK

Slide 68

Slide 68 text

No content

Slide 69

Slide 69 text

Structured logging

Slide 70

Slide 70 text

Structured logging

Slide 71

Slide 71 text

Structured logging

Slide 72

Slide 72 text

Structured logging

Slide 73

Slide 73 text

Zap https://github.com/uber-go/zap

Slide 74

Slide 74 text

Zap

Slide 75

Slide 75 text

ELK

Slide 76

Slide 76 text

Benefits of iterative approach Verify regressions Verify performance problems Verify feature parity