Slide 1

Slide 1 text

Mistakes were made… Modelling prototypes to critical systems with Cassandra Matt Heath, Monzo

Slide 2

Slide 2 text

Hi, I’m Matt

Slide 3

Slide 3 text

@mattheath

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

Nov 2015 Mar 2019

Slide 8

Slide 8 text

Nov 2015 Mar 2019

Slide 9

Slide 9 text

Nov 2015 Mar 2019

Slide 10

Slide 10 text

Nov 2015 Mar 2019

Slide 11

Slide 11 text

Nov 2015 Mar 2019 Fastest growing UK Bank 1.6M Customers

Slide 12

Slide 12 text

☁ ?

Slide 13

Slide 13 text

“construct a highly agile and highly available service from ephemeral and assumed broken components” - Adrian Cockcroſt

Slide 14

Slide 14 text

How does Monzo operate? - AWS, GCP, and physical data centres - Cloud Native technologies:
 Kubernetes, Docker, Calico, Cassandra, Kafka, NSQ, Etcd, Prometheus, Jaeger, Envoy, Elasticsearch… - Go based micro services
 ⛅

Slide 15

Slide 15 text

Feb 2015 Mar 2019 1000 services

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

Why Cassandra? - High Availability - Tuneable Consistency - Linear Scalability - Geographic Replication


Slide 23

Slide 23 text

eu-west-1a eu-west-1b eu-west-1c

Slide 24

Slide 24 text

CREATE KEYSPACE account WITH replication = { 'class': 'NetworkTopologyStrategy', 'eu-west-1': '3' }

Slide 25

Slide 25 text

eu-west-1a eu-west-1b eu-west-1c

Slide 26

Slide 26 text

eu-west-1a eu-west-1b eu-west-1c

Slide 27

Slide 27 text

eu-west-1a eu-west-1b eu-west-1c

Slide 28

Slide 28 text

Data Modelling

Slide 29

Slide 29 text

@jrecursive

Slide 30

Slide 30 text

Column B Column C Key Column A

Slide 31

Slide 31 text

Column B Column C ID Column A

Slide 32

Slide 32 text

Key Value Value Value Column A Column B Column C Timestamp Timestamp Timestamp

Slide 33

Slide 33 text

CREATE TABLE IF NOT EXISTS account ( id text, userid text, created timestamp, currency text, country text, description text, type text, PRIMARY KEY ((id)) );

Slide 34

Slide 34 text

Key Value Value Value Column A Column B Column C Timestamp Timestamp Timestamp

Slide 35

Slide 35 text

acc_as8d… 2018-xx-xx Matt’s account user_007xUi8… “created" “description” “userid” 1526917782000 1526917782000 1526917782000

Slide 36

Slide 36 text

CREATE TABLE IF NOT EXISTS accounts_by_userid ( id text, userid text, created timestamp, currency text, country text, description text, type text, PRIMARY KEY ((userid), id) );

Slide 37

Slide 37 text

Partition Key Row 1 Row 2 Row 3

Slide 38

Slide 38 text

Partition
 Key Row 1 Row 2 Row 3

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

Partition Key

Slide 41

Slide 41 text

Partition
 Key Row 1 Row 2 Row 3

Slide 42

Slide 42 text

Partition
 Key Row 1 Row infinity

Slide 43

Slide 43 text

User ID Account 1 Account 2 Account 3 No-one has infinite accounts!

Slide 44

Slide 44 text

CREATE TABLE IF NOT EXISTS transaction ( id text, accountid text, created timestamp, currency text, amount bigint, description text, PRIMARY KEY ((id)) );

Slide 45

Slide 45 text

Column B Column C Transaction
 ID Column A

Slide 46

Slide 46 text

Column B Column C Transaction
 ID Column A Must know primary key — can’t iterate

Slide 47

Slide 47 text

CREATE TABLE IF NOT EXISTS transaction_by_account ( id text, accountid text, created timestamp, currency text, amount bigint, description text, PRIMARY KEY ((accountid), id) );

Slide 48

Slide 48 text

Account
 ID Transaction 1 Transaction
 infinity

Slide 49

Slide 49 text

Timeseries: Partition by Time ⏱

Slide 50

Slide 50 text

Partition 2 3 4 5 7

Slide 51

Slide 51 text

Bucket 2 3 4 5 7 Composite Partition Key (Time range and Account ID) PRIMARY KEY ((accountid, timebucket), created, id)

Slide 52

Slide 52 text

Day 1 2 3 4 5 7 Composite Partition Key (Time range and Account ID) PRIMARY KEY ((accountid, timebucket), created, id)

Slide 53

Slide 53 text

Day 1 2 3 4 5 7 9 12 15 16 17 18 19 20 Day 2 21 22 24 23

Slide 54

Slide 54 text

Day 1 2 3 4 5 7 9 12 15 16 17 18 19 20 Day 2 21 22 24 23 “Return last 100 transactions”

Slide 55

Slide 55 text

9 12 15 16 17 18 19 20 Day 3 21 22 24 23 Day 2 Day 1 2 3 4 5 7 No data in this time period

Slide 56

Slide 56 text

9 12 15 16 17 18 19 20 Day 1000 21 22 24 23 Day 999 Day 1 2 3 4 5 7 Day 998 Day 3 Day 2 … No data in this time period

Slide 57

Slide 57 text

9 12 15 16 17 18 19 20 Day 1000 21 22 24 23 Day 999 Day 1 Day 998 Day 3 Day 2 … Did the data ever exist?! When do we stop…?

Slide 58

Slide 58 text

Is your data predictable? How do you choose your bucket size?

Slide 59

Slide 59 text

Day 1 2 3 4 5 7 9 12 15 16 17 18 19 20 Day 2 21 22 24 23

Slide 60

Slide 60 text

Day 1 2 3 4 5 7 9 12 15 16 17 18 19 20 Day 2 21 22 24 23 Hot Partition Key

Slide 61

Slide 61 text

eu-west-1a eu-west-1b eu-west-1c Partition Key = Day 1

Slide 62

Slide 62 text

eu-west-1a eu-west-1b eu-west-1c Partition Key = Day 2

Slide 63

Slide 63 text

Is your data predictable? How do you choose your bucket size?
 Ensure correct partitioning!

Slide 64

Slide 64 text

Flakeseries: Partition by Time ⏱ Retrieve by ID

Slide 65

Slide 65 text

Bucket 2 3 4 5 7 Composite Partition Key (Time range and Account ID) PRIMARY KEY ((accountid, timebucket), created, id)

Slide 66

Slide 66 text

Bucket 2 3 4 5 7 Time range: We need to know the timestamp to read PRIMARY KEY ((accountid, timebucket), created, id)

Slide 67

Slide 67 text

Bucket 2 3 4 5 7 Time range: We need to know the timestamp to read PRIMARY KEY ((accountid, timebucket), created, id) accountid = acc_00009Wd3Yeh2O329bFTVHF

Slide 68

Slide 68 text

accountid = acc_00009Wd3Yeh2O329bFTVHF

Slide 69

Slide 69 text

accountid = acc_00009Wd3Yeh2O329bFTVHF Flake IDs = Time based lexically sortable IDs

Slide 70

Slide 70 text

accountid = acc_00009Wd3Yeh2O329bFTVHF Flake IDs = Time based lexically sortable IDs Base62 encoded 128bit Int
 eg 26341991268378369512474991263748

Slide 71

Slide 71 text

accountid = acc_00009Wd3Yeh2O329bFTVHF Flake IDs = Time based lexically sortable IDs Base62 encoded 128bit Int
 eg 26341991268378369512474991263748 64 bits - Time in ms since epoch
 48 bits - Worker ID 16 bits - Sequence ID

Slide 72

Slide 72 text

accountid = acc_00009Wd3Yeh2O329bFTVHF

Slide 73

Slide 73 text

accountid = acc_00009Wd3Yeh2O329bFTVHF transactionid = tx_00009gEBzyFoAtFYllr9Qf

Slide 74

Slide 74 text

accountid = acc_00009Wd3Yeh2O329bFTVHF transactionid = tx_00009gEBzyFoAtFYllr9Qf PRIMARY KEY ((bucket), flake_created, transactionid)

Slide 75

Slide 75 text

accountid = acc_00009Wd3Yeh2O329bFTVHF transactionid = tx_00009gEBzyFoAtFYllr9Qf PRIMARY KEY ((bucket), flake_created, transactionid) PRIMARY KEY ((accountid, bucket), flake_created, transactionid)

Slide 76

Slide 76 text

TTLing your data ⏱

Slide 77

Slide 77 text

TTLing your data

Slide 78

Slide 78 text

Key 2 3 4 5 7 9 12 15 16 17 18 19 20 Key 21 22 24 23

Slide 79

Slide 79 text

Key 2 3 4 5 7 9 12 15 16 17 18 19 20 Key 21 22 24 23 Mixed TTLs across same data set


Slide 80

Slide 80 text

Key 2 3 4 5 7 9 12 15 16 17 18 19 20 Key 21 22 24 23 Mixed TTLs across same data set
 Compacted into same sstables

Slide 81

Slide 81 text

Key 2 4 5 7 9 12 15 17 18 19 20 Key 21 24 23 First set of rows expire

Slide 82

Slide 82 text

Key 2 4 7 9 12 15 17 Key 21 24 23 Second set of rows expire

Slide 83

Slide 83 text

No data is removed until compaction occurs Key 2 3 4 5 7 9 12 15 16 17 18 19 20 Key 21 22 24 23

Slide 84

Slide 84 text

Time Window Compaction Strategy Create sstables per time range Bucket 1 Bucket 2 9 12 15 16 17 18 19 20 Bucket 3 21 22 24 23 2 3 4 5 7 9 12 15 16 17 18 19 20 21 22 24 23

Slide 85

Slide 85 text

Time Window Compaction Strategy sstables dropped once all data has expired Bucket 1 Bucket 2 9 12 15 16 17 18 19 20 Bucket 3 21 22 24 23 2 3 4 5 7 9 12 15 16 17 18 19 20 21 22 24 23

Slide 86

Slide 86 text

Time Window Compaction Strategy sstables dropped once all data has expired Bucket 1 Bucket 2 9 12 15 16 17 18 19 20 Bucket 3 21 22 24 23 2 3 4 5 7 9 12 15 16 17 18 19 20 21 22 24 23

Slide 87

Slide 87 text

Time Window Compaction Strategy sstables dropped once all data has expired Bucket 1 Bucket 2 9 12 15 16 17 18 19 20 Bucket 3 21 22 24 23 2 3 4 5 7 9 12 15 16 17 18 19 20 21 22 24 23

Slide 88

Slide 88 text

Data modelling take aways - Correct data modelling is incredibly important! - Wide rows are ok to a point - Repairs on wide rows are problematic - Make Timeseries buckets predictable - Watch for Hot Keys! - TTLs don’t always mean your data is deleted

Slide 89

Slide 89 text

Nov 2015 Mar 2019 What works here Might not work here

Slide 90

Slide 90 text

No content

Slide 91

Slide 91 text

monzo.com/careers