PAT HELLAND AND ME
HOW TO BUILD STATEFUL DISTRIBUTED APPLICATIONS THAT CAN SCALE ALMOST INFINITELY
Slide 2
Slide 2 text
SEAN T. ALLEN
VP OF ENGINEERING AT WALLAROO LABS
MEMBER OF THE PONY CORE TEAM
AUTHOR OF “STORM APPLIED”
LOVER OF FRENCH STREET ART
@SEANTALLEN
@WALLAROOLABS
@PONYLANG
Slide 3
Slide 3 text
PAT HELLAND
AND ME
Slide 4
Slide 4 text
PAT HELLAND
WRITER OF PAPERS I LOVE
Slide 5
Slide 5 text
PAT HELLAND
LIFE BEYOND DISTRIBUTED
TRANSACTIONS
Slide 6
Slide 6 text
WHAT’S IN THIS TALK…
Slide 7
Slide 7 text
WHAT IS SCALING?
Slide 8
Slide 8 text
MORE AND MORE
THINGS
BUT, THEY DON’T GET BIGGER.
THERE’S JUST…
MORE OF THEM. LOTS MORE.
Slide 9
Slide 9 text
SOME AXIOMS…
Slide 10
Slide 10 text
TO SCALE INFINITELY,
WE HAVE TO SCALE HORIZONTALLY
Slide 11
Slide 11 text
TO SCALE INFINITELY,
WE MUST AVOID COORDINATION
Slide 12
Slide 12 text
DISTRIBUTED TRANSACTIONS ARE
A FORM OF COORDINATION
Slide 13
Slide 13 text
THEREFORE…
TO SCALE INFINITELY,
WE CAN’T USE TRANSACTIONS
Slide 14
Slide 14 text
WE SCALE ENTITIES
ENTITIES:
LIVE ON A SINGLE MACHINE
AND ARE MANIPULATED INDIVIDUALLY
Slide 15
Slide 15 text
WHAT IS AN ENTITY?
Slide 16
Slide 16 text
ENTITIES ARE BOUNDARIES OF ATOMICITY
Slide 17
Slide 17 text
Bob
6 3
5
8
Alice
4 2
7
1
Slide 18
Slide 18 text
Bob
6 3
5
8
Alice
4 2
7
1
Slide 19
Slide 19 text
Bob
6 3
5
8
Alice
4
7
1
2
Slide 20
Slide 20 text
Bob
6 3
5
8
Alice
4 2
7
1
Slide 21
Slide 21 text
Bob
6 3
5
8
Alice
4 2
7
1
Slide 22
Slide 22 text
DENORMALIZE..
ALL THE THINGS!
Slide 23
Slide 23 text
TWO-LAYER ARCHITECTURE
Slide 24
Slide 24 text
scale-agnostic
scale-aware
API
Slide 25
Slide 25 text
scale-agnostic
scale-aware
API
Slide 26
Slide 26 text
scale-agnostic
scale-aware
API
Slide 27
Slide 27 text
scale-agnostic
scale-aware
API
Slide 28
Slide 28 text
scale-agnostic
scale-aware
API
Slide 29
Slide 29 text
scale-agnostic
scale-aware
API
Slide 30
Slide 30 text
TO SCALE INFINITELY,
YOUR BUSINESS LOGIC HAS TO BE
INDEPENDENT OF SCALE
Slide 31
Slide 31 text
TWO BIG IDEAS
A WORLD OF POSSIBILITIES
Slide 32
Slide 32 text
WALLAROO
SCALE INDEPENDENT COMPUTING
FOR PYTHON
Slide 33
Slide 33 text
ENTITIES
BUT WE CALL THEM…
“STATE OBJECTS”
Slide 34
Slide 34 text
TWO-LAYER
ARCHITECTURE
BUT WE CALL IT…
“SCALE INDEPENDENCE”
Slide 35
Slide 35 text
user supplied logic
Wallaroo runtime
Wallaroo API
Slide 36
Slide 36 text
user supplied logic
Wallaroo runtime
Wallaroo API
Slide 37
Slide 37 text
user supplied logic
Wallaroo runtime
Wallaroo API
Slide 38
Slide 38 text
user supplied logic
Wallaroo runtime
Wallaroo API
Slide 39
Slide 39 text
WALLAROO API
MARKET SPREAD EXAMPLE
Slide 40
Slide 40 text
MARKET SPREAD
REAL-TIME “SOMETHING AIN’T RIGHT” TRADE CHECKS
Market Spread
State
Market
Data
Orders
Update
APPL
Check
MSFT
Rejections
Slide 41
Slide 41 text
MARKET SPREAD
TWO SOURCES OF DATA
Market Spread
State
Market
Data
Orders
Update
APPL
Check
MSFT
Rejections
Slide 42
Slide 42 text
MARKET SPREAD
ONE SINK
Market Spread
State
Market
Data
Orders
Update
APPL
Check
MSFT
Rejections
Slide 43
Slide 43 text
MARKET SPREAD
ORDER PIPELINE
Market Spread
State
Market
Data
Orders
Update
APPL
Check
MSFT
Rejections
Slide 44
Slide 44 text
MARKET SPREAD
MARKET DATA PIPELINE
Market Spread
State
Market
Data
Orders
Update
APPL
Check
MSFT
Rejections
Slide 45
Slide 45 text
MARKET SPREAD
APPLICATION DEFINITION
Slide 46
Slide 46 text
APPLICATION DEFINITION
FLOW OF DATA FROM SOURCE TO SINK
Slide 47
Slide 47 text
TWO DATA PIPELINES
ORDERS
Slide 48
Slide 48 text
TWO DATA PIPELINES
MARKET DATA
Slide 49
Slide 49 text
DEFINE OUR SOURCES
1 PER PIPELINE
Slide 50
Slide 50 text
DEFINE OUR OPERATIONS
1 PER PIPELINE
Slide 51
Slide 51 text
DEFINE OUR OPERATIONS
CHECK ORDER AGAINST SYMBOL DATA
Slide 52
Slide 52 text
DEFINE OUR OPERATIONS
UPDATE SYMBOL DATA WITH LATEST MARKET DATA
Slide 53
Slide 53 text
DEFINE OUR SINKS
1 PER PIPELINE
Slide 54
Slide 54 text
DEFINE OUR SINKS
ORDERS PIPELINE MIGHT HAVE OUTPUT
Slide 55
Slide 55 text
DEFINE OUR SINKS
MARKET DATA ONLY UPDATES SYMBOL DATA- NO OUTPUT
Slide 56
Slide 56 text
SCALE INDEPENDENT
ONLY FLOW OF DATA AND OPERATIONS
Slide 57
Slide 57 text
USER SUPPLIED
LOGIC
Slide 58
Slide 58 text
UPDATE MARKET DATA STATE COMPUTATION
UPDATES SYMBOL DATA STATE
Slide 59
Slide 59 text
WALLAROO
RUNTIME
MESH NETWORK OF COOPERATING
PROCESSES
Slide 60
Slide 60 text
STATE OBJECTS
ONE BIG MAP?
Slide 61
Slide 61 text
STATE OBJECTS
CONCEPTUALLY ITS LIKE A BIG MAP
Market
Data Update State
Slide 62
Slide 62 text
STATE OBJECTS
WITH A KEY FOR EACH OBJECT
APPL IBM
MSFT AMZN
INTC NVDA
Market
Data Update
Slide 63
Slide 63 text
STATE OBJECTS
WHERE WE MAY FROM INCOMING DATA’S KEY
APPL IBM
MSFT AMZN
INTC NVDA
Market
Data MSFT
Slide 64
Slide 64 text
STATE OBJECTS
TO THE STATE OBJECT FOR THAT KEY
APPL IBM
MSFT AMZN
INTC NVDA
Market
Data MSFT
Slide 65
Slide 65 text
HASH PARTITIONING
DISTRIBUTING STATE OBJECTS
ACROSS A CLUSTER
Slide 66
Slide 66 text
SINGLE WORKER
ALL SYMBOLS TOGETHER
APPL
AMZN
MSFT
IBM
Slide 67
Slide 67 text
SINGLE WORKER
ALL SYMBOLS TOGETHER
APPL
AMZN
MSFT
IBM
Slide 68
Slide 68 text
ADD ANOTHER WORKER
STATE OBJECTS WILL BE REDISTRIBUTED ACROSS THE CLUSTER
APPL
AMZN
MSFT
IBM
Slide 69
Slide 69 text
ADD ANOTHER WORKER
STATE OBJECTS WILL BE REDISTRIBUTED ACROSS THE CLUSTER
APPL
AMZN
MSFT
IBM
Slide 70
Slide 70 text
ADD ANOTHER WORKER
STATE OBJECTS WILL BE REDISTRIBUTED ACROSS THE CLUSTER
APPL
AMZN
MSFT
IBM
Slide 71
Slide 71 text
ADD ANOTHER WORKER
STATE OBJECTS WILL BE REDISTRIBUTED ACROSS THE CLUSTER
APPL
AMZN
MSFT
IBM
Slide 72
Slide 72 text
ADD ANOTHER WORKER
STATE OBJECTS WILL BE REDISTRIBUTED ACROSS THE CLUSTER
APPL
AMZN
MSFT
IBM
Slide 73
Slide 73 text
ADD ANOTHER WORKER
STATE OBJECTS WILL BE REDISTRIBUTED ACROSS THE CLUSTER
APPL
AMZN
IBM
MSFT
Slide 74
Slide 74 text
ADD ANOTHER WORKER
STATE OBJECTS WILL BE REDISTRIBUTED ACROSS THE CLUSTER
APPL
AMZN
IBM
MSFT
Slide 75
Slide 75 text
STATE OBJECTS
Slide 76
Slide 76 text
A WALLAROO STATE OBJECT
PLAIN OLD PYTHON
Slide 77
Slide 77 text
LEARN MORE
GITHUB.COM/SEANTALLEN/
PAT-HELLAND-AND-ME