PAT HELLAND AND ME
HOW TO BUILD STATEFUL DISTRIBUTED APPLICATIONS THAT CAN SCALE ALMOST INFINITELY
Slide 2
Slide 2 text
SEAN T. ALLEN
VP OF ENGINEERING AT WALLAROO LABS
MEMBER OF THE PONY CORE TEAM
AUTHOR OF “STORM APPLIED”
LOVER OF FRENCH STREET ART
@SEANTALLEN
@WALLAROOLABS
@PONYLANG
Slide 3
Slide 3 text
PAT HELLAND
AND ME
Slide 4
Slide 4 text
SOME AXIOMS…
Slide 5
Slide 5 text
TO SCALE INFINITELY,
WE HAVE TO SCALE HORIZONTALLY
Slide 6
Slide 6 text
TO SCALE INFINITELY,
WE MUST AVOID COORDINATION
Slide 7
Slide 7 text
DISTRIBUTED TRANSACTIONS ARE
A FORM OF COORDINATION
Slide 8
Slide 8 text
THEREFORE…
TO SCALE INFINITELY,
WE CAN’T USE TRANSACTIONS
Slide 9
Slide 9 text
WHAT IS SCALING?
Slide 10
Slide 10 text
MORE AND
MORE THINGS
BUT, THEY DON’T GET
BIGGER. THERE’S JUST…
MORE OF THEM. LOTS MORE.
Slide 11
Slide 11 text
WE SCALE
ENTITIES
ENTITIES:
LIVE ON A SINGLE MACHINE
AND ARE MANIPULATED
INDIVIDUALLY
Slide 12
Slide 12 text
WHAT IS AN ENTITY?
Slide 13
Slide 13 text
ENTITIES ARE BOUNDARIES OF
ATOMICITY
Slide 14
Slide 14 text
Bob
6 3
5
8
Alice
4 2
7
1
Slide 15
Slide 15 text
Bob
6 3
5
8
Alice
4 2
7
1
Slide 16
Slide 16 text
Bob
6 3
5
8
Alice
4
7
1
2
Slide 17
Slide 17 text
Bob
6 3
5
8
Alice
4 2
7
1
Slide 18
Slide 18 text
Bob
6 3
5
8
Alice
4 2
7
1
Slide 19
Slide 19 text
DENORMALIZE..
ALL THE THINGS!
Slide 20
Slide 20 text
TWO-LAYER ARCHITECTURE
Slide 21
Slide 21 text
scale-agnostic
scale-aware
API
Slide 22
Slide 22 text
scale-agnostic
scale-aware
API
Slide 23
Slide 23 text
scale-agnostic
scale-aware
API
Slide 24
Slide 24 text
scale-agnostic
scale-aware
API
Slide 25
Slide 25 text
scale-agnostic
scale-aware
API
Slide 26
Slide 26 text
scale-agnostic
scale-aware
API
Slide 27
Slide 27 text
TO SCALE INFINITELY,
YOUR BUSINESS LOGIC HAS TO BE
INDEPENDENT OF SCALE
Slide 28
Slide 28 text
WALLAROO
SCALE INDEPENDENT
COMPUTING FOR PYTHON
Slide 29
Slide 29 text
ENTITIES
BUT WE CALL THEM…
“STATE OBJECTS”
Slide 30
Slide 30 text
TWO-LAYER
ARCHITECTURE
BUT WE CALL IT…
“SCALE INDEPENDENCE”
Slide 31
Slide 31 text
user supplied logic
Wallaroo runtime
Wallaroo API
Slide 32
Slide 32 text
user supplied logic
Wallaroo runtime
Wallaroo API
Slide 33
Slide 33 text
user supplied logic
Wallaroo runtime
Wallaroo API
Slide 34
Slide 34 text
user supplied logic
Wallaroo runtime
Wallaroo API
Slide 35
Slide 35 text
WALLAROO API
MARKET SPREAD EXAMPLE
Slide 36
Slide 36 text
MARKET SPREAD
REAL-TIME “SOMETHING AIN’T RIGHT” TRADE CHECKS
Market Spread
State
Market
Data
Orders
Update
APPL
Check
MSFT
Rejections
Slide 37
Slide 37 text
MARKET SPREAD
TWO SOURCES OF DATA
Market Spread
State
Market
Data
Orders
Update
APPL
Check
MSFT
Rejections
Slide 38
Slide 38 text
MARKET SPREAD
ONE SINK
Market Spread
State
Market
Data
Orders
Update
APPL
Check
MSFT
Rejections
Slide 39
Slide 39 text
MARKET SPREAD
ORDER PIPELINE
Market Spread
State
Market
Data
Orders
Update
APPL
Check
MSFT
Rejections
Slide 40
Slide 40 text
MARKET SPREAD
MARKET DATA PIPELINE
Market Spread
State
Market
Data
Orders
Update
APPL
Check
MSFT
Rejections
Slide 41
Slide 41 text
MARKET SPREAD
APPLICATION DEFINITION
Slide 42
Slide 42 text
APPLICATION DEFINITION
FLOW OF DATA FROM SOURCE TO SINK
Slide 43
Slide 43 text
TWO DATA PIPELINES
ORDERS
Slide 44
Slide 44 text
TWO DATA PIPELINES
MARKET DATA
Slide 45
Slide 45 text
DEFINE OUR SOURCES
1 PER PIPELINE
Slide 46
Slide 46 text
DEFINE OUR OPERATIONS
1 PER PIPELINE
Slide 47
Slide 47 text
DEFINE OUR OPERATIONS
CHECK ORDER AGAINST SYMBOL DATA
Slide 48
Slide 48 text
DEFINE OUR OPERATIONS
UPDATE SYMBOL DATA WITH LATEST MARKET DATA
Slide 49
Slide 49 text
DEFINE OUR SINKS
1 PER PIPELINE
Slide 50
Slide 50 text
DEFINE OUR SINKS
ORDERS PIPELINE MIGHT HAVE OUTPUT
Slide 51
Slide 51 text
DEFINE OUR SINKS
MARKET DATA ONLY UPDATES SYMBOL DATA- NO OUTPUT
Slide 52
Slide 52 text
SCALE INDEPENDENT
ONLY FLOW OF DATA AND OPERATIONS
Slide 53
Slide 53 text
USER SUPPLIED
LOGIC
Slide 54
Slide 54 text
UPDATE MARKET DATA STATE COMPUTATION
UPDATES SYMBOL DATA STATE
Slide 55
Slide 55 text
WALLAROO
RUNTIME
MESH NETWORK OF
COOPERATING PROCESSES
Slide 56
Slide 56 text
STATE OBJECTS
ONE BIG MAP?
Slide 57
Slide 57 text
STATE OBJECTS
CONCEPTUALLY ITS LIKE A BIG MAP
Market
Data Update State
Slide 58
Slide 58 text
STATE OBJECTS
WITH A KEY FOR EACH OBJECT
APPL IBM
MSFT AMZN
INTC NVDA
Market
Data Update
Slide 59
Slide 59 text
STATE OBJECTS
WHERE WE MAY FROM INCOMING DATA’S KEY
APPL IBM
MSFT AMZN
INTC NVDA
Market
Data MSFT
Slide 60
Slide 60 text
STATE OBJECTS
TO THE STATE OBJECT FOR THAT KEY
APPL IBM
MSFT AMZN
INTC NVDA
Market
Data MSFT
Slide 61
Slide 61 text
HASH
PARTITIONING
DISTRIBUTING STATE
OBJECTS ACROSS A
CLUSTER
Slide 62
Slide 62 text
SINGLE WORKER
ALL SYMBOLS TOGETHER
APPL
AMZN
MSFT
IBM
Slide 63
Slide 63 text
SINGLE WORKER
ALL SYMBOLS TOGETHER
APPL
AMZN
MSFT
IBM
Slide 64
Slide 64 text
ADD ANOTHER WORKER
STATE OBJECTS WILL BE REDISTRIBUTED ACROSS THE CLUSTER
APPL
AMZN
MSFT
IBM
Slide 65
Slide 65 text
ADD ANOTHER WORKER
STATE OBJECTS WILL BE REDISTRIBUTED ACROSS THE CLUSTER
APPL
AMZN
MSFT
IBM
Slide 66
Slide 66 text
ADD ANOTHER WORKER
STATE OBJECTS WILL BE REDISTRIBUTED ACROSS THE CLUSTER
APPL
AMZN
MSFT
IBM
Slide 67
Slide 67 text
ADD ANOTHER WORKER
STATE OBJECTS WILL BE REDISTRIBUTED ACROSS THE CLUSTER
APPL
AMZN
MSFT
IBM
Slide 68
Slide 68 text
ADD ANOTHER WORKER
STATE OBJECTS WILL BE REDISTRIBUTED ACROSS THE CLUSTER
APPL
AMZN
MSFT
IBM
Slide 69
Slide 69 text
ADD ANOTHER WORKER
STATE OBJECTS WILL BE REDISTRIBUTED ACROSS THE CLUSTER
APPL
AMZN
IBM
MSFT
Slide 70
Slide 70 text
ADD ANOTHER WORKER
STATE OBJECTS WILL BE REDISTRIBUTED ACROSS THE CLUSTER
APPL
AMZN
IBM
MSFT
Slide 71
Slide 71 text
DATA MODEL
YOU CAN GET WITH THIS, OR
YOU CAN GET WITH THAT.
Slide 72
Slide 72 text
A WALLAROO STATE OBJECT
PLAIN OLD PYTHON
Slide 73
Slide 73 text
PERFORMANCE
THERE’S NO REAL WORLD
SCALING WITHOUT IT
Slide 74
Slide 74 text
LEARN MORE
GITHUB.COM/SEANTALLEN/
PAT-HELLAND-AND-ME