Coordination-Free Designs
for Mobile Gaming
Christopher Meiklejohn // @cmeik
Code Mesh 2015, September 3rd, 2015
1
Slide 2
Slide 2 text
RA
RB
Slide 3
Slide 3 text
RA
RB
1
set(1)
Slide 4
Slide 4 text
RA
RB
1
set(1)
3
2
set(2)
set(3)
Slide 5
Slide 5 text
RA
RB
1
set(1)
3
2
set(2)
set(3)
?
?
Slide 6
Slide 6 text
Synchronization
• To enforce an order
Makes programming easier
6
Slide 7
Slide 7 text
Synchronization
• To enforce an order
Makes programming easier
• Eliminate accidental nondeterminism
Prevent race conditions
6
Slide 8
Slide 8 text
Synchronization
• To enforce an order
Makes programming easier
• Eliminate accidental nondeterminism
Prevent race conditions
• Techniques
Locks, mutexes, semaphores, monitors, etc.
6
Slide 9
Slide 9 text
Difficult Cases
• “Internet of Things”
Low power, limited memory and connectivity
7
Slide 10
Slide 10 text
Difficult Cases
• “Internet of Things”
Low power, limited memory and connectivity
• Mobile Gaming
Offline operation with replicated, shared state
7
Slide 11
Slide 11 text
Zero Synchronization
• We want to have our cake and eat it too.
8
Slide 12
Slide 12 text
Zero Synchronization
• We want to have our cake and eat it too.
• “Zero synchronization” as a starting point
Add synchronization only where it is necessary
8
Slide 13
Slide 13 text
Zero Synchronization
• We want to have our cake and eat it too.
• “Zero synchronization” as a starting point
Add synchronization only where it is necessary
• Explore the limits
Can we design a model of computation where it’s
easy to write coordination-free computations that
are free from concurrency anomalies?
8
Slide 14
Slide 14 text
The Fundamentals of
Distributed Computation
9
Slide 15
Slide 15 text
Distributed Computation
• Distributed to concurrent programming
Consistency and now partial failure
10
Slide 16
Slide 16 text
Distributed Computation
• Distributed to concurrent programming
Consistency and now partial failure
• Enforcing the “single system” illusion
10
Slide 17
Slide 17 text
Distributed Computation
• Distributed to concurrent programming
Consistency and now partial failure
• Enforcing the “single system” illusion
• Consistency model as a contract
10
Slide 18
Slide 18 text
Distributed Computation
• Distributed to concurrent programming
Consistency and now partial failure
• Enforcing the “single system” illusion
• Consistency model as a contract
• Enforced through synchronization
10
Slide 19
Slide 19 text
Distributed Computation
• Distributed to concurrent programming
Consistency and now partial failure
• Enforcing the “single system” illusion
• Consistency model as a contract
• Enforced through synchronization
• Consistency model
We can think of a consistency model as analogous
to a programming paradigm
10
Slide 20
Slide 20 text
Why Is Synchronization
Undesirable?
11
Slide 21
Slide 21 text
The Avatars of Time
• The problem of time
Handling physical time in applications is difficult
12
Slide 22
Slide 22 text
The Avatars of Time
• The problem of time
Handling physical time in applications is difficult
• Time has three major avatars in computing
12
Slide 23
Slide 23 text
The Avatars of Time
• The problem of time
Handling physical time in applications is difficult
• Time has three major avatars in computing
• Mutable state in sequential systems
12
Slide 24
Slide 24 text
The Avatars of Time
• The problem of time
Handling physical time in applications is difficult
• Time has three major avatars in computing
• Mutable state in sequential systems
• Nondeterminism in concurrent systems
12
Slide 25
Slide 25 text
The Avatars of Time
• The problem of time
Handling physical time in applications is difficult
• Time has three major avatars in computing
• Mutable state in sequential systems
• Nondeterminism in concurrent systems
̣ Network latency in distributed systems
12
Slide 26
Slide 26 text
The Avatars of Time
• The problem of time
Handling physical time in applications is difficult
• Time has three major avatars in computing
• Mutable state in sequential systems
• Nondeterminism in concurrent systems
̣ Network latency in distributed systems
• Unavoidable
Time is how users interact with programs
12
Slide 27
Slide 27 text
Parable of the Car
• “Car driving on a highway”
13
Slide 28
Slide 28 text
Parable of the Car
• “Car driving on a highway”
• Friction needed for the car to grip the road
13
Slide 29
Slide 29 text
Parable of the Car
• “Car driving on a highway”
• Friction needed for the car to grip the road
• Motors want as little friction as possible
13
Slide 30
Slide 30 text
Parable of the Car
• “Car driving on a highway”
• Friction needed for the car to grip the road
• Motors want as little friction as possible
• Time is like friction
We can not completely eliminate friction, but aim to
reduce it as much as possible
13
Slide 31
Slide 31 text
Physical
Time
(real world)
Parable of the Car
14
• Time only at the interface
The interface is a small part of the system, and this is where we
introduction friction
Slide 32
Slide 32 text
Physical
Time
(real world)
Parable of the Car
14
• Time only at the interface
The interface is a small part of the system, and this is where we
introduction friction
• Physical time-free execution
Internally, avoid synchronization as much as possible and virtualize
notion of time
Slide 33
Slide 33 text
No Time In The Box?
15
Slide 34
Slide 34 text
No Time In The Box?
15
You program the box.
Slide 35
Slide 35 text
Weak Synchronization
• Can we achieve anything without synchronization?
Not really.
16
Slide 36
Slide 36 text
Weak Synchronization
• Can we achieve anything without synchronization?
Not really.
• Strong Eventual Consistency (SEC)
“Replicas that deliver the same updates have equivalent state”
16
Slide 37
Slide 37 text
Weak Synchronization
• Can we achieve anything without synchronization?
Not really.
• Strong Eventual Consistency (SEC)
“Replicas that deliver the same updates have equivalent state”
• Primary requirement
Eventual replica-to-replica communication
16
Slide 38
Slide 38 text
Weak Synchronization
• Can we achieve anything without synchronization?
Not really.
• Strong Eventual Consistency (SEC)
“Replicas that deliver the same updates have equivalent state”
• Primary requirement
Eventual replica-to-replica communication
• Order insensitive! (Commutativity)
16
Slide 39
Slide 39 text
Weak Synchronization
• Can we achieve anything without synchronization?
Not really.
• Strong Eventual Consistency (SEC)
“Replicas that deliver the same updates have equivalent state”
• Primary requirement
Eventual replica-to-replica communication
• Order insensitive! (Commutativity)
• Duplicate insensitive! (Idempotent)
16
Lattice Processing (Lasp)
• Distributed, deterministic dataflow
Distributed, deterministic dataflow programming
model for “eventually consistent” computations
33
Slide 60
Slide 60 text
Lattice Processing (Lasp)
• Distributed, deterministic dataflow
Distributed, deterministic dataflow programming
model for “eventually consistent” computations
• Convergent data structures
Primary data abstraction is the CRDT
33
Slide 61
Slide 61 text
Lattice Processing (Lasp)
• Distributed, deterministic dataflow
Distributed, deterministic dataflow programming
model for “eventually consistent” computations
• Convergent data structures
Primary data abstraction is the CRDT
• Enables composition
Provides functional composition of CRDTs that
preserves the SEC property
33
Slide 62
Slide 62 text
34
%% Create initial set.
S1 = declare(set),
%% Add elements to initial set and update.
update(S1, {add, [1,2,3]}),
%% Create second set.
S2 = declare(set),
%% Apply map operation between S1 and S2.
map(S1, fun(X) -> X * 2 end, S2).
Slide 63
Slide 63 text
35
%% Create initial set.
S1 = declare(set),
%% Add elements to initial set and update.
update(S1, {add, [1,2,3]}),
%% Create second set.
S2 = declare(set),
%% Apply map operation between S1 and S2.
map(S1, fun(X) -> X * 2 end, S2).
Slide 64
Slide 64 text
36
%% Create initial set.
S1 = declare(set),
%% Add elements to initial set and update.
update(S1, {add, [1,2,3]}),
%% Create second set.
S2 = declare(set),
%% Apply map operation between S1 and S2.
map(S1, fun(X) -> X * 2 end, S2).
Slide 65
Slide 65 text
37
%% Create initial set.
S1 = declare(set),
%% Add elements to initial set and update.
update(S1, {add, [1,2,3]}),
%% Create second set.
S2 = declare(set),
%% Apply map operation between S1 and S2.
map(S1, fun(X) -> X * 2 end, S2).
Slide 66
Slide 66 text
38
%% Create initial set.
S1 = declare(set),
%% Add elements to initial set and update.
update(S1, {add, [1,2,3]}),
%% Create second set.
S2 = declare(set),
%% Apply map operation between S1 and S2.
map(S1, fun(X) -> X * 2 end, S2).
Slide 67
Slide 67 text
Lattice Processing (Lasp)
• Functional and set-theoretic operations on sets
Product, intersection, union, filter, map, fold
39
Slide 68
Slide 68 text
Lattice Processing (Lasp)
• Functional and set-theoretic operations on sets
Product, intersection, union, filter, map, fold
• Metadata computation
Performs transformation on the internal metadata of
CRDTs allowing creation of “composed” CRDTs
39
Slide 69
Slide 69 text
Lasp Processes
• Replicas as monotonic streams
Each replica of a CRDT produces a monotonic
stream of states
40
Slide 70
Slide 70 text
Lasp Processes
• Replicas as monotonic streams
Each replica of a CRDT produces a monotonic
stream of states
• Monotonic processes
Read from one or more input replica streams and
produce a single output replica stream
40
Slide 71
Slide 71 text
Lasp Processes
• Replicas as monotonic streams
Each replica of a CRDT produces a monotonic
stream of states
• Monotonic processes
Read from one or more input replica streams and
produce a single output replica stream
• Inflationary reads
Read operation ensures that we only read
inflationary updates to replicas
40
Selective Hearing
• Epidemic broadcast based runtime system
Provide a runtime system that can scale to large numbers of
nodes, that is resilient to failures and provides efficient execution
59
Slide 91
Slide 91 text
Selective Hearing
• Epidemic broadcast based runtime system
Provide a runtime system that can scale to large numbers of
nodes, that is resilient to failures and provides efficient execution
• Well-matched to Lattice Processing (Lasp)
59
Slide 92
Slide 92 text
Selective Hearing
• Epidemic broadcast based runtime system
Provide a runtime system that can scale to large numbers of
nodes, that is resilient to failures and provides efficient execution
• Well-matched to Lattice Processing (Lasp)
• Epidemic broadcast mechanisms provide weak ordering but
are resilient and efficient
59
Slide 93
Slide 93 text
Selective Hearing
• Epidemic broadcast based runtime system
Provide a runtime system that can scale to large numbers of
nodes, that is resilient to failures and provides efficient execution
• Well-matched to Lattice Processing (Lasp)
• Epidemic broadcast mechanisms provide weak ordering but
are resilient and efficient
• Lasp’s programming model is tolerant to message re-ordering,
disconnections, and node failures
59
Slide 94
Slide 94 text
Selective Hearing
• Epidemic broadcast based runtime system
Provide a runtime system that can scale to large numbers of
nodes, that is resilient to failures and provides efficient execution
• Well-matched to Lattice Processing (Lasp)
• Epidemic broadcast mechanisms provide weak ordering but
are resilient and efficient
• Lasp’s programming model is tolerant to message re-ordering,
disconnections, and node failures
• “Selective Receive”
Nodes selectively receive and process messages based on
interest
59
Slide 95
Slide 95 text
Programming SEC
1. Eliminate accidental nondeterminism
(ex. deterministic, modeling non-monotonic operations monotonically)
2. Retain the properties of functional programming
(ex. confluence, referential transparency over composition)
3. Distributed, and fault-tolerant runtime
(ex. replication)
60
Slide 96
Slide 96 text
What can we build?
Leaderboard
61
Slide 97
Slide 97 text
Leaderboard
• Mobile game platform
Local leaderboard tracking top-k highest scored
games
62
Slide 98
Slide 98 text
Leaderboard
• Mobile game platform
Local leaderboard tracking top-k highest scored
games
• Clients will go offline
Clients have limited connectivity and the system still
needs to make progress while clients are offline
62
Leaderboard
• Peer-to-peer dissemination
Nodes periodically “merge” their state with a
random peer
64
Slide 101
Slide 101 text
Leaderboard
• Peer-to-peer dissemination
Nodes periodically “merge” their state with a
random peer
• Complexity in the data type
Each node tracks a top-k set of its own games in a
bounded set
64
Slide 102
Slide 102 text
65
%% Create a leaderboard datatype.
L = declare({top_k, [2]}).
%% Update leaderboard.
update({set, Name, Score}, L).
Slide 103
Slide 103 text
66
%% Create a leaderboard datatype.
L = declare({top_k, [2]}).
%% Update leaderboard.
update({set, Name, Score}, L).
Slide 104
Slide 104 text
67
%% Create a leaderboard datatype.
L = declare({top_k, [2]}).
%% Update leaderboard.
update({set, Name, Score}, L).
Slide 105
Slide 105 text
What if we want to enhance
the behavior?
68
Slide 106
Slide 106 text
What if we want to enhance
the behavior?
68
Without the creation of a
new datatype
Slide 107
Slide 107 text
What can we build?
Per-User Leaderboard
69
EdgeCom 2016
Slide 108
Slide 108 text
Per-User Leaderboard
• Enhance existing design
Only the top score for each user at each device
70
Slide 109
Slide 109 text
Per-User Leaderboard
• Enhance existing design
Only the top score for each user at each device
• Minimize transmitted state
Prevent transmission of state that is not necessary
to perform the computation
70
Slide 110
Slide 110 text
Per-User Leaderboard
• Enhance existing design
Only the top score for each user at each device
• Minimize transmitted state
Prevent transmission of state that is not necessary
to perform the computation
• Compose data types
Build a per-user leaderboard through the
composition of existing types
70
Slide 111
Slide 111 text
Client1
Scores
Local
Top-K
Fold
Global
Top-K
Lasp Operation
Input
User-Maintained CRDT
Output
Lasp-Maintained CRDT Fold
Client3
Scores
Local
Top-K
Fold
Global
Top-K
Fold
Client2
Scores
Local
Top-K
Fold
Global
Top-K
Fold
71
Slide 112
Slide 112 text
Client1
Scores
Local
Top-K
Fold
Global
Top-K
Fold
72
Client1
Scores
Local
Top-K
Fold
Global
Top-K
Fold
Client3
Scores
Local
Top-K
Fold
Global
Top-K
Fold
Client2
Scores
Local
Top-K
Fold
Global
Top-K
Fold
Slide 113
Slide 113 text
Client1
Scores
Local
Top-K
Fold
Global
Top-K
Fold
73
Client1
Scores
Local
Top-K
Fold
Global
Top-K
Fold
Client3
Scores
Local
Top-K
Fold
Global
Top-K
Fold
Client2
Scores
Local
Top-K
Fold
Global
Top-K
Fold
Slide 114
Slide 114 text
Client1
Scores
Local
Top-K
Fold
Global
Top-K
Fold
74
Client1
Scores
Local
Top-K
Fold
Global
Top-K
Fold
Client3
Scores
Local
Top-K
Fold
Global
Top-K
Fold
Client2
Scores
Local
Top-K
Fold
Global
Top-K
Fold
Slide 115
Slide 115 text
Client1
Scores
Local
Top-K
Fold
Global
Top-K
Fold
75
Client1
Scores
Local
Top-K
Fold
Global
Top-K
Fold
Client3
Scores
Local
Top-K
Fold
Global
Top-K
Fold
Client2
Scores
Local
Top-K
Fold
Global
Top-K
Fold
Slide 116
Slide 116 text
Client1
Scores
Local
Top-K
Fold
Global
Top-K
Fold
76
Client1
Scores
Local
Top-K
Fold
Global
Top-K
Fold
Client3
Scores
Local
Top-K
Fold
Global
Top-K
Fold
Client2
Scores
Local
Top-K
Fold
Global
Top-K
Fold
Slide 117
Slide 117 text
Client1
Scores
Local
Top-K
Fold
Global
Top-K
Fold
77
Client1
Scores
Local
Top-K
Fold
Global
Top-K
Fold
Client3
Scores
Local
Top-K
Fold
Global
Top-K
Fold
Client2
Scores
Local
Top-K
Fold
Global
Top-K
Fold
Slide 118
Slide 118 text
Client1
Scores
Local
Top-K
Fold
Global
Top-K
Lasp Operation
Input
User-Maintained CRDT
Output
Lasp-Maintained CRDT Fold
Client3
Scores
Local
Top-K
Fold
Global
Top-K
Fold
Client2
Scores
Local
Top-K
Fold
Global
Top-K
Fold
78
Client1
Scores
Local
Top-K
Fold
Global
Top-K
Fold
Client3
Scores
Local
Top-K
Fold
Global
Top-K
Fold
Client2
Scores
Local
Top-K
Fold
Global
Top-K
Fold
Slide 119
Slide 119 text
Per-User Leaderboard
• Dynamically scoped variables
Variable which take different values depending on
where it is executing
79
Slide 120
Slide 120 text
Per-User Leaderboard
• Dynamically scoped variables
Variable which take different values depending on
where it is executing
• Dynamically scoped fold operation
Perform a distributed “reduce” operation that
combines the state of a dynamically scoped
variables across
79
Slide 121
Slide 121 text
80
%% Create a global leaderboard.
G = declare({top_k, [10]}).
%% Create a local leaderboard.
L = declare_dynamic({top_k, [10]}).
%% Create a set of scores.
S = declare_dynamic(set).
%% Compute local top-k list.
fold(S, fun max_by_name/2, L).
%% Compute global top-k list.
fold_dynamic(L, fun max_by_name/2, G).
Slide 122
Slide 122 text
81
%% Create a global leaderboard.
G = declare({top_k, [10]}).
%% Create a local leaderboard.
L = declare_dynamic({top_k, [10]}).
%% Create a set of scores.
S = declare_dynamic(set).
%% Compute local top-k list.
fold(S, fun max_by_name/2, L).
%% Compute global top-k list.
fold_dynamic(L, fun max_by_name/2, G).
Slide 123
Slide 123 text
82
%% Create a global leaderboard.
G = declare({top_k, [10]}).
%% Create a local leaderboard.
L = declare_dynamic({top_k, [10]}).
%% Create a set of scores.
S = declare_dynamic(set).
%% Compute local top-k list.
fold(S, fun max_by_name/2, L).
%% Compute global top-k list.
fold_dynamic(L, fun max_by_name/2, G).
Slide 124
Slide 124 text
83
%% Create a global leaderboard.
G = declare({top_k, [10]}).
%% Create a local leaderboard.
L = declare_dynamic({top_k, [10]}).
%% Create a set of scores.
S = declare_dynamic(set).
%% Compute local top-k list.
fold(S, fun max_by_name/2, L).
%% Compute global top-k list.
fold_dynamic(L, fun max_by_name/2, G).
Slide 125
Slide 125 text
84
%% Create a global leaderboard.
G = declare({top_k, [10]}).
%% Create a local leaderboard.
L = declare_dynamic({top_k, [10]}).
%% Create a set of scores.
S = declare_dynamic(set).
%% Compute local top-k list.
fold(S, fun max_by_name/2, L).
%% Compute global top-k list.
fold_dynamic(L, fun max_by_name/2, G).
Slide 126
Slide 126 text
85
%% Create a global leaderboard.
G = declare({top_k, [10]}).
%% Create a local leaderboard.
L = declare_dynamic({top_k, [10]}).
%% Create a set of scores.
S = declare_dynamic(set).
%% Compute local top-k list.
fold(S, fun max_by_name/2, L).
%% Compute global top-k list.
fold_dynamic(L, fun max_by_name/2, G).
Slide 127
Slide 127 text
Let’s look at a larger
application
86
Slide 128
Slide 128 text
Let’s look at a larger
application
86
With visible
non-monotonicity
Slide 129
Slide 129 text
What can we build?
Advertisement Counter
87
Slide 130
Slide 130 text
Advertisement Counter
• Mobile game platform selling advertisement space
Advertisements are paid according to a minimum
number of impressions
88
Slide 131
Slide 131 text
Advertisement Counter
• Mobile game platform selling advertisement space
Advertisements are paid according to a minimum
number of impressions
• Clients will go offline
Clients have limited connectivity and the system still
needs to make progress while clients are offline
88
Slide 132
Slide 132 text
Ads
Rovio Ad
Counter 1
Rovio Ad
Counter 2
Riot Ad
Counter 1
Riot Ad
Counter 2
Contracts
Ads
Contracts
Ads
With
Contracts
Riot Ads
Rovio
Ads
Filter
Product
Read
50,000
Remove
Increment
Read
Union
Lasp Operation
User-Maintained CRDT
Lasp-Maintained CRDT
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Client Side, Single Copy at Client
89
Slide 133
Slide 133 text
Ads
Rovio Ad
Counter 1
Rovio Ad
Counter 2
Riot Ad
Counter 1
Riot Ad
Counter 2
Contracts
Ads
Contracts
Riot Ads
Rovio
Ads
Product
Read
50,000
Remove
Increment
Union
90
Ads
Rovio Ad
Counter 1
Rovio Ad
Counter 2
Riot Ad
Counter 1
Riot Ad
Counter 2
Contracts
Ads
Contracts
Ads
With
Contracts
Riot Ads
Rovio
Ads
Filter
Product
Read
50,000
Remove
Increment
Read
Union
Lasp Operation
User-Maintained CRDT
Lasp-Maintained CRDT
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Client Side, Single Copy at Client
Slide 134
Slide 134 text
Ads
Rovio Ad
Counter 1
Rovio Ad
Counter 2
Riot Ad
Counter 1
Riot Ad
Counter 2
Contracts
Ads
Contracts
Ads
With
Contracts
Riot Ads
Rovio
Ads
Filter
Product
Read
50,000
Remove
Increment
Read
Union
Rovio Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
1
Client
91
Ads
Rovio Ad
Counter 1
Rovio Ad
Counter 2
Riot Ad
Counter 1
Riot Ad
Counter 2
Contracts
Ads
Contracts
Ads
With
Contracts
Riot Ads
Rovio
Ads
Filter
Product
Read
50,000
Remove
Increment
Read
Union
Lasp Operation
User-Maintained CRDT
Lasp-Maintained CRDT
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Client Side, Single Copy at Client
Slide 135
Slide 135 text
Ads
ovio Ad
ounter 1
ovio Ad
ounter 2
iot Ad
ounter 1
iot Ad
ounter 2
Contracts
Ads
Contracts
Ads
With
Contracts
Riot Ads
Rovio
Ads
Filter
Product
Read
50,000
Remove
Increment
Read
Union
Rovio Ad
Counter
1
Ro
C
Rovio Ad
Counter
1
Ro
C
Rovio Ad
Counter
1
Ro
C
Rovio Ad
Counter
1
Ro
C
Client Side, Sing
92
Ads
Rovio Ad
Counter 1
Rovio Ad
Counter 2
Riot Ad
Counter 1
Riot Ad
Counter 2
Contracts
Ads
Contracts
Ads
With
Contracts
Riot Ads
Rovio
Ads
Filter
Product
Read
50,000
Remove
Increment
Read
Union
Lasp Operation
User-Maintained CRDT
Lasp-Maintained CRDT
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Client Side, Single Copy at Client
Slide 136
Slide 136 text
Ads
Contracts
Ads
Contracts
Ads
With
Contracts
Riot Ads
Rovio
Ads
Filter
Product
move Read
Union
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Client Side, Single Copy at Client
93
Ads
Rovio Ad
Counter 1
Rovio Ad
Counter 2
Riot Ad
Counter 1
Riot Ad
Counter 2
Contracts
Ads
Contracts
Ads
With
Contracts
Riot Ads
Rovio
Ads
Filter
Product
Read
50,000
Remove
Increment
Read
Union
Lasp Operation
User-Maintained CRDT
Lasp-Maintained CRDT
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Client Side, Single Copy at Client
Slide 137
Slide 137 text
Ads
Contracts
Ads
Contracts
Ads
With
Contracts
Filter
Product
Read
Union
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Client Side, Single Copy at Client
94
Ads
Rovio Ad
Counter 1
Rovio Ad
Counter 2
Riot Ad
Counter 1
Riot Ad
Counter 2
Contracts
Ads
Contracts
Ads
With
Contracts
Riot Ads
Rovio
Ads
Filter
Product
Read
50,000
Remove
Increment
Read
Union
Lasp Operation
User-Maintained CRDT
Lasp-Maintained CRDT
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Client Side, Single Copy at Client
Slide 138
Slide 138 text
Ads
Rovio Ad
Counter 1
Rovio Ad
Counter 2
Riot Ad
Counter 1
Riot Ad
Counter 2
Contracts
Ads
Contracts
Ads
With
Contracts
Riot Ads
Rovio
Ads
Filter
Product
Read
50,000
Remove
Increment
Read
Union
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Client Side, Single Copy at Client
95
Ads
Rovio Ad
Counter 1
Rovio Ad
Counter 2
Riot Ad
Counter 1
Riot Ad
Counter 2
Contracts
Ads
Contracts
Ads
With
Contracts
Riot Ads
Rovio
Ads
Filter
Product
Read
50,000
Remove
Increment
Read
Union
Lasp Operation
User-Maintained CRDT
Lasp-Maintained CRDT
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Client Side, Single Copy at Client
Slide 139
Slide 139 text
Ads
Rovio Ad
Counter 1
Rovio Ad
Counter 2
Riot Ad
Counter 1
Riot Ad
Counter 2
Contracts
Ads
Contracts
Riot Ads
Rovio
Ads
Fil
Product
Read
50,000
Remove
Increment
Union
96
Ads
Rovio Ad
Counter 1
Rovio Ad
Counter 2
Riot Ad
Counter 1
Riot Ad
Counter 2
Contracts
Ads
Contracts
Ads
With
Contracts
Riot Ads
Rovio
Ads
Filter
Product
Read
50,000
Remove
Increment
Read
Union
Lasp Operation
User-Maintained CRDT
Lasp-Maintained CRDT
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Client Side, Single Copy at Client
Slide 140
Slide 140 text
Ads
Rovio Ad
Counter 1
Rovio Ad
Counter 2
Riot Ad
Counter 1
Riot Ad
Counter 2
Contracts
Ads
Contracts
Ads
With
Contracts
Riot Ads
Rovio
Ads
Filter
Product
Read
50,000
Remove
Increment
Read
Union
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Client Side, Single Copy at Client
97
Ads
Rovio Ad
Counter 1
Rovio Ad
Counter 2
Riot Ad
Counter 1
Riot Ad
Counter 2
Contracts
Ads
Contracts
Ads
With
Contracts
Riot Ads
Rovio
Ads
Filter
Product
Read
50,000
Remove
Increment
Read
Union
Lasp Operation
User-Maintained CRDT
Lasp-Maintained CRDT
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Riot Ad
Counter
1
Client Side, Single Copy at Client
Slide 141
Slide 141 text
Advertisement Counter
• Completely monotonic
Disabling advertisements and contracts are all
modeled through monotonic state growth
98
Slide 142
Slide 142 text
Advertisement Counter
• Completely monotonic
Disabling advertisements and contracts are all
modeled through monotonic state growth
• Arbitrary distribution
Use of convergent data structures allows
computational graph to be arbitrarily distributed
98
Slide 143
Slide 143 text
Advertisement Counter
• Completely monotonic
Disabling advertisements and contracts are all
modeled through monotonic state growth
• Arbitrary distribution
Use of convergent data structures allows
computational graph to be arbitrarily distributed
• Divergence
Divergence is a factor of synchronization period
98
Slide 144
Slide 144 text
99
%% Generate a series of contracts.
Contracts = declare(set),
%% Generate advertisements.
RiotAds = declare(set),
RovioAds = declare(set),
create_advertisements_and_contracts(RiotAds, Contracts),
create_advertisements_and_contracts(RovioAds, Contracts),
%% Union ads.
Ads = declare(set),
union(RovioAds, RiotAds, Ads),
%% Compute the product of both ads and contracts.
AdsContracts = declare(set),
product(Ads, Contracts, AdsContracts),
%% Filter items by join on item it.
AdsWithContracts = declare(set),
FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) ->
Id1 =:= Id2
end,
filter(AdsContracts, FilterFun, AdsWithContracts).
Slide 145
Slide 145 text
100
%% Generate a series of contracts.
Contracts = declare(set),
%% Generate advertisements.
RiotAds = declare(set),
RovioAds = declare(set),
create_advertisements_and_contracts(RiotAds, Contracts),
create_advertisements_and_contracts(RovioAds, Contracts),
%% Union ads.
Ads = declare(set),
union(RovioAds, RiotAds, Ads),
%% Compute the product of both ads and contracts.
AdsContracts = declare(set),
product(Ads, Contracts, AdsContracts),
%% Filter items by join on item it.
AdsWithContracts = declare(set),
FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) ->
Id1 =:= Id2
end,
filter(AdsContracts, FilterFun, AdsWithContracts).
Slide 146
Slide 146 text
101
%% Generate a series of contracts.
Contracts = declare(set),
%% Generate advertisements.
RiotAds = declare(set),
RovioAds = declare(set),
create_advertisements_and_contracts(RiotAds, Contracts),
create_advertisements_and_contracts(RovioAds, Contracts),
%% Union ads.
Ads = declare(set),
union(RovioAds, RiotAds, Ads),
%% Compute the product of both ads and contracts.
AdsContracts = declare(set),
product(Ads, Contracts, AdsContracts),
%% Filter items by join on item it.
AdsWithContracts = declare(set),
FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) ->
Id1 =:= Id2
end,
filter(AdsContracts, FilterFun, AdsWithContracts).
Slide 147
Slide 147 text
102
%% Generate a series of contracts.
Contracts = declare(set),
%% Generate advertisements.
RiotAds = declare(set),
RovioAds = declare(set),
create_advertisements_and_contracts(RiotAds, Contracts),
create_advertisements_and_contracts(RovioAds, Contracts),
%% Union ads.
Ads = declare(set),
union(RovioAds, RiotAds, Ads),
%% Compute the product of both ads and contracts.
AdsContracts = declare(set),
product(Ads, Contracts, AdsContracts),
%% Filter items by join on item it.
AdsWithContracts = declare(set),
FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) ->
Id1 =:= Id2
end,
filter(AdsContracts, FilterFun, AdsWithContracts).
Slide 148
Slide 148 text
103
%% Generate a series of contracts.
Contracts = declare(set),
%% Generate advertisements.
RiotAds = declare(set),
RovioAds = declare(set),
create_advertisements_and_contracts(RiotAds, Contracts),
create_advertisements_and_contracts(RovioAds, Contracts),
%% Union ads.
Ads = declare(set),
union(RovioAds, RiotAds, Ads),
%% Compute the product of both ads and contracts.
AdsContracts = declare(set),
product(Ads, Contracts, AdsContracts),
%% Filter items by join on item it.
AdsWithContracts = declare(set),
FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) ->
Id1 =:= Id2
end,
filter(AdsContracts, FilterFun, AdsWithContracts).
Slide 149
Slide 149 text
104
%% Generate a series of contracts.
Contracts = declare(set),
%% Generate advertisements.
RiotAds = declare(set),
RovioAds = declare(set),
create_advertisements_and_contracts(RiotAds, Contracts),
create_advertisements_and_contracts(RovioAds, Contracts),
%% Union ads.
Ads = declare(set),
union(RovioAds, RiotAds, Ads),
%% Compute the product of both ads and contracts.
AdsContracts = declare(set),
product(Ads, Contracts, AdsContracts),
%% Filter items by join on item it.
AdsWithContracts = declare(set),
FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) ->
Id1 =:= Id2
end,
filter(AdsContracts, FilterFun, AdsWithContracts).
Slide 150
Slide 150 text
Client3
Lasp Operation
User-Maintained CRDT
Lasp-Maintained CRDT
Riot Ad
Counter
1
Client1
Client2
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Rovio Ad
Counter
1
Riot Ad
Counter
2
Riot Ad
Counter
1
Riot Ad
Counter
1
Ads
With
Contracts
Ads
With
Contracts
Ads
With
Contracts
Server
Ads
With
Contracts
Server
Computation!
105
Slide 151
Slide 151 text
Client3
Lasp Operation
User-Maintained CRDT
Lasp-Maintained CRDT
Riot Ad
Counter
1
Client1
Client2
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Rovio Ad
Counter
1
Riot Ad
Counter
2
Riot Ad
Counter
1
Riot Ad
Counter
1
Ads
With
Contracts
Ads
With
Contracts
Ads
With
Contracts
Server
Ads
With
Contracts
Server
Computation!
106
Slide 152
Slide 152 text
Client3
Lasp Operation
User-Maintained CRDT
Lasp-Maintained CRDT
Riot Ad
Counter
1
Client1
Client2
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Rovio Ad
Counter
1
Rovio Ad
Counter
2
Rovio Ad
Counter
1
Riot Ad
Counter
2
Riot Ad
Counter
1
Riot Ad
Counter
1
Ads
With
Contracts
Ads
With
Contracts
Ads
With
Contracts
Server
Ads
With
Contracts
Server
Computation!
107
Slide 153
Slide 153 text
Advertisement Counter
• “Servers” as peers to “clients”
Servers are peers to clients that perform additional
computation
108
Slide 154
Slide 154 text
Advertisement Counter
• “Servers” as peers to “clients”
Servers are peers to clients that perform additional
computation
• Any node can disable an advertisement under this
model given enough information
108
Slide 155
Slide 155 text
Advertisement Counter
• “Servers” as peers to “clients”
Servers are peers to clients that perform additional
computation
• Any node can disable an advertisement under this
model given enough information
• “Servers” as trusted nodes
Serve as a location for performing “exactly once” side-
effects
108
Slide 156
Slide 156 text
Advertisement Counter
• “Servers” as peers to “clients”
Servers are peers to clients that perform additional
computation
• Any node can disable an advertisement under this
model given enough information
• “Servers” as trusted nodes
Serve as a location for performing “exactly once” side-
effects
• Billing customers must be done at a central point by a
trusted node in the system
108
Slide 157
Slide 157 text
We’ve build up from zero
synchronization
109
Slide 158
Slide 158 text
We’ve build up from zero
synchronization
109
Instead of working to
remove synchronization
Slide 159
Slide 159 text
What have we learned?
110
Slide 160
Slide 160 text
Key Points
• Synchronization is expensive
Locking and other synchronization mechanisms limit
scalability to the critical section
111
Slide 161
Slide 161 text
Key Points
• Synchronization is expensive
Locking and other synchronization mechanisms limit
scalability to the critical section
• Synchronization is sometimes not possible
Mobile and “Internet of Things” applications make
synchronization for replicated state impractical
111
Slide 162
Slide 162 text
Key Points
• Synchronization is expensive
Locking and other synchronization mechanisms limit
scalability to the critical section
• Synchronization is sometimes not possible
Mobile and “Internet of Things” applications make
synchronization for replicated state impractical
• Apply synchronization only where required
Global invariants, atomic visibility, etc.
111
Slide 163
Slide 163 text
How do I learn more?
112
Slide 164
Slide 164 text
Publications
• “Lasp: A Language for Distributed, Coordination-Free Programming”
ACM SIGPLAN PPDP 2015
• “Selective Hearing: An Approach to Distributed, Eventually Consistent Edge Computation”
IEEE W-PSDS 2015
• “The Implementation and Use of a Generic Dataflow Behaviour in Erlang”
ACM SIGPLAN Erlang Workshop ’15
• “Lasp: A Language for Distributed, Eventually Consistent Computations with CRDTs"
PaPoC 2015
• “Declarative, Sliding Window Aggregations for Computations at the Edge"
IEEE EdgeCom 2016
113
Slide 165
Slide 165 text
SyncFree is a European research project taking place
for 3 years, staring October 2013, and is funded by
the European Union, grant agreement n° 609551.
114