eat it too. • “Zero synchronization” as a starting point Add synchronization only where it is necessary • Explore the limits Can we design a model of computation where it’s easy to write coordination-free computations that are free from concurrency anomalies? 8
partial failure • Enforcing the “single system” illusion • Consistency model as a contract • Enforced through synchronization • Consistency model We can think of a consistency model as analogous to a programming paradigm 10
physical time in applications is difﬁcult • Time has three major avatars in computing • Mutable state in sequential systems • Nondeterminism in concurrent systems 12
physical time in applications is difﬁcult • Time has three major avatars in computing • Mutable state in sequential systems • Nondeterminism in concurrent systems ̣ Network latency in distributed systems 12
physical time in applications is difﬁcult • Time has three major avatars in computing • Mutable state in sequential systems • Nondeterminism in concurrent systems ̣ Network latency in distributed systems • Unavoidable Time is how users interact with programs 12
• Friction needed for the car to grip the road • Motors want as little friction as possible • Time is like friction We can not completely eliminate friction, but aim to reduce it as much as possible 13
Time only at the interface The interface is a small part of the system, and this is where we introduction friction • Physical time-free execution Internally, avoid synchronization as much as possible and virtualize notion of time
really. • Strong Eventual Consistency (SEC) “Replicas that deliver the same updates have equivalent state” • Primary requirement Eventual replica-to-replica communication 16
really. • Strong Eventual Consistency (SEC) “Replicas that deliver the same updates have equivalent state” • Primary requirement Eventual replica-to-replica communication • Order insensitive! (Commutativity) 16
really. • Strong Eventual Consistency (SEC) “Replicas that deliver the same updates have equivalent state” • Primary requirement Eventual replica-to-replica communication • Order insensitive! (Commutativity) • Duplicate insensitive! (Idempotent) 16
programming model for “eventually consistent” computations • Convergent data structures Primary data abstraction is the CRDT • Enables composition Provides functional composition of CRDTs that preserves the SEC property 33
elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
a CRDT produces a monotonic stream of states • Monotonic processes Read from one or more input replica streams and produce a single output replica stream 40
a CRDT produces a monotonic stream of states • Monotonic processes Read from one or more input replica streams and produce a single output replica stream • Inﬂationary reads Read operation ensures that we only read inﬂationary updates to replicas 40
runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efﬁcient execution • Well-matched to Lattice Processing (Lasp) 59
runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efﬁcient execution • Well-matched to Lattice Processing (Lasp) • Epidemic broadcast mechanisms provide weak ordering but are resilient and efﬁcient 59
runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efﬁcient execution • Well-matched to Lattice Processing (Lasp) • Epidemic broadcast mechanisms provide weak ordering but are resilient and efﬁcient • Lasp’s programming model is tolerant to message re-ordering, disconnections, and node failures 59
runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efﬁcient execution • Well-matched to Lattice Processing (Lasp) • Epidemic broadcast mechanisms provide weak ordering but are resilient and efﬁcient • Lasp’s programming model is tolerant to message re-ordering, disconnections, and node failures • “Selective Receive” Nodes selectively receive and process messages based on interest 59
for each user at each device • Minimize transmitted state Prevent transmission of state that is not necessary to perform the computation • Compose data types Build a per-user leaderboard through the composition of existing types 70
User-Maintained CRDT Output Lasp-Maintained CRDT Fold Client3 Scores Local Top-K Fold Global Top-K Fold Client2 Scores Local Top-K Fold Global Top-K Fold 78 Client1 Scores Local Top-K Fold Global Top-K Fold Client3 Scores Local Top-K Fold Global Top-K Fold Client2 Scores Local Top-K Fold Global Top-K Fold
values depending on where it is executing • Dynamically scoped fold operation Perform a distributed “reduce” operation that combines the state of a dynamically scoped variables across 79
%% Create a local leaderboard. L = declare_dynamic({top_k, [10]}). %% Create a set of scores. S = declare_dynamic(set). %% Compute local top-k list. fold(S, fun max_by_name/2, L). %% Compute global top-k list. fold_dynamic(L, fun max_by_name/2, G).
%% Create a local leaderboard. L = declare_dynamic({top_k, [10]}). %% Create a set of scores. S = declare_dynamic(set). %% Compute local top-k list. fold(S, fun max_by_name/2, L). %% Compute global top-k list. fold_dynamic(L, fun max_by_name/2, G).
%% Create a local leaderboard. L = declare_dynamic({top_k, [10]}). %% Create a set of scores. S = declare_dynamic(set). %% Compute local top-k list. fold(S, fun max_by_name/2, L). %% Compute global top-k list. fold_dynamic(L, fun max_by_name/2, G).
%% Create a local leaderboard. L = declare_dynamic({top_k, [10]}). %% Create a set of scores. S = declare_dynamic(set). %% Compute local top-k list. fold(S, fun max_by_name/2, L). %% Compute global top-k list. fold_dynamic(L, fun max_by_name/2, G).
%% Create a local leaderboard. L = declare_dynamic({top_k, [10]}). %% Create a set of scores. S = declare_dynamic(set). %% Compute local top-k list. fold(S, fun max_by_name/2, L). %% Compute global top-k list. fold_dynamic(L, fun max_by_name/2, G).
%% Create a local leaderboard. L = declare_dynamic({top_k, [10]}). %% Create a set of scores. S = declare_dynamic(set). %% Compute local top-k list. fold(S, fun max_by_name/2, L). %% Compute global top-k list. fold_dynamic(L, fun max_by_name/2, G).
are paid according to a minimum number of impressions • Clients will go oﬄine Clients have limited connectivity and the system still needs to make progress while clients are ofﬂine 88
Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 89
Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Rovio Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 1 Client 91 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
Ad ounter 1 iot Ad ounter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Rovio Ad Counter 1 Ro C Rovio Ad Counter 1 Ro C Rovio Ad Counter 1 Ro C Rovio Ad Counter 1 Ro C Client Side, Sing 92 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
Ads Filter Product move Read Union Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 93 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
Union Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 94 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 95 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 97 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
all modeled through monotonic state growth • Arbitrary distribution Use of convergent data structures allows computational graph to be arbitrarily distributed 98
all modeled through monotonic state growth • Arbitrary distribution Use of convergent data structures allows computational graph to be arbitrarily distributed • Divergence Divergence is a factor of synchronization period 98
1 Client1 Client2 Rovio Ad Counter 1 Rovio Ad Counter 2 Rovio Ad Counter 1 Rovio Ad Counter 2 Rovio Ad Counter 1 Riot Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 1 Ads With Contracts Ads With Contracts Ads With Contracts Server Ads With Contracts Server Computation! 105
1 Client1 Client2 Rovio Ad Counter 1 Rovio Ad Counter 2 Rovio Ad Counter 1 Rovio Ad Counter 2 Rovio Ad Counter 1 Riot Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 1 Ads With Contracts Ads With Contracts Ads With Contracts Server Ads With Contracts Server Computation! 106
1 Client1 Client2 Rovio Ad Counter 1 Rovio Ad Counter 2 Rovio Ad Counter 1 Rovio Ad Counter 2 Rovio Ad Counter 1 Riot Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 1 Ads With Contracts Ads With Contracts Ads With Contracts Server Ads With Contracts Server Computation! 107
peers to clients that perform additional computation • Any node can disable an advertisement under this model given enough information • “Servers” as trusted nodes Serve as a location for performing “exactly once” side- effects 108
peers to clients that perform additional computation • Any node can disable an advertisement under this model given enough information • “Servers” as trusted nodes Serve as a location for performing “exactly once” side- effects • Billing customers must be done at a central point by a trusted node in the system 108
mechanisms limit scalability to the critical section • Synchronization is sometimes not possible Mobile and “Internet of Things” applications make synchronization for replicated state impractical 111
mechanisms limit scalability to the critical section • Synchronization is sometimes not possible Mobile and “Internet of Things” applications make synchronization for replicated state impractical • Apply synchronization only where required Global invariants, atomic visibility, etc. 111
ACM SIGPLAN PPDP 2015 • “Selective Hearing: An Approach to Distributed, Eventually Consistent Edge Computation” IEEE W-PSDS 2015 • “The Implementation and Use of a Generic Dataﬂow Behaviour in Erlang” ACM SIGPLAN Erlang Workshop ’15 • “Lasp: A Language for Distributed, Eventually Consistent Computations with CRDTs" PaPoC 2015 • “Declarative, Sliding Window Aggregations for Computations at the Edge" IEEE EdgeCom 2016 113