Synchronization • To enforce an order Makes programming easier • Eliminate accidental nondeterminism Prevent race conditions • Techniques Locks, mutexes, semaphores, monitors, etc. 6
Zero Synchronization • We want to have our cake and eat it too. • “Zero synchronization” as a starting point Add synchronization only where it is necessary 8
Zero Synchronization • We want to have our cake and eat it too. • “Zero synchronization” as a starting point Add synchronization only where it is necessary • Explore the limits Can we design a model of computation where it’s easy to write coordination-free computations that are free from concurrency anomalies? 8
Distributed Computation • Distributed to concurrent programming Consistency and now partial failure • Enforcing the “single system” illusion • Consistency model as a contract 11
Distributed Computation • Distributed to concurrent programming Consistency and now partial failure • Enforcing the “single system” illusion • Consistency model as a contract • Enforced through synchronization 11
Distributed Computation • Distributed to concurrent programming Consistency and now partial failure • Enforcing the “single system” illusion • Consistency model as a contract • Enforced through synchronization • Consistency model We can think of a consistency model as analogous to a programming paradigm 11
The Avatars of Time • The problem of time Handling physical time in applications is difficult • Time has three major avatars in computing • Mutable state in sequential systems 13
The Avatars of Time • The problem of time Handling physical time in applications is difficult • Time has three major avatars in computing • Mutable state in sequential systems • Nondeterminism in concurrent systems 13
The Avatars of Time • The problem of time Handling physical time in applications is difficult • Time has three major avatars in computing • Mutable state in sequential systems • Nondeterminism in concurrent systems ̣ Network latency in distributed systems 13
The Avatars of Time • The problem of time Handling physical time in applications is difficult • Time has three major avatars in computing • Mutable state in sequential systems • Nondeterminism in concurrent systems ̣ Network latency in distributed systems • Unavoidable Time is how users interact with programs 13
Parable of the Car • “Car driving on a highway” • Friction needed for the car to grip the road • Motors want as little friction as possible • Time is like friction We can not completely eliminate friction, but aim to reduce it as much as possible 14
Physical Time (real world) Parable of the Car 15 • Time only at the interface The interface is a small part of the system, and this is where we introduction friction
Physical Time (real world) Parable of the Car 15 • Time only at the interface The interface is a small part of the system, and this is where we introduction friction • Physical time-free execution Internally, avoid synchronization as much as possible and virtualize notion of time
Weak Synchronization • Can we achieve anything without synchronization? Not really. • Strong Eventual Consistency (SEC) “Replicas that deliver the same updates have equivalent state” 17
Weak Synchronization • Can we achieve anything without synchronization? Not really. • Strong Eventual Consistency (SEC) “Replicas that deliver the same updates have equivalent state” • Primary requirement Eventual replica-to-replica communication 17
Weak Synchronization • Can we achieve anything without synchronization? Not really. • Strong Eventual Consistency (SEC) “Replicas that deliver the same updates have equivalent state” • Primary requirement Eventual replica-to-replica communication • Order insensitive! (Commutativity) 17
Weak Synchronization • Can we achieve anything without synchronization? Not really. • Strong Eventual Consistency (SEC) “Replicas that deliver the same updates have equivalent state” • Primary requirement Eventual replica-to-replica communication • Order insensitive! (Commutativity) • Duplicate insensitive! (Idempotent) 17
Lattice Processing (Lasp) • Distributed, deterministic dataflow Distributed, deterministic dataflow programming model for “eventually consistent” computations • Convergent data structures Primary data abstraction is the CRDT 34
Lattice Processing (Lasp) • Distributed, deterministic dataflow Distributed, deterministic dataflow programming model for “eventually consistent” computations • Convergent data structures Primary data abstraction is the CRDT • Enables composition Provides functional composition of CRDTs that preserves the SEC property 34
Lasp Processes • Replicas as monotonic streams Each replica of a CRDT produces a monotonic stream of states • Monotonic processes Read from one or more input replica streams and produce a single output replica stream 41
Lasp Processes • Replicas as monotonic streams Each replica of a CRDT produces a monotonic stream of states • Monotonic processes Read from one or more input replica streams and produce a single output replica stream • Inflationary reads Read operation ensures that we only read inflationary updates to replicas 41
Selective Hearing • Epidemic broadcast based runtime system Provide a runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efficient execution 60
Selective Hearing • Epidemic broadcast based runtime system Provide a runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efficient execution • Well-matched to Lattice Processing (Lasp) 60
Selective Hearing • Epidemic broadcast based runtime system Provide a runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efficient execution • Well-matched to Lattice Processing (Lasp) • Epidemic broadcast mechanisms provide weak ordering but are resilient and efficient 60
Selective Hearing • Epidemic broadcast based runtime system Provide a runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efficient execution • Well-matched to Lattice Processing (Lasp) • Epidemic broadcast mechanisms provide weak ordering but are resilient and efficient • Lasp’s programming model is tolerant to message re-ordering, disconnections, and node failures 60
Selective Hearing • Epidemic broadcast based runtime system Provide a runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efficient execution • Well-matched to Lattice Processing (Lasp) • Epidemic broadcast mechanisms provide weak ordering but are resilient and efficient • Lasp’s programming model is tolerant to message re-ordering, disconnections, and node failures • “Selective Receive” Nodes selectively receive and process messages based on interest 60
Leaderboard • Mobile game platform Local leaderboard tracking top-k highest scored games • Clients will go offline Clients have limited connectivity and the system still needs to make progress while clients are offline 63
Leaderboard • Peer-to-peer dissemination Nodes periodically “merge” their state with a random peer • Complexity in the data type Each node tracks a top-k set of its own games in a bounded set 65
Per-User Leaderboard • Enhance existing design Only the top score for each user at each device • Minimize transmitted state Prevent transmission of state that is not necessary to perform the computation 71
Per-User Leaderboard • Enhance existing design Only the top score for each user at each device • Minimize transmitted state Prevent transmission of state that is not necessary to perform the computation • Compose data types Build a per-user leaderboard through the composition of existing types 71
Client1 Scores Local Top-K Fold Global Top-K Fold 73 Client1 Scores Local Top-K Fold Global Top-K Fold Client3 Scores Local Top-K Fold Global Top-K Fold Client2 Scores Local Top-K Fold Global Top-K Fold
Client1 Scores Local Top-K Fold Global Top-K Fold 74 Client1 Scores Local Top-K Fold Global Top-K Fold Client3 Scores Local Top-K Fold Global Top-K Fold Client2 Scores Local Top-K Fold Global Top-K Fold
Client1 Scores Local Top-K Fold Global Top-K Fold 75 Client1 Scores Local Top-K Fold Global Top-K Fold Client3 Scores Local Top-K Fold Global Top-K Fold Client2 Scores Local Top-K Fold Global Top-K Fold
Client1 Scores Local Top-K Fold Global Top-K Fold 76 Client1 Scores Local Top-K Fold Global Top-K Fold Client3 Scores Local Top-K Fold Global Top-K Fold Client2 Scores Local Top-K Fold Global Top-K Fold
Client1 Scores Local Top-K Fold Global Top-K Fold 77 Client1 Scores Local Top-K Fold Global Top-K Fold Client3 Scores Local Top-K Fold Global Top-K Fold Client2 Scores Local Top-K Fold Global Top-K Fold
Client1 Scores Local Top-K Fold Global Top-K Fold 78 Client1 Scores Local Top-K Fold Global Top-K Fold Client3 Scores Local Top-K Fold Global Top-K Fold Client2 Scores Local Top-K Fold Global Top-K Fold
Client1 Scores Local Top-K Fold Global Top-K Lasp Operation Input User-Maintained CRDT Output Lasp-Maintained CRDT Fold Client3 Scores Local Top-K Fold Global Top-K Fold Client2 Scores Local Top-K Fold Global Top-K Fold 79 Client1 Scores Local Top-K Fold Global Top-K Fold Client3 Scores Local Top-K Fold Global Top-K Fold Client2 Scores Local Top-K Fold Global Top-K Fold
Per-User Leaderboard • Dynamically scoped variables Variable which take different values depending on where it is executing • Dynamically scoped fold operation Perform a distributed “reduce” operation that combines the state of a dynamically scoped variables across 80
81 %% Create a global leaderboard. G = declare({top_k, [10]}). %% Create a local leaderboard. L = declare_dynamic({top_k, [10]}). %% Create a set of scores. S = declare_dynamic(set). %% Compute local top-k list. fold(S, fun max_by_name/2, L). %% Compute global top-k list. fold_dynamic(L, fun max_by_name/2, G).
82 %% Create a global leaderboard. G = declare({top_k, [10]}). %% Create a local leaderboard. L = declare_dynamic({top_k, [10]}). %% Create a set of scores. S = declare_dynamic(set). %% Compute local top-k list. fold(S, fun max_by_name/2, L). %% Compute global top-k list. fold_dynamic(L, fun max_by_name/2, G).
83 %% Create a global leaderboard. G = declare({top_k, [10]}). %% Create a local leaderboard. L = declare_dynamic({top_k, [10]}). %% Create a set of scores. S = declare_dynamic(set). %% Compute local top-k list. fold(S, fun max_by_name/2, L). %% Compute global top-k list. fold_dynamic(L, fun max_by_name/2, G).
84 %% Create a global leaderboard. G = declare({top_k, [10]}). %% Create a local leaderboard. L = declare_dynamic({top_k, [10]}). %% Create a set of scores. S = declare_dynamic(set). %% Compute local top-k list. fold(S, fun max_by_name/2, L). %% Compute global top-k list. fold_dynamic(L, fun max_by_name/2, G).
85 %% Create a global leaderboard. G = declare({top_k, [10]}). %% Create a local leaderboard. L = declare_dynamic({top_k, [10]}). %% Create a set of scores. S = declare_dynamic(set). %% Compute local top-k list. fold(S, fun max_by_name/2, L). %% Compute global top-k list. fold_dynamic(L, fun max_by_name/2, G).
86 %% Create a global leaderboard. G = declare({top_k, [10]}). %% Create a local leaderboard. L = declare_dynamic({top_k, [10]}). %% Create a set of scores. S = declare_dynamic(set). %% Compute local top-k list. fold(S, fun max_by_name/2, L). %% Compute global top-k list. fold_dynamic(L, fun max_by_name/2, G).
Advertisement Counter • Mobile game platform selling advertisement space Advertisements are paid according to a minimum number of impressions • Clients will go offline Clients have limited connectivity and the system still needs to make progress while clients are offline 89
Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 90
Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Rovio Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 1 Client 92 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
Ads ovio Ad ounter 1 ovio Ad ounter 2 iot Ad ounter 1 iot Ad ounter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Rovio Ad Counter 1 Ro C Rovio Ad Counter 1 Ro C Rovio Ad Counter 1 Ro C Rovio Ad Counter 1 Ro C Client Side, Sing 93 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
Ads Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product move Read Union Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 94 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
Ads Contracts Ads Contracts Ads With Contracts Filter Product Read Union Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 95 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 96 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 98 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
Advertisement Counter • Completely monotonic Disabling advertisements and contracts are all modeled through monotonic state growth • Arbitrary distribution Use of convergent data structures allows computational graph to be arbitrarily distributed 99
Advertisement Counter • Completely monotonic Disabling advertisements and contracts are all modeled through monotonic state growth • Arbitrary distribution Use of convergent data structures allows computational graph to be arbitrarily distributed • Divergence Divergence is a factor of synchronization period 99
Client3 Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Riot Ad Counter 1 Client1 Client2 Rovio Ad Counter 1 Rovio Ad Counter 2 Rovio Ad Counter 1 Rovio Ad Counter 2 Rovio Ad Counter 1 Riot Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 1 Ads With Contracts Ads With Contracts Ads With Contracts Server Ads With Contracts Server Computation! 106
Client3 Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Riot Ad Counter 1 Client1 Client2 Rovio Ad Counter 1 Rovio Ad Counter 2 Rovio Ad Counter 1 Rovio Ad Counter 2 Rovio Ad Counter 1 Riot Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 1 Ads With Contracts Ads With Contracts Ads With Contracts Server Ads With Contracts Server Computation! 107
Client3 Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Riot Ad Counter 1 Client1 Client2 Rovio Ad Counter 1 Rovio Ad Counter 2 Rovio Ad Counter 1 Rovio Ad Counter 2 Rovio Ad Counter 1 Riot Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 1 Ads With Contracts Ads With Contracts Ads With Contracts Server Ads With Contracts Server Computation! 108
Advertisement Counter • “Servers” as peers to “clients” Servers are peers to clients that perform additional computation • Any node can disable an advertisement under this model given enough information 109
Advertisement Counter • “Servers” as peers to “clients” Servers are peers to clients that perform additional computation • Any node can disable an advertisement under this model given enough information • “Servers” as trusted nodes Serve as a location for performing “exactly once” side- effects 109
Advertisement Counter • “Servers” as peers to “clients” Servers are peers to clients that perform additional computation • Any node can disable an advertisement under this model given enough information • “Servers” as trusted nodes Serve as a location for performing “exactly once” side- effects • Billing customers must be done at a central point by a trusted node in the system 109
Key Points • Synchronization is expensive Locking and other synchronization mechanisms limit scalability to the critical section • Synchronization is sometimes not possible Mobile and “Internet of Things” applications make synchronization for replicated state impractical 112
Key Points • Synchronization is expensive Locking and other synchronization mechanisms limit scalability to the critical section • Synchronization is sometimes not possible Mobile and “Internet of Things” applications make synchronization for replicated state impractical • Apply synchronization only where required Global invariants, atomic visibility, etc. 112
Publications • “Lasp: A Language for Distributed, Coordination-Free Programming” ACM SIGPLAN PPDP 2015
• “Selective Hearing: An Approach to Distributed, Eventually Consistent Edge Computation” IEEE W-PSDS 2015
• “The Implementation and Use of a Generic Dataflow Behaviour in Erlang” ACM SIGPLAN Erlang Workshop ’15 • “Lasp: A Language for Distributed, Eventually Consistent Computations with CRDTs" PaPoC 2015 • “Declarative, Sliding Window Aggregations for Computations at the Edge" IEEE EdgeCom 2016 114
SyncFree is a European research project taking place for 3 years, staring October 2013, and is funded by the European Union, grant agreement n° 609551. 115