Tweet
Tweet

Slide 1

Slide 1 text

Distributed Eventually Consistent Computations Christopher Meiklejohn // @cmeik Strange Loop 2015, September 24 - 26th, 2015 1

Slide 2

Slide 2 text

RA RB

Slide 3

Slide 3 text

RA RB 1 set(1)

Slide 4

Slide 4 text

RA RB 1 set(1) 3 2 set(2) set(3)

Slide 5

Slide 5 text

RA RB 1 set(1) 3 2 set(2) set(3) ? ?

Slide 6

Slide 6 text

Synchronization • To enforce an order
 Makes programming easier 6

Slide 7

Slide 7 text

Synchronization • To enforce an order
 Makes programming easier • Eliminate accidental nondeterminism
 Prevent race conditions 6

Slide 8

Slide 8 text

Synchronization • To enforce an order
 Makes programming easier • Eliminate accidental nondeterminism
 Prevent race conditions • Techniques
 Locks, mutexes, semaphores, monitors, etc. 6

Slide 9

Slide 9 text

Difficult Cases • “Internet of Things”
 Low power, limited memory and connectivity 7

Slide 10

Slide 10 text

Difficult Cases • “Internet of Things”
 Low power, limited memory and connectivity • Mobile Gaming
 Offline operation with replicated, shared state 7

Slide 11

Slide 11 text

Zero Synchronization • We want to have our cake and eat it too. 8

Slide 12

Slide 12 text

Zero Synchronization • We want to have our cake and eat it too. • “Zero synchronization” as a starting point
 Add synchronization only where it is necessary 8

Slide 13

Slide 13 text

Zero Synchronization • We want to have our cake and eat it too. • “Zero synchronization” as a starting point
 Add synchronization only where it is necessary • Explore the limits
 Can we design a model of computation where it’s easy to write coordination-free computations that are free from concurrency anomalies? 8

Slide 14

Slide 14 text

9 Strong Weak Strong (Data Center) Weak (Open Internet) Sharing (Application Property) Coupling (Infrastructure Property)

Slide 15

Slide 15 text

9 Strong Weak Strong (Data Center) Weak (Open Internet) Sharing (Application Property) Coupling (Infrastructure Property) MapReduce SETI@Home

Slide 16

Slide 16 text

9 Strong Weak Strong (Data Center) Weak (Open Internet) Sharing (Application Property) Coupling (Infrastructure Property) Twitter MapReduce SETI@Home

Slide 17

Slide 17 text

9 Strong Weak Strong (Data Center) Weak (Open Internet) Sharing (Application Property) Coupling (Infrastructure Property) Twitter MapReduce SETI@Home Synchronization-Free Computations

Slide 18

Slide 18 text

The Fundamentals of Distributed Computation 10

Slide 19

Slide 19 text

Distributed Computation • Distributed to concurrent programming
 Consistency and now partial failure 11

Slide 20

Slide 20 text

Distributed Computation • Distributed to concurrent programming
 Consistency and now partial failure • Enforcing the “single system” illusion 11

Slide 21

Slide 21 text

Distributed Computation • Distributed to concurrent programming
 Consistency and now partial failure • Enforcing the “single system” illusion • Consistency model as a contract 11

Slide 22

Slide 22 text

Distributed Computation • Distributed to concurrent programming
 Consistency and now partial failure • Enforcing the “single system” illusion • Consistency model as a contract • Enforced through synchronization 11

Slide 23

Slide 23 text

Distributed Computation • Distributed to concurrent programming
 Consistency and now partial failure • Enforcing the “single system” illusion • Consistency model as a contract • Enforced through synchronization • Consistency model
 We can think of a consistency model as analogous to a programming paradigm 11

Slide 24

Slide 24 text

Why Is Synchronization Undesirable? 12

Slide 25

Slide 25 text

The Avatars of Time • The problem of time
 Handling physical time in applications is difficult 13

Slide 26

Slide 26 text

The Avatars of Time • The problem of time
 Handling physical time in applications is difficult • Time has three major avatars in computing 13

Slide 27

Slide 27 text

The Avatars of Time • The problem of time
 Handling physical time in applications is difficult • Time has three major avatars in computing • Mutable state in sequential systems 13

Slide 28

Slide 28 text

The Avatars of Time • The problem of time
 Handling physical time in applications is difficult • Time has three major avatars in computing • Mutable state in sequential systems • Nondeterminism in concurrent systems 13

Slide 29

Slide 29 text

The Avatars of Time • The problem of time
 Handling physical time in applications is difficult • Time has three major avatars in computing • Mutable state in sequential systems • Nondeterminism in concurrent systems ★ Network latency in distributed systems 13

Slide 30

Slide 30 text

The Avatars of Time • The problem of time
 Handling physical time in applications is difficult • Time has three major avatars in computing • Mutable state in sequential systems • Nondeterminism in concurrent systems ★ Network latency in distributed systems • Unavoidable
 Time is how users interact with programs 13

Slide 31

Slide 31 text

Parable of the Car • “Car driving on a highway” 14

Slide 32

Slide 32 text

Parable of the Car • “Car driving on a highway” • Friction needed for the car to grip the road 14

Slide 33

Slide 33 text

Parable of the Car • “Car driving on a highway” • Friction needed for the car to grip the road • Motors want as little friction as possible 14

Slide 34

Slide 34 text

Parable of the Car • “Car driving on a highway” • Friction needed for the car to grip the road • Motors want as little friction as possible • Time is like friction
 We can not completely eliminate friction, but aim to reduce it as much as possible 14

Slide 35

Slide 35 text

Physical Time (real world) Parable of the Car 15 • Time only at the interface
 The interface is a small part of the system, and this is where we introduction friction

Slide 36

Slide 36 text

Physical Time (real world) Parable of the Car 15 • Time only at the interface
 The interface is a small part of the system, and this is where we introduction friction • Physical time-free execution
 Internally, avoid synchronization as much as possible and virtualize notion of time

Slide 37

Slide 37 text

No Time In The Box? 16

Slide 38

Slide 38 text

No Time In The Box? 16 You program the box.

Slide 39

Slide 39 text

Weak Synchronization • Can we achieve anything without synchronization?
 Not really. 17

Slide 40

Slide 40 text

Weak Synchronization • Can we achieve anything without synchronization?
 Not really. • Strong Eventual Consistency (SEC)
 “Replicas that deliver the same updates have equivalent state” 17

Slide 41

Slide 41 text

Weak Synchronization • Can we achieve anything without synchronization?
 Not really. • Strong Eventual Consistency (SEC)
 “Replicas that deliver the same updates have equivalent state” • Primary requirement
 Eventual replica-to-replica communication 17

Slide 42

Slide 42 text

Weak Synchronization • Can we achieve anything without synchronization?
 Not really. • Strong Eventual Consistency (SEC)
 “Replicas that deliver the same updates have equivalent state” • Primary requirement
 Eventual replica-to-replica communication • Order insensitive! (Commutativity) 17

Slide 43

Slide 43 text

Weak Synchronization • Can we achieve anything without synchronization?
 Not really. • Strong Eventual Consistency (SEC)
 “Replicas that deliver the same updates have equivalent state” • Primary requirement
 Eventual replica-to-replica communication • Order insensitive! (Commutativity) • Duplicate insensitive! (Idempotent) 17

Slide 44

Slide 44 text

RA RB

Slide 45

Slide 45 text

RA RB 1 set(1)

Slide 46

Slide 46 text

RA RB 1 set(1) 3 2 set(2) set(3)

Slide 47

Slide 47 text

RA RB 1 3 2 3 3 set(1) set(2) set(3) max(2,3) max(2,3)

Slide 48

Slide 48 text

How can we succeed with Strong Eventual Consistency? 22

Slide 49

Slide 49 text

Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic operations monotonically) 23

Slide 50

Slide 50 text

Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, referential transparency over composition) 23

Slide 51

Slide 51 text

Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, referential transparency over composition) 3. Distributed, and fault-tolerant runtime
 (ex. replication) 23

Slide 52

Slide 52 text

Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, referential transparency over composition) 3. Distributed, and fault-tolerant runtime
 (ex. replication) 24

Slide 53

Slide 53 text

Convergent Objects
 Conflict-Free 
 Replicated Data Types 25 SSS 2011

Slide 54

Slide 54 text

Conflict-Free 
 Replicated Data Types • Many types exist with different properties
 Sets, counters, registers, flags, maps, graphs 26

Slide 55

Slide 55 text

Conflict-Free 
 Replicated Data Types • Many types exist with different properties
 Sets, counters, registers, flags, maps, graphs • Strong Eventual Consistency
 Instances satisfy SEC property per-object 26

Slide 56

Slide 56 text

RA RB RC

Slide 57

Slide 57 text

RA RB RC {1} (1, {a}, {}) add(1)

Slide 58

Slide 58 text

RA RB RC {1} (1, {a}, {}) add(1) {1} (1, {b}, {}) add(1)

Slide 59

Slide 59 text

RA RB RC {1} (1, {a}, {}) add(1) {1} (1, {b}, {}) add(1) {} (1, {b}, {b}) remove(1)

Slide 60

Slide 60 text

RA RB RC {1} (1, {a}, {}) add(1) {1} (1, {b}, {}) add(1) {} (1, {b}, {b}) remove(1) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b})

Slide 61

Slide 61 text

Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, referential transparency over composition) 3. Distributed, and fault-tolerant runtime
 (ex. replication) 32

Slide 62

Slide 62 text

Convergent Programs Lattice Processing 33 PPDP 2015

Slide 63

Slide 63 text

Lattice Processing (Lasp) • Distributed, deterministic dataflow
 Distributed, deterministic dataflow programming model for “eventually consistent” computations 34

Slide 64

Slide 64 text

Lattice Processing (Lasp) • Distributed, deterministic dataflow
 Distributed, deterministic dataflow programming model for “eventually consistent” computations • Convergent data structures
 Primary data abstraction is the CRDT 34

Slide 65

Slide 65 text

Lattice Processing (Lasp) • Distributed, deterministic dataflow
 Distributed, deterministic dataflow programming model for “eventually consistent” computations • Convergent data structures
 Primary data abstraction is the CRDT • Enables composition
 Provides functional composition of CRDTs that preserves the SEC property 34

Slide 66

Slide 66 text

35 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).

Slide 67

Slide 67 text

36 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).

Slide 68

Slide 68 text

37 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).

Slide 69

Slide 69 text

38 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).

Slide 70

Slide 70 text

39 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).

Slide 71

Slide 71 text

Lattice Processing (Lasp) • Functional and set-theoretic operations on sets
 Product, intersection, union, filter, map, fold 40

Slide 72

Slide 72 text

Lattice Processing (Lasp) • Functional and set-theoretic operations on sets
 Product, intersection, union, filter, map, fold • Metadata computation
 Performs transformation on the internal metadata of CRDTs allowing creation of “composed” CRDTs 40

Slide 73

Slide 73 text

Lasp Processes • Replicas as monotonic streams
 Each replica of a CRDT produces a monotonic stream of states 41

Slide 74

Slide 74 text

Lasp Processes • Replicas as monotonic streams
 Each replica of a CRDT produces a monotonic stream of states • Monotonic processes
 Read from one or more input replica streams and produce a single output replica stream 41

Slide 75

Slide 75 text

Lasp Processes • Replicas as monotonic streams
 Each replica of a CRDT produces a monotonic stream of states • Monotonic processes
 Read from one or more input replica streams and produce a single output replica stream • Inflationary reads
 Read operation ensures that we only read inflationary updates to replicas 41

Slide 76

Slide 76 text

Lattice Processing Monotonic Streams 42

Slide 77

Slide 77 text

RA {} C1 {} C2 {} 43

Slide 78

Slide 78 text

RA {} (1, {a}, {}) C1 (1, {a}, {}) {} C2 {} 44

Slide 79

Slide 79 text

RA {} (1, {a}, {}) (1, {a, b}, {}) C1 (1, {a}, {}) {} C2 (1, {b}, {}) {} 45

Slide 80

Slide 80 text

RA {} (1, {a}, {}) (1, {a, b}, {}) (1, {a, b}, {a}) C1 (1, {a}, {}) {} (1, {a}, {a}) C2 (1, {b}, {}) {} 46

Slide 81

Slide 81 text

RA {} (1, {a}, {}) (1, {a, b}, {}) (1, {a, b}, {a}) C1 (1, {a}, {}) {} (1, {a}, {a}) C2 (1, {b}, {}) {} (1, {a, b}, {a}) (1, {a, b}, {a}) (1, {a, b}, {a}) 47

Slide 82

Slide 82 text

RA {} (1, {a}, {}) (1, {a, b}, {}) (1, {a, b}, {a}) C1 (1, {a}, {}) {} (1, {a}, {a}) C2 (1, {b}, {}) {} (1, {a, b}, {a}) (1, {a, b}, {a}) (1, {a, b}, {a}) 48

Slide 83

Slide 83 text

Lattice Processing Monotonic Processes 49

Slide 84

Slide 84 text

RA {} P1 F(RA) {} 50

Slide 85

Slide 85 text

RA {} P1 F(RA) {} strict_read({}) 51

Slide 86

Slide 86 text

RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) 52

Slide 87

Slide 87 text

RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1, {a}, {})) 53

Slide 88

Slide 88 text

RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1, {a}, {})) (2, {a}, {}) 54

Slide 89

Slide 89 text

RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1, {a}, {})) (2, {a}, {}) strict_read((1, {a}, {}) (1, {a}, {}) 55

Slide 90

Slide 90 text

RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1, {a}, {})) (2, {a}, {}) strict_read((1, {a}, {}) (1, {a}, {}) (1, {a}, {a}) F((1, {a}, {a})) (2, {a}, {a}) strict_read((1, {a}, {a}) (1, {a}, {a}) 56

Slide 91

Slide 91 text

RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) (1, {a}, {}) (1, {a}, {a}) F((1, {a}, {a})) (2, {a}, {a}) strict_read((1, {a}, {a}) (1, {a}, {a}) 57

Slide 92

Slide 92 text

Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, referential transparency over composition) 3. Distributed, and fault-tolerant runtime
 (ex. replication) 58

Slide 93

Slide 93 text

Distributed Runtime Selective Hearing 59 W-PSDS 2015

Slide 94

Slide 94 text

Selective Hearing • Epidemic broadcast based runtime system
 Provide a runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efficient execution 60

Slide 95

Slide 95 text

Selective Hearing • Epidemic broadcast based runtime system
 Provide a runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efficient execution • Well-matched to Lattice Processing (Lasp) 60

Slide 96

Slide 96 text

Selective Hearing • Epidemic broadcast based runtime system
 Provide a runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efficient execution • Well-matched to Lattice Processing (Lasp) • Epidemic broadcast mechanisms provide weak ordering but are resilient and efficient 60

Slide 97

Slide 97 text

Selective Hearing • Epidemic broadcast based runtime system
 Provide a runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efficient execution • Well-matched to Lattice Processing (Lasp) • Epidemic broadcast mechanisms provide weak ordering but are resilient and efficient • Lasp’s programming model is tolerant to message re-ordering, disconnections, and node failures 60

Slide 98

Slide 98 text

Selective Hearing • Epidemic broadcast based runtime system
 Provide a runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efficient execution • Well-matched to Lattice Processing (Lasp) • Epidemic broadcast mechanisms provide weak ordering but are resilient and efficient • Lasp’s programming model is tolerant to message re-ordering, disconnections, and node failures • “Selective Receive”
 Nodes selectively receive and process messages based on interest. 60

Slide 99

Slide 99 text

Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, referential transparency over composition) 3. Distributed, and fault-tolerant runtime
 (ex. replication) 61

Slide 100

Slide 100 text

What can we build? Advertisement Counter 62

Slide 101

Slide 101 text

Advertisement Counter • Mobile game platform selling advertisement space
 Advertisements are paid according to a minimum number of impressions 63

Slide 102

Slide 102 text

Advertisement Counter • Mobile game platform selling advertisement space
 Advertisements are paid according to a minimum number of impressions • Clients will go offline
 Clients have limited connectivity and the system still needs to make progress while clients are offline 63

Slide 103

Slide 103 text

Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 64

Slide 104

Slide 104 text

Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Riot Ads Rovio Ads Product Read 50,000 Remove Increment Union 65 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client

Slide 105

Slide 105 text

Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Rovio Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 1 Client 66 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client

Slide 106

Slide 106 text

Ads ovio Ad ounter 1 ovio Ad ounter 2 iot Ad ounter 1 iot Ad ounter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Rovio Ad Counter 1 Ro C Rovio Ad Counter 1 Ro C Rovio Ad Counter 1 Ro C Rovio Ad Counter 1 Ro C Client Side, Sing 67 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client

Slide 107

Slide 107 text

Ads Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product move Read Union Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 68 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client

Slide 108

Slide 108 text

Ads Contracts Ads Contracts Ads With Contracts Filter Product Read Union Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 69 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client

Slide 109

Slide 109 text

Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 70 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client

Slide 110

Slide 110 text

Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Riot Ads Rovio Ads Fil Product Read 50,000 Remove Increment Union 71 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client

Slide 111

Slide 111 text

Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 72 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client

Slide 112

Slide 112 text

Advertisement Counter • Completely monotonic
 Disabling advertisements and contracts are all modeled through monotonic state growth 73

Slide 113

Slide 113 text

Advertisement Counter • Completely monotonic
 Disabling advertisements and contracts are all modeled through monotonic state growth • Arbitrary distribution
 Use of convergent data structures allows computational graph to be arbitrarily distributed 73

Slide 114

Slide 114 text

Advertisement Counter • Completely monotonic
 Disabling advertisements and contracts are all modeled through monotonic state growth • Arbitrary distribution
 Use of convergent data structures allows computational graph to be arbitrarily distributed • Divergence
 Divergence is a factor of synchronization period 73

Slide 115

Slide 115 text

We’ve build up from zero synchronization 74

Slide 116

Slide 116 text

We’ve build up from zero synchronization 74 Instead of working to remove synchronization

Slide 117

Slide 117 text

What’s next? 75

Slide 118

Slide 118 text

What’s Next? • Reduce metadata
 Metadata can grow very large 76

Slide 119

Slide 119 text

What’s Next? • Reduce metadata
 Metadata can grow very large • Provide a more general ‘fold’ operation
 Currently very restrictive, how general can it be? 76

Slide 120

Slide 120 text

What’s Next? • Reduce metadata
 Metadata can grow very large • Provide a more general ‘fold’ operation
 Currently very restrictive, how general can it be? • Auto-placement
 What can the dataflow graph tell us about placement? 76

Slide 121

Slide 121 text

What have we learned? 77

Slide 122

Slide 122 text

Programming SEC 1. Eliminate accidental nondeterminism
 Convergent Objects: Conflict-Free Replicated Data Types 78

Slide 123

Slide 123 text

Programming SEC 1. Eliminate accidental nondeterminism
 Convergent Objects: Conflict-Free Replicated Data Types 2. Retain the properties of functional programming
 Convergent Programs: Lattice Processing 78

Slide 124

Slide 124 text

Programming SEC 1. Eliminate accidental nondeterminism
 Convergent Objects: Conflict-Free Replicated Data Types 2. Retain the properties of functional programming
 Convergent Programs: Lattice Processing 3. Distributed, and fault-tolerant runtime
 Distributed Runtime: Selective Hearing 78

Slide 125

Slide 125 text

Key Points • Synchronization is expensive
 Locking and other synchronization mechanisms limit scalability to the critical section 79

Slide 126

Slide 126 text

Key Points • Synchronization is expensive
 Locking and other synchronization mechanisms limit scalability to the critical section • Synchronization is sometimes not possible
 Mobile and “Internet of Things” applications make synchronization for replicated state impractical 79

Slide 127

Slide 127 text

Key Points • Synchronization is expensive
 Locking and other synchronization mechanisms limit scalability to the critical section • Synchronization is sometimes not possible
 Mobile and “Internet of Things” applications make synchronization for replicated state impractical • Apply synchronization only where required
 Global invariants, atomic visibility, etc. 79

Slide 128

Slide 128 text

How do I learn more? 80

Slide 129

Slide 129 text

Publications • “Lasp: A Language for Distributed, Coordination-Free Programming” 
 ACM SIGPLAN PPDP 2015 81

Slide 130

Slide 130 text

Publications • “Lasp: A Language for Distributed, Coordination-Free Programming” 
 ACM SIGPLAN PPDP 2015 • “Selective Hearing: An Approach to Distributed, Eventually Consistent Edge Computation”
 IEEE W-PSDS 2015 81

Slide 131

Slide 131 text

Publications • “Lasp: A Language for Distributed, Coordination-Free Programming” 
 ACM SIGPLAN PPDP 2015 • “Selective Hearing: An Approach to Distributed, Eventually Consistent Edge Computation”
 IEEE W-PSDS 2015 • “The Implementation and Use of a Generic Dataflow Behaviour in Erlang”
 ACM SIGPLAN Erlang Workshop ’15 81

Slide 132

Slide 132 text

Publications • “Lasp: A Language for Distributed, Coordination-Free Programming” 
 ACM SIGPLAN PPDP 2015 • “Selective Hearing: An Approach to Distributed, Eventually Consistent Edge Computation”
 IEEE W-PSDS 2015 • “The Implementation and Use of a Generic Dataflow Behaviour in Erlang”
 ACM SIGPLAN Erlang Workshop ’15 • “Lasp: A Language for Distributed, Eventually Consistent Computations with CRDTs"
 PaPoC 2015 81

Slide 133

Slide 133 text

82 Lasp Release 0.0.1 http://lasp-lang.org

Slide 134

Slide 134 text

SyncFree is a European research project taking place for 3 years, staring October 2013, and is funded by the European Union, grant agreement n° 609551. 83

Slide 135

Slide 135 text

Thanks! 84 Christopher Meiklejohn @cmeik