Tweet
Tweet

Slide 1

Slide 1 text

Declarative, Convergent, Edge Computation Christopher S. Meiklejohn Université catholique de Louvain, Belgium Instituto Superior Técnico, Portugal LIGHT ONE

Slide 2

Slide 2 text

RA RB

Slide 3

Slide 3 text

RA RB 1 set(1)

Slide 4

Slide 4 text

RA RB 1 set(1) 3 2 set(2) set(3) Concurrent operations.

Slide 5

Slide 5 text

RA RB 1 set(1) 3 2 set(2) set(3) ? ? Nondeterministic outcome.

Slide 6

Slide 6 text

Synchronization • To enforce an order
 Makes programming easier 6

Slide 7

Slide 7 text

Synchronization • To enforce an order
 Makes programming easier • Eliminate accidental nondeterminism
 Prevent race conditions 6

Slide 8

Slide 8 text

Synchronization • To enforce an order
 Makes programming easier • Eliminate accidental nondeterminism
 Prevent race conditions • Techniques
 Locks, mutexes, semaphores, monitors, Paxos, Raft, etc. 6

Slide 9

Slide 9 text

Difficult Cases • “Internet of Things”, 
 Low power, limited memory and connectivity 7

Slide 10

Slide 10 text

Difficult Cases • “Internet of Things”, 
 Low power, limited memory and connectivity • Mobile Gaming
 Offline operation with replicated, shared state 7

Slide 11

Slide 11 text

8

Slide 12

Slide 12 text

The Fundamentals of Distributed Computation 9

Slide 13

Slide 13 text

Distributed Computation • Concurrent to distributed programming
 Consistency and now partial failure 10

Slide 14

Slide 14 text

Distributed Computation • Concurrent to distributed programming
 Consistency and now partial failure • Enforcing the “single system” illusion
 Traditional approach for gaining performance, scalability, fault- tolerance while still “easy to program” 10

Slide 15

Slide 15 text

Distributed Computation • Concurrent to distributed programming
 Consistency and now partial failure • Enforcing the “single system” illusion
 Traditional approach for gaining performance, scalability, fault- tolerance while still “easy to program” • Consistency models
 Contract between the system and the programmer 10

Slide 16

Slide 16 text

Distributed Computation • Concurrent to distributed programming
 Consistency and now partial failure • Enforcing the “single system” illusion
 Traditional approach for gaining performance, scalability, fault- tolerance while still “easy to program” • Consistency models
 Contract between the system and the programmer • Correctness criteria
 For both distributed and concurrent programs 10

Slide 17

Slide 17 text

Distributed Computation • Concurrent to distributed programming
 Consistency and now partial failure • Enforcing the “single system” illusion
 Traditional approach for gaining performance, scalability, fault- tolerance while still “easy to program” • Consistency models
 Contract between the system and the programmer • Correctness criteria
 For both distributed and concurrent programs • Synchronization as “knob”
 Models live along a spectrum requiring more or less synchronization 10

Slide 18

Slide 18 text

Databases Consistency Models 11

Slide 19

Slide 19 text

Databases Strong Consistency 12

Slide 20

Slide 20 text

R1 C1 C2 13

Slide 21

Slide 21 text

R1 C1 C2 14

Slide 22

Slide 22 text

R1 C1 C2 Read 15

Slide 23

Slide 23 text

R1 C1 C2 Read 16

Slide 24

Slide 24 text

R1 C1 C2 17

Slide 25

Slide 25 text

R1 C1 C2 CAS 18

Slide 26

Slide 26 text

R1 C1 C2 19

Slide 27

Slide 27 text

R1 C1 C2 CAS 20 Operation refused because value changed between read/write.

Slide 28

Slide 28 text

I won’t diagram the Paxos protocol 21

Slide 29

Slide 29 text

R2 C1 C2 Value 2 Value 1 Value 2 R1 R3 Paxos 22 Coordination is extremely expensive.

Slide 30

Slide 30 text

Databases Eventual Consistency 23

Slide 31

Slide 31 text

R1 R2 R3 C1 C2 24

Slide 32

Slide 32 text

R1 R2 R3 C1 C2 25

Slide 33

Slide 33 text

R1 R2 R3 C1 C2 Read 26

Slide 34

Slide 34 text

R1 R2 R3 C1 C2 Read 27

Slide 35

Slide 35 text

R1 R2 R3 C1 C2 Write 28

Slide 36

Slide 36 text

R1 R2 R3 C1 C2 Write 29

Slide 37

Slide 37 text

R1 R2 R3 C1 C2 Write C1 30 Do I store both values or do I choose a value arbitrarily?

Slide 38

Slide 38 text

R1 R2 R3 C1 C2 Read 31 If I store both, the programmer must expect multiple values…

Slide 39

Slide 39 text

R1 R2 R3 C1 C2 Write 32 …and be able to semantically resolve them in application code.

Slide 40

Slide 40 text

Strong Eventual Consistency (SEC) • Convergent
 Same updates realize the same state 33

Slide 41

Slide 41 text

Strong Eventual Consistency (SEC) • Convergent
 Same updates realize the same state • Primary requirement
 “Eventual” replica-to-replica communication 33

Slide 42

Slide 42 text

Strong Eventual Consistency (SEC) • Convergent
 Same updates realize the same state • Primary requirement
 “Eventual” replica-to-replica communication • Order insensitive!
 Operations are commutative 33

Slide 43

Slide 43 text

Strong Eventual Consistency (SEC) • Convergent
 Same updates realize the same state • Primary requirement
 “Eventual” replica-to-replica communication • Order insensitive!
 Operations are commutative • Duplicate insensitive!
 Operations are idempotent 33

Slide 44

Slide 44 text

RA RB

Slide 45

Slide 45 text

RA RB 1 set(1)

Slide 46

Slide 46 text

RA RB 1 set(1) 3 2 set(2) set(3)

Slide 47

Slide 47 text

RA RB 1 3 2 3 3 set(1) set(2) set(3) max(2,3) max(2,3) Automatic resolution of conflicting updates.

Slide 48

Slide 48 text

How can we succeed with Strong Eventual Consistency? 38

Slide 49

Slide 49 text

Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic operations monotonically) 39

Slide 50

Slide 50 text

Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, “correct-by-construction” applications) 39

Slide 51

Slide 51 text

Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, “correct-by-construction” applications) 3. Distributed, and fault-tolerant runtime
 (ex. replication, membership, dissemination) 39

Slide 52

Slide 52 text

Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, “correct-by-construction” applications) 3. Distributed, and fault-tolerant runtime
 (ex. replication, membership, dissemination) 40

Slide 53

Slide 53 text

Convergent Objects
 Conflict-Free 
 Replicated Data Types 41 SSS 2011

Slide 54

Slide 54 text

Conflict-Free 
 Replicated Data Types • Many types exist with different properties
 Sets, counters, registers, flags, maps 42

Slide 55

Slide 55 text

Conflict-Free 
 Replicated Data Types • Many types exist with different properties
 Sets, counters, registers, flags, maps • Strong Eventual Consistency
 Instances satisfy SEC property per-object 42

Slide 56

Slide 56 text

Conflict-Free 
 Replicated Data Types • Many types exist with different properties
 Sets, counters, registers, flags, maps • Strong Eventual Consistency
 Instances satisfy SEC property per-object • Bounded join-semilattices
 Formalized using bounded join-semilattices where the merge operation is the join 42

Slide 57

Slide 57 text

Convergent Objects Observed-Remove Set 43

Slide 58

Slide 58 text

RA RB RC

Slide 59

Slide 59 text

RA RB RC {1} (1, {a}, {}) add(1)

Slide 60

Slide 60 text

RA RB RC {1} (1, {a}, {}) add(1) {1} (1, {b}, {}) add(1)

Slide 61

Slide 61 text

RA RB RC {1} (1, {a}, {}) add(1) {1} (1, {b}, {}) add(1) {} (1, {b}, {b}) remove(1)

Slide 62

Slide 62 text

RA RB RC {1} (1, {a}, {}) add(1) {1} (1, {b}, {}) add(1) {} (1, {b}, {b}) remove(1) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) Convergence reached.

Slide 63

Slide 63 text

Convergent Objects Composition 49

Slide 64

Slide 64 text

RA {1} (1, {a}, {}) {1} (1, {a, b}, {a}) RB {1} (1, {b}, {}) {1} (1, {a, b}, {a}) {} (1, {a}, {a}) add(1) remove(1) add(1) Replicated set of naturals across two nodes.

Slide 65

Slide 65 text

RA {1} (1, {a}, {}) {1} (1, {a, b}, {a}) RB {1} (1, {b}, {}) {1} (1, {a, b}, {a}) {} (1, {a}, {a}) add(1) remove(1) add(1) Map Process \x.2x F(RA) {2} {2} {} F(RB) {2} ? Map Process \x.2x Map Process \x.2x Map Process \x.2x Map Process \x.2x {2} Map \x.2x over a set of naturals.

Slide 66

Slide 66 text

RA {1} (1, {a}, {}) {1} (1, {a, b}, {a}) RB {1} (1, {b}, {}) {1} (1, {a, b}, {a}) {} (1, {a}, {a}) add(1) remove(1) add(1) Map Process \x.2x F(RA) {2} {2} {} F(RB) {2} ? Map Process \x.2x Map Process \x.2x Map Process \x.2x Map Process \x.2x {2} One node…

Slide 67

Slide 67 text

RA {1} (1, {a}, {}) {1} (1, {a, b}, {a}) RB {1} (1, {b}, {}) {1} (1, {a, b}, {a}) {} (1, {a}, {a}) add(1) remove(1) add(1) Map Process \x.2x F(RA) {2} {2} {} F(RB) {2} ? Map Process \x.2x Map Process \x.2x Map Process \x.2x Map Process \x.2x {2} …another node.

Slide 68

Slide 68 text

RA {1} (1, {a}, {}) {1} (1, {a, b}, {a}) RB {1} (1, {b}, {}) {1} (1, {a, b}, {a}) {} (1, {a}, {a}) add(1) remove(1) add(1) Map Process \x.2x F(RA) {2} {2} {} F(RB) {2} ? Map Process \x.2x Map Process \x.2x Map Process \x.2x Map Process \x.2x {2}

Slide 69

Slide 69 text

RA {1} (1, {a}, {}) {1} (1, {a, b}, {a}) RB {1} (1, {b}, {}) {1} (1, {a, b}, {a}) {} (1, {a}, {a}) add(1) remove(1) add(1) Map Process \x.2x F(RA) {2} {2} {} F(RB) {2} ? Map Process \x.2x Map Process \x.2x Map Process \x.2x Map Process \x.2x {2} Nondeterministic outcome.

Slide 70

Slide 70 text

RA {1} (1, {a}, {}) {1} (1, {a, b}, {a}) RB {1} (1, {b}, {}) {1} (1, {a, b}, {a}) {} (1, {a}, {a}) add(1) remove(1) add(1) Map Process \x.2x F(RA) {2} {2} {} F(RB) {2} ? Map Process \x.2x Map Process \x.2x Map Process \x.2x Map Process \x.2x {2} Correct output that’s seen all updates.

Slide 71

Slide 71 text

RA {1} (1, {a}, {}) {1} (1, {a, b}, {a}) RB {1} (1, {b}, {}) {1} (1, {a, b}, {a}) {} (1, {a}, {a}) add(1) remove(1) add(1) Map Process \x.2x F(RA) {2} {2} {} F(RB) {2} ? Map Process \x.2x Map Process \x.2x Map Process \x.2x Map Process \x.2x {2} “Earlier” value that’s been delayed.

Slide 72

Slide 72 text

Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, “correct-by-construction” applications) 3. Distributed, and fault-tolerant runtime
 (ex. replication, membership, dissemination) 58

Slide 73

Slide 73 text

Convergent Programs Lattice Processing 59 PPDP 2015

Slide 74

Slide 74 text

Lattice Processing • Asynchronous dataflow with streams
 Combine and transform streams of inputs into streams of outputs 60

Slide 75

Slide 75 text

Lattice Processing • Asynchronous dataflow with streams
 Combine and transform streams of inputs into streams of outputs • Convergent data structures
 Data abstraction (inputs/outputs) is the CRDT 60

Slide 76

Slide 76 text

Lattice Processing • Asynchronous dataflow with streams
 Combine and transform streams of inputs into streams of outputs • Convergent data structures
 Data abstraction (inputs/outputs) is the CRDT • Confluence
 Provides composition that preserves the SEC property 60

Slide 77

Slide 77 text

Lattice Processing Confluence 61

Slide 78

Slide 78 text

A B Map Process \x.2x 62 Sequential specification.

Slide 79

Slide 79 text

A’ A’’ B’ B’’ Map Process \x.2x Map Process \x.2x A B Map Process \x.2x 63 Replication per node.

Slide 80

Slide 80 text

A’ A’’ B’ B’’ Map Process \x.2x Map Process \x.2x A B Map Process \x.2x 64 Replication per node.

Slide 81

Slide 81 text

A’ A’’ B’ B’’ Map Process \x.2x Map Process \x.2x A B Map Process \x.2x 65 Replication per node.

Slide 82

Slide 82 text

A’ A’’ B’ B’’ Map Process \x.2x Map Process \x.2x A B Map Process \x.2x 66 One possible schedule….

Slide 83

Slide 83 text

A’ A’’ B’ B’’ Map Process \x.2x Map Process \x.2x A B Map Process \x.2x 67 …another possible schedule.

Slide 84

Slide 84 text

A’ A’’ B’ B’’ Map Process \x.2x Map Process \x.2x A B Map Process \x.2x 68

Slide 85

Slide 85 text

A B Map Process \x.2x 69 All schedules equivalent to sequential schedule.

Slide 86

Slide 86 text

A B F 70 Arbitrary application.

Slide 87

Slide 87 text

Lattice Processing Example 71

Slide 88

Slide 88 text

72 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).

Slide 89

Slide 89 text

73 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).

Slide 90

Slide 90 text

74 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).

Slide 91

Slide 91 text

75 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).

Slide 92

Slide 92 text

76 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).

Slide 93

Slide 93 text

Processes • Replicas as monotonic streams
 Each replica of a CRDT produces a monotonic stream of states 77

Slide 94

Slide 94 text

Processes • Replicas as monotonic streams
 Each replica of a CRDT produces a monotonic stream of states • Monotonic processes
 Read from one or more input replica streams and produce a single output replica stream 77

Slide 95

Slide 95 text

Processes • Replicas as monotonic streams
 Each replica of a CRDT produces a monotonic stream of states • Monotonic processes
 Read from one or more input replica streams and produce a single output replica stream • Inflationary reads
 Read operation ensures that we only read inflationary updates to replicas 77

Slide 96

Slide 96 text

Lattice Processing Monotonic Streams 78

Slide 97

Slide 97 text

RA {} C1 {} C2 {} 79

Slide 98

Slide 98 text

RA {} (1, {a}, {}) C1 (1, {a}, {}) {} C2 {} 80

Slide 99

Slide 99 text

RA {} (1, {a}, {}) (1, {a, b}, {}) C1 (1, {a}, {}) {} C2 (1, {b}, {}) {} 81

Slide 100

Slide 100 text

RA {} (1, {a}, {}) (1, {a, b}, {}) (1, {a, b}, {a}) C1 (1, {a}, {}) {} (1, {a}, {a}) C2 (1, {b}, {}) {} 82

Slide 101

Slide 101 text

RA {} (1, {a}, {}) (1, {a, b}, {}) (1, {a, b}, {a}) C1 (1, {a}, {}) {} (1, {a}, {a}) C2 (1, {b}, {}) {} (1, {a, b}, {a}) (1, {a, b}, {a}) (1, {a, b}, {a}) 83

Slide 102

Slide 102 text

RA {} (1, {a}, {}) (1, {a, b}, {}) (1, {a, b}, {a}) C1 (1, {a}, {}) {} (1, {a}, {a}) C2 (1, {b}, {}) {} (1, {a, b}, {a}) (1, {a, b}, {a}) (1, {a, b}, {a}) 84 Clients can operate with partial state…

Slide 103

Slide 103 text

RA {} (1, {a}, {}) (1, {a, b}, {}) (1, {a, b}, {a}) C1 (1, {a}, {}) {} (1, {a}, {a}) C2 (1, {b}, {}) {} (1, {a, b}, {a}) (1, {a, b}, {a}) (1, {a, b}, {a}) 85 … and synchronize with their local replica.

Slide 104

Slide 104 text

Lattice Processing Monotonic Processes 86

Slide 105

Slide 105 text

RA {} P1 F(RA) {} 87

Slide 106

Slide 106 text

RA {} P1 F(RA) {} strict_read({}) 88

Slide 107

Slide 107 text

RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) 89

Slide 108

Slide 108 text

RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1, {a}, {})) 90

Slide 109

Slide 109 text

RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1, {a}, {})) (2, {a}, {}) 91 Every time replica changes…

Slide 110

Slide 110 text

RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1, {a}, {})) (2, {a}, {}) 92 ….the process will compute a new result.

Slide 111

Slide 111 text

RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1, {a}, {})) (2, {a}, {}) strict_read((1, {a}, {}) (1, {a}, {}) 93

Slide 112

Slide 112 text

RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1, {a}, {})) (2, {a}, {}) strict_read((1, {a}, {}) (1, {a}, {}) (1, {a}, {a}) F((1, {a}, {a})) (2, {a}, {a}) strict_read((1, {a}, {a}) (1, {a}, {a}) 94

Slide 113

Slide 113 text

RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) (1, {a}, {}) (1, {a}, {a}) F((1, {a}, {a})) (2, {a}, {a}) strict_read((1, {a}, {a}) (1, {a}, {a}) 95 Omitted interleaving does not sacrifice correctness.

Slide 114

Slide 114 text

Lattice Processing Map Example 96

Slide 115

Slide 115 text

RC F(RC) 97

Slide 116

Slide 116 text

RC F(RC) {2} Map Process \x.2x (2, {b}, {}) (1, {b}, {}) {1} 98 Transform element, map metadata.

Slide 117

Slide 117 text

RC F(RC) {2} Map Process \x.2x (2, {b}, {}) (1, {b}, {}) {1} {} (1, {b}, {b}) {} Map Process \x.2x (2, {b}, {b}) 99

Slide 118

Slide 118 text

RC F(RC) {2} Map Process \x.2x (2, {b}, {}) (1, {b}, {}) {1} {} (1, {b}, {b}) {} Map Process \x.2x (2, {b}, {b}) {1} (1, {a, b}, {b}) {2} Map Process \x.2x (2, {a, b}, {b}) 100

Slide 119

Slide 119 text

RC F(RC) {2} Map Process \x.2x (2, {b}, {}) (1, {b}, {}) {1} {} (1, {b}, {b}) {} Map Process \x.2x (2, {b}, {b}) {1} (1, {a, b}, {b}) {2} Map Process \x.2x (2, {a, b}, {b}) {1} {2} 101

Slide 120

Slide 120 text

Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, “correct-by-construction” applications) 3. Distributed, and fault-tolerant runtime
 (ex. replication, membership, dissemination) 102

Slide 121

Slide 121 text

Example Application Advertisement Counter 103

Slide 122

Slide 122 text

Advertisement Counter • Lower-bound invariant
 Advertisements are paid according to a minimum number of impressions 104

Slide 123

Slide 123 text

Advertisement Counter • Lower-bound invariant
 Advertisements are paid according to a minimum number of impressions • Clients will go offline
 Clients have limited connectivity and the system still needs to make progress while clients are offline 104

Slide 124

Slide 124 text

Advertisement Counter • Lower-bound invariant
 Advertisements are paid according to a minimum number of impressions • Clients will go offline
 Clients have limited connectivity and the system still needs to make progress while clients are offline • No lost updates
 All displayed advertisements should be accounted for, with no lost updates 104

Slide 125

Slide 125 text

Advertisement Counter Losing Updates 105

Slide 126

Slide 126 text

RA RB 106

Slide 127

Slide 127 text

RA RB 1 set(1) 107

Slide 128

Slide 128 text

RA RB 1 set(1) 2 2 set(2) set(2) 108

Slide 129

Slide 129 text

RA RB 1 set(1) 2 2 set(2) set(2) 2 2 max(2,2) max(2,2) 109

Slide 130

Slide 130 text

RA RB 1 set(1) 2 2 set(2) set(2) 2 2 max(2,2) max(2,2) 110 Incorrect value is computed because of incompatible lattice.

Slide 131

Slide 131 text

Advertisement Counter Application Flow 111

Slide 132

Slide 132 text

Server Client Client 112

Slide 133

Slide 133 text

Server Client Client 113 Client reads state from the server.

Slide 134

Slide 134 text

Server Client Client 114 Client locally mutates state.

Slide 135

Slide 135 text

Server Client Client 115 Client pushes changes back to the server.

Slide 136

Slide 136 text

Server Client Client 116

Slide 137

Slide 137 text

Server Client Client 117 Server enforces invariants over state.

Slide 138

Slide 138 text

Server Client Client 118 Client retrieves updated state periodically.

Slide 139

Slide 139 text

Server Client Client 119 Clients unable to communicate may violate invariant.

Slide 140

Slide 140 text

Advertisement Counter Application Design 120

Slide 141

Slide 141 text

Ads Contracts Product Filter Map Ads X Contracts Ads With Contracts Ads To Display Asynchronous Dataflow Counters Counters Counters Counter Ads Update (Insert) Application Initialization Counter Read > 50,000 Ads Update (Remove) Epsilon-Invariant Ads Map Configure Invariant Enforce Invariant 121

Slide 142

Slide 142 text

Ads Contracts Product Filter Map Ads X Contracts Ads With Contracts Ads To Display Asynchronous Dataflow 122

Slide 143

Slide 143 text

Counters Counters Counters Counter Ads Update (Insert) Application Initialization 123

Slide 144

Slide 144 text

Counter Read > 50,000 Ads Update (Remove) Epsilon-Invariant Ads Map Configure Invariant Enforce Invariant 124

Slide 145

Slide 145 text

Counter Read > 50,000 Ads Update (Remove) Epsilon-Invariant Ads Map Configure Invariant Enforce Invariant 125 Configure invariants for all of the advertisements.

Slide 146

Slide 146 text

Counter Read > 50,000 Ads Update (Remove) Epsilon-Invariant Ads Map Configure Invariant Enforce Invariant 126 Remove the advertisement from the list.

Slide 147

Slide 147 text

Advertisement Counter • Completely monotonic
 Disabling advertisements and contracts are all modeled through monotonic state growth 127

Slide 148

Slide 148 text

Advertisement Counter • Completely monotonic
 Disabling advertisements and contracts are all modeled through monotonic state growth • Arbitrary distribution
 Use of convergent data structures allows computational graph to be arbitrarily distributed 127

Slide 149

Slide 149 text

Advertisement Counter • Completely monotonic
 Disabling advertisements and contracts are all modeled through monotonic state growth • Arbitrary distribution
 Use of convergent data structures allows computational graph to be arbitrarily distributed • Divergence
 Divergence is a factor of synchronization period, concurrency, and throughput rate 127

Slide 150

Slide 150 text

Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, “correct-by-construction” applications) 3. Distributed, and fault-tolerant runtime
 (ex. replication, membership, dissemination) 128

Slide 151

Slide 151 text

Distributed Runtime Anabranch 129 work-in-progress

Slide 152

Slide 152 text

Anabranch • Layered approach
 Cluster membership and state dissemination for large clusters 130

Slide 153

Slide 153 text

Anabranch • Layered approach
 Cluster membership and state dissemination for large clusters • Delta-state synchronization 
 Efficient incremental state dissemination and anti-entropy mechanism [Almeida et al. 2016] 130

Slide 154

Slide 154 text

Anabranch • Layered approach
 Cluster membership and state dissemination for large clusters • Delta-state synchronization 
 Efficient incremental state dissemination and anti-entropy mechanism [Almeida et al. 2016] • Epsilon-invariants
 Lower-bound invariants, configurable at runtime 130

Slide 155

Slide 155 text

Anabranch • Layered approach
 Cluster membership and state dissemination for large clusters • Delta-state synchronization 
 Efficient incremental state dissemination and anti-entropy mechanism [Almeida et al. 2016] • Epsilon-invariants
 Lower-bound invariants, configurable at runtime • Scalable
 Demonstrated high scalability in production Cloud environments 130

Slide 156

Slide 156 text

Anabranch Layered Approach 131

Slide 157

Slide 157 text

Layered Approach 132

Slide 158

Slide 158 text

Layered Approach • Backend
 Configurable persistence layer depending on application. 132

Slide 159

Slide 159 text

Layered Approach • Backend
 Configurable persistence layer depending on application. • Membership
 Configurable membership protocol which can operate in a client-server or peer-to-peer mode [Leitao et al. 2007] 132

Slide 160

Slide 160 text

Layered Approach • Backend
 Configurable persistence layer depending on application. • Membership
 Configurable membership protocol which can operate in a client-server or peer-to-peer mode [Leitao et al. 2007] • Broadcast (via Gossip, Tree, etc.)
 Efficient dissemination of both program state and application state via gossip, broadcast tree, or hybrid mode [Leitao et al. 2007] 132

Slide 161

Slide 161 text

Mobile Phone Distributed Hash Table Application Language Execution KV Store KV Backend Membership Broadcast Layer Client/Server Peer-to-Peer

Slide 162

Slide 162 text

Mobile Phone Distributed Hash Table Application Language Execution KV Store KV Backend Membership Broadcast Layer Client/Server Peer-to-Peer Language and applications.

Slide 163

Slide 163 text

Mobile Phone Distributed Hash Table Application Language Execution KV Store KV Backend Membership Broadcast Layer Client/Server Peer-to-Peer Storage for CRDT state.

Slide 164

Slide 164 text

Mobile Phone Distributed Hash Table Application Language Execution KV Store KV Backend Membership Broadcast Layer Client/Server Peer-to-Peer State dissemination.

Slide 165

Slide 165 text

Anabranch Delta-state CRDTs 137

Slide 166

Slide 166 text

Delta-based Dissemination • Delta-state based CRDTs
 Reduces state transmission for clients 138

Slide 167

Slide 167 text

Delta-based Dissemination • Delta-state based CRDTs
 Reduces state transmission for clients • Operate locally
 Objects are mutated locally; delta’s buffered locally and periodically gossiped 138

Slide 168

Slide 168 text

Delta-based Dissemination • Delta-state based CRDTs
 Reduces state transmission for clients • Operate locally
 Objects are mutated locally; delta’s buffered locally and periodically gossiped • Only fixed number of clients
 Clients resort to full state synchronization when they’ve been partitioned too long 138

Slide 169

Slide 169 text

RC BufferA BufferB

Slide 170

Slide 170 text

RC BufferA BufferB (1, {b}, {}) {1} (1, {b}, {}) (1, {b}, {}) Buffer minimal change representation…

Slide 171

Slide 171 text

RC BufferA BufferB (1, {b}, {}) {1} (1, {b}, {}) (1, {b}, {}) {} (1, {b}, {b}) (1, {b}, {b}) (1, {b}, {b})

Slide 172

Slide 172 text

RC BufferA BufferB (1, {b}, {}) {1} (1, {b}, {}) (1, {b}, {}) {} (1, {b}, {b}) (1, {b}, {b}) (1, {b}, {b}) {2} (1, {b}, {b}) (2, {c}, {}) (2, {c}, {}) (2, {c}, {}) …then, disseminate state in causal order to neighbors.

Slide 173

Slide 173 text

RC BufferA BufferB (1, {b}, {}) {1} (1, {b}, {}) (1, {b}, {}) {} (1, {b}, {b}) (1, {b}, {b}) (1, {b}, {b}) {2} (1, {b}, {b}) (2, {c}, {}) (2, {c}, {}) (2, {c}, {}) {1, 2} (1, {a, b}, {b}) (2, {c}, {}) {1, 2} (1, {a}, {}) (1, {a}, {}) Only ship inflation from incoming state.

Slide 174

Slide 174 text

Anabranch Scalability 144

Slide 175

Slide 175 text

Scalability • 1024+ nodes
 Demonstrated scalability to 1024 nodes in Amazon cloud computing environment 145

Slide 176

Slide 176 text

Scalability • 1024+ nodes
 Demonstrated scalability to 1024 nodes in Amazon cloud computing environment • Modular approach
 Many of the components built and can be operated outside of Lasp to improve scalability of Erlang 145

Slide 177

Slide 177 text

Scalability • 1024+ nodes
 Demonstrated scalability to 1024 nodes in Amazon cloud computing environment • Modular approach
 Many of the components built and can be operated outside of Lasp to improve scalability of Erlang • Automated and repeatable
 Fully automated deployment, scenario execution, log aggregation and archival of experimental results 145

Slide 178

Slide 178 text

Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, “correct-by-construction” applications) 3. Distributed, and fault-tolerant runtime
 (ex. replication, membership, dissemination) 146

Slide 179

Slide 179 text

What have we learned? 147

Slide 180

Slide 180 text

We’ve build up from zero synchronization 148

Slide 181

Slide 181 text

We’ve build up from zero synchronization 148 Instead of working to remove synchronization

Slide 182

Slide 182 text

Programming SEC 1. Eliminate accidental nondeterminism
 Convergent Objects: Conflict-Free Replicated Data Types [Shapiro et al. 2011] 149

Slide 183

Slide 183 text

Programming SEC 1. Eliminate accidental nondeterminism
 Convergent Objects: Conflict-Free Replicated Data Types [Shapiro et al. 2011] 2. Retain the properties of functional programming
 Convergent Programs: Lattice Processing
 [Meiklejohn et al. 2015] 149

Slide 184

Slide 184 text

Programming SEC 1. Eliminate accidental nondeterminism
 Convergent Objects: Conflict-Free Replicated Data Types [Shapiro et al. 2011] 2. Retain the properties of functional programming
 Convergent Programs: Lattice Processing
 [Meiklejohn et al. 2015] 3. Distributed, and fault-tolerant runtime
 Distributed Runtime: Anabranch
 [Meiklejohn et al. work-in-progress] 149

Slide 185

Slide 185 text

Key Points • Synchronization is sometimes not possible
 Mobile and “Internet of Things” applications make synchronization for replicated state impractical 150

Slide 186

Slide 186 text

Key Points • Synchronization is sometimes not possible
 Mobile and “Internet of Things” applications make synchronization for replicated state impractical • Apply synchronization only where required
 Global invariants, atomic visibility, etc. 150

Slide 187

Slide 187 text

Key Points • Synchronization is sometimes not possible
 Mobile and “Internet of Things” applications make synchronization for replicated state impractical • Apply synchronization only where required
 Global invariants, atomic visibility, etc. • Holistic approach
 Taking a holistic approach to the design of distributed systems is important for building higher-availability applications 150

Slide 188

Slide 188 text

Artifacts • Lasp
 CRDT based programming model: testbed for Loquat
 http://github.com/lasp-lang/lasp 151

Slide 189

Slide 189 text

Artifacts • Lasp
 CRDT based programming model: testbed for Loquat
 http://github.com/lasp-lang/lasp • Partisan
 TCP-based pluggable membership service offering client/server, static, and HyParView- based protocols
 https://github.com/lasp-lang/partisan 151

Slide 190

Slide 190 text

Artifacts • Lasp
 CRDT based programming model: testbed for Loquat
 http://github.com/lasp-lang/lasp • Partisan
 TCP-based pluggable membership service offering client/server, static, and HyParView- based protocols
 https://github.com/lasp-lang/partisan • Plumtree
 Epidemic broadcast trees for use with Partisan
 https://github.com/lasp-lang/plumtree 151

Slide 191

Slide 191 text

Artifacts • Lasp
 CRDT based programming model: testbed for Loquat
 http://github.com/lasp-lang/lasp • Partisan
 TCP-based pluggable membership service offering client/server, static, and HyParView- based protocols
 https://github.com/lasp-lang/partisan • Plumtree
 Epidemic broadcast trees for use with Partisan
 https://github.com/lasp-lang/plumtree • Sprinter
 Service discovery and deployment for Mesos and Kubernetes
 https://github.com/lasp-lang/sprinter 151

Slide 192

Slide 192 text

Artifacts • Lasp
 CRDT based programming model: testbed for Loquat
 http://github.com/lasp-lang/lasp • Partisan
 TCP-based pluggable membership service offering client/server, static, and HyParView- based protocols
 https://github.com/lasp-lang/partisan • Plumtree
 Epidemic broadcast trees for use with Partisan
 https://github.com/lasp-lang/plumtree • Sprinter
 Service discovery and deployment for Mesos and Kubernetes
 https://github.com/lasp-lang/sprinter • Types
 CRDT implementations for Erlang
 https://github.com/lasp-lang/types 151

Slide 193

Slide 193 text

152 Christopher Meiklejohn @cmeik http://www.lasp-lang.org Thanks!