Selective Hearing: An Approach to Distributed, Eventually Consistent Edge Computation

Slide 1

Slide 1 text

An Approach to Distributed, Eventually Consistent Edge Computation Selective Hearing Christopher Meiklejohn, Machine Zone, Inc. Peter Van Roy, Université catholique de Louvain W-PSDS 2015, September 28, 2015 1

Slide 2

Slide 2 text

What is Edge Computation? 2

Slide 3

Slide 3 text

Edge Computation • Logical extremes of the network  Applications, data, and computation 3

Slide 4

Slide 4 text

Edge Computation • Logical extremes of the network  Applications, data, and computation • Especially important where synchronization is hard 3

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Edge Computation • Logical extremes of the network  Applications, data, and computation • Especially important where synchronization is hard • “Internet of Things”  Low power, limited memory and connectivity • Mobile Applications  Ofﬂine operation with replicated, shared state 3

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Selective Hearing • Epidemic broadcast based runtime system  Provide a runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efﬁcient execution 4

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Selective Hearing • Epidemic broadcast based runtime system  Provide a runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efﬁcient execution • Well-matched to Lattice Processing (Lasp) • Epidemic broadcast mechanisms provide weak ordering but are resilient and efﬁcient • Lasp’s programming model is tolerant to message re-ordering, disconnections, and node failures 4

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Background Lattice Processing 5

Slide 14

Slide 14 text

Lattice Processing (Lasp) • Distributed, deterministic dataﬂow  Distributed, deterministic dataﬂow programming model for “eventually consistent” computations 6

Slide 15

Slide 15 text

Lattice Processing (Lasp) • Distributed, deterministic dataﬂow  Distributed, deterministic dataﬂow programming model for “eventually consistent” computations • Convergent data structures  Primary data abstraction is the CRDT 6

Slide 16

Slide 16 text

Slide 17

Slide 17 text

7 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).

Slide 18

Slide 18 text

8 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).

Slide 19

Slide 19 text

9 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).

Slide 20

Slide 20 text

10 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).

Slide 21

Slide 21 text

11 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).

Slide 22

Slide 22 text

Lattice Processing (Lasp) • Functional and set-theoretic operations on sets  Product, intersection, union, ﬁlter, map, fold 12

Slide 23

Slide 23 text

Lattice Processing (Lasp) • Functional and set-theoretic operations on sets  Product, intersection, union, ﬁlter, map, fold • Metadata computation  Performs transformation on the internal metadata of CRDTs allowing creation of “composed” CRDTs 12

Slide 24

Slide 24 text

Lasp Processes • Replicas as monotonic streams  Each replica of a CRDT produces a monotonic stream of states 13

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Lasp Processes • Replicas as monotonic streams  Each replica of a CRDT produces a monotonic stream of states • Monotonic processes  Read from one or more input replica streams and produce a single output replica stream • Inﬂationary reads  Read operation ensures that we only read inﬂationary updates to replicas 13

Slide 27

Slide 27 text

Lattice Processing Monotonic Processes 14

Slide 28

Slide 28 text

RA {} P1 F(RA) {} 15

Slide 29

Slide 29 text

RA {} P1 F(RA) {} strict_read({}) 16

Slide 30

Slide 30 text

RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) 17

Slide 31

Slide 31 text

RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1, {a}, {})) 18

Slide 32

Slide 32 text

RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1, {a}, {})) (2, {a}, {}) 19

Slide 33

Slide 33 text

RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1, {a}, {})) (2, {a}, {}) strict_read((1, {a}, {}) (1, {a}, {}) 20

Slide 34

Slide 34 text

RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1, {a}, {})) (2, {a}, {}) strict_read((1, {a}, {}) (1, {a}, {}) (1, {a}, {a}) F((1, {a}, {a})) (2, {a}, {a}) strict_read((1, {a}, {a}) (1, {a}, {a}) 21

Slide 35

Slide 35 text

RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) (1, {a}, {}) (1, {a}, {a}) F((1, {a}, {a})) (2, {a}, {a}) strict_read((1, {a}, {a}) (1, {a}, {a}) 22

Slide 36

Slide 36 text

Example Application Advertisement Counter 23

Slide 37

Slide 37 text

Advertisement Counter • Mobile game platform selling advertisement space  Advertisements are paid according to a minimum number of impressions 24

Slide 38

Slide 38 text

Advertisement Counter • Mobile game platform selling advertisement space  Advertisements are paid according to a minimum number of impressions • Clients will go oﬄine  Clients have limited connectivity and the system still needs to make progress while clients are ofﬂine 24

Slide 39

Slide 39 text

Slide 40

Slide 40 text

Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Riot Ads Rovio Ads Product Read 50,000 Remove Increment Union 26 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client

Slide 41

Slide 41 text

Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Rovio Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 1 Client 27 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client

Slide 42

Slide 42 text

Ads ovio Ad ounter 1 ovio Ad ounter 2 Riot Ad ounter 1 Riot Ad ounter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Rovio Ad Counter 1 Ro C Rovio Ad Counter 1 Ro C Rovio Ad Counter 1 Ro C Rovio Ad Counter 1 Ro C Client Side, Sing 28 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client

Slide 43

Slide 43 text

Ads Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product move Read Union Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 29 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client

Slide 44

Slide 44 text

Ads Contracts Ads Contracts Ads With Contracts Filter Product Read Union Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 30 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client

Slide 45

Slide 45 text

Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 31 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client

Slide 46

Slide 46 text

Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Riot Ads Rovio Ads Fil Product Read 50,000 Remove Increment Union 32 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client

Slide 47

Slide 47 text

Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 33 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client

Slide 48

Slide 48 text

Advertisement Counter • Completely monotonic  Disabling advertisements and contracts are all modeled through monotonic state growth 34

Slide 49

Slide 49 text

Slide 50

Slide 50 text

Advertisement Counter • Completely monotonic  Disabling advertisements and contracts are all modeled through monotonic state growth • Divergence  Divergence is a factor of synchronization period • Arbitrary distribution  Use of convergent data structures allows computational graph to be arbitrarily distributed 34

Slide 51

Slide 51 text

Client as processes 35

Slide 52

Slide 52 text

Client as processes 35 Server as cluster of nodes in a hash ring

Slide 53

Slide 53 text

Slide 54

Slide 54 text

What if we want to have multiple replication schemes used in a single dataﬂow graph? 37

Slide 55

Slide 55 text

Slide 56

Slide 56 text

Distribution Abstraction • Previous model assumes Dynamo-style  Instead, allow nodes in the graph to represent objects that have their own replication scheme 39

Slide 57

Slide 57 text

Slide 58

Slide 58 text

Slide 59

Slide 59 text

Slide 60

Slide 60 text

Slide 61

Slide 61 text

Distribution Abstraction • Previous model assumes Dynamo-style  Instead, allow nodes in the graph to represent objects that have their own replication scheme • Each node in the dataﬂow graph: • Replicated state machine • Dynamo-style “quorum-intersection” • Single value • Epidemic broadcast for data propagation  Facilitates distribution of “program state” across nodes in the dataﬂow graph 39

Slide 62

Slide 62 text

Selective Hearing Semantics 40

Slide 63

Slide 63 text

Two Layer Approach • Lasp 41

Slide 64

Slide 64 text

Two Layer Approach • Lasp • Variables and values are stored on all nodes 41

Slide 65

Slide 65 text

Two Layer Approach • Lasp • Variables and values are stored on all nodes • Read operations: 41

Slide 66

Slide 66 text

Two Layer Approach • Lasp • Variables and values are stored on all nodes • Read operations: • Declare “interest” in a value 41

Slide 67

Slide 67 text

Slide 68

Slide 68 text

Two Layer Approach • Lasp • Variables and values are stored on all nodes • Read operations: • Declare “interest” in a value • Store a continuation to invoke   Predicate function on a lattice position • Bind operations (update operations, as well): 41

Slide 69

Slide 69 text

Slide 70

Slide 70 text

Two Layer Approach • Gossip 42

Slide 71

Slide 71 text

Two Layer Approach • Gossip • Weak ordering of broadcast messages  Variable state and metadata for declare and bind operations 42

Slide 72

Slide 72 text

Two Layer Approach • Gossip • Weak ordering of broadcast messages  Variable state and metadata for declare and bind operations • Bind operations: 42

Slide 73

Slide 73 text

Slide 74

Slide 74 text

Two Layer Approach • Gossip • Weak ordering of broadcast messages  Variable state and metadata for declare and bind operations • Bind operations: • Uniquely identify messages  Pair of global message identiﬁer and version vector • Incremented by coordinating replica  When receiving incoming vector, merge with current vector and advance 42

Slide 75

Slide 75 text

Slide 76

Slide 76 text

Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Riot Ads Rovio Ads Fil Product Read 50,000 Remove Increment Union 43 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client

Slide 77

Slide 77 text

44 server({#ad{counter=Counter}=Ad, _}, Ads) -> strict_read(Counter, 50000), update(Ads, {remove, Ad}, Ad). Ad Counter Process

Slide 78

Slide 78 text

44 server({#ad{counter=Counter}=Ad, _}, Ads) -> strict_read(Counter, 50000), update(Ads, {remove, Ad}, Ad). Ad Counter Process • Blocking read of the counter for value 50000

Slide 79

Slide 79 text

Slide 80

Slide 80 text

44 server({#ad{counter=Counter}=Ad, _}, Ads) -> strict_read(Counter, 50000), update(Ads, {remove, Ad}, Ad). Ad Counter Process • Blocking read of the counter for value 50000 • Records interest in variable Counter • Registers a continuation that will invoke when Counter >= 50000 Continuation

Slide 81

Slide 81 text

Slide 82

Slide 82 text

45 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).

Slide 83

Slide 83 text

46 Map Process map(S1, Function, S2) -> map(S1, Function, Bottom(S1), S2). map(S1, Function, Last, S2) -> Continuation = fun(Current) -> Result = compute(Function, Current), bind(S2, Result), map(S1, P, Current, S2) end, read(S1, fun(Current) -> Last < Current, Continuation).

Slide 84

Slide 84 text

47 Map Process map(S1, Function, S2) -> map(S1, Function, Bottom(S1), S2). map(S1, Function, Last, S2) -> Continuation = fun(Current) -> Result = compute(Function, Current), bind(S2, Result), map(S1, P, Current, S2) end, read(S1, fun(Current) -> Last < Current, Continuation). Invoke Map

Slide 85

Slide 85 text

48 Map Process map(S1, Function, S2) -> map(S1, Function, Bottom(S1), S2). map(S1, Function, Last, S2) -> Continuation = fun(Current) -> Result = compute(Function, Current), bind(S2, Result), map(S1, P, Current, S2) end, read(S1, fun(Current) -> Last < Current, Continuation). Generate continuation

Slide 86

Slide 86 text

49 Map Process map(S1, Function, S2) -> map(S1, Function, Bottom(S1), S2). map(S1, Function, Last, S2) -> Continuation = fun(Current) -> Result = compute(Function, Current), bind(S2, Result), map(S1, P, Current, S2) end, read(S1, fun(Current) -> Last < Current, Continuation). Invoke continuation on change

Slide 87

Slide 87 text

What comes next? 50

Slide 88

Slide 88 text

Future Work • Quantitative evaluation  Evaluation and optimization of the prototype implementation in Erlang with Lasp and Plumtree 51

Slide 89

Slide 89 text

Slide 90

Slide 90 text

Future Work • Quantitative evaluation  Evaluation and optimization of the prototype implementation in Erlang with Lasp and Plumtree • Extensions to increase locality  Local variables that do not get distributed • Partial evaluation  Can we derive optimized distribution models given dataﬂow execution graphs? 51

Slide 91

Slide 91 text

Conclusion • Successful pairing  Pairing of a weak delivery model with a model that does not rely on ordering guarantees 52

Slide 92

Slide 92 text

Conclusion • Successful pairing  Pairing of a weak delivery model with a model that does not rely on ordering guarantees • Distributed epidemic-based runtime  First implementation of a epidemic broadcast based runtime in a general programming model 52

Slide 93

Slide 93 text

SyncFree is a European research project taking place for 3 years, staring October 2013, and is funded by the European Union, grant agreement n° 609551. 53

Slide 94

Slide 94 text

Thanks! 54 Christopher Meiklejohn @cmeik