Selective Hearing: An Approach to Distributed, Eventually Consistent Edge Computation

W-PSDS 2015, with SRDS 2015

Transcript

1. An Approach to Distributed, Eventually Consistent Edge Computation Selective Hearing Christopher Meiklejohn, Machine Zone, Inc. Peter Van Roy, Université catholique de Louvain W-PSDS 2015, September 28, 2015 1
2. What is Edge Computation? 2
3. Edge Computation • Logical extremes of the network
 Applications, data, and computation 3
4. Edge Computation • Logical extremes of the network
 Applications, data, and computation • Especially important where synchronization is hard 3
5. Edge Computation • Logical extremes of the network
 Applications, data, and computation • Especially important where synchronization is hard • “Internet of Things”
 Low power, limited memory and connectivity 3
6. Edge Computation • Logical extremes of the network
 Applications, data, and computation • Especially important where synchronization is hard • “Internet of Things”
 Low power, limited memory and connectivity • Mobile Applications
 Offline operation with replicated, shared state 3
7. Edge Computation • Logical extremes of the network
 Applications, data, and computation • Especially important where synchronization is hard • “Internet of Things”
 Low power, limited memory and connectivity • Mobile Applications
 Offline operation with replicated, shared state • How should we manage state? 3
8. Selective Hearing • Epidemic broadcast based runtime system
 Provide a runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efficient execution 4
9. Selective Hearing • Epidemic broadcast based runtime system
 Provide a runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efficient execution • Well-matched to Lattice Processing (Lasp) 4
10. Selective Hearing • Epidemic broadcast based runtime system
 Provide a runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efficient execution • Well-matched to Lattice Processing (Lasp) • Epidemic broadcast mechanisms provide weak ordering but are resilient and efficient 4
11. Selective Hearing • Epidemic broadcast based runtime system
 Provide a runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efficient execution • Well-matched to Lattice Processing (Lasp) • Epidemic broadcast mechanisms provide weak ordering but are resilient and efficient • Lasp’s programming model is tolerant to message re-ordering, disconnections, and node failures 4
12. Selective Hearing • Epidemic broadcast based runtime system
 Provide a runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efficient execution • Well-matched to Lattice Processing (Lasp) • Epidemic broadcast mechanisms provide weak ordering but are resilient and efficient • Lasp’s programming model is tolerant to message re-ordering, disconnections, and node failures • “Selective Receive”
 Nodes selectively receive and process messages based on interest. 4
13. Background Lattice Processing 5
14. Lattice Processing (Lasp) • Distributed, deterministic dataflow
 Distributed, deterministic dataflow programming model for “eventually consistent” computations 6
15. Lattice Processing (Lasp) • Distributed, deterministic dataflow
 Distributed, deterministic dataflow programming model for “eventually consistent” computations • Convergent data structures
 Primary data abstraction is the CRDT 6
16. Lattice Processing (Lasp) • Distributed, deterministic dataflow
 Distributed, deterministic dataflow programming model for “eventually consistent” computations • Convergent data structures
 Primary data abstraction is the CRDT • Enables composition
 Provides functional composition of CRDTs that preserves the SEC property 6
17. 7 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
18. 8 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
19. 9 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
20. 10 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
21. 11 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
22. Lattice Processing (Lasp) • Functional and set-theoretic operations on sets
 Product, intersection, union, filter, map, fold 12
23. Lattice Processing (Lasp) • Functional and set-theoretic operations on sets
 Product, intersection, union, filter, map, fold • Metadata computation
 Performs transformation on the internal metadata of CRDTs allowing creation of “composed” CRDTs 12
24. Lasp Processes • Replicas as monotonic streams
 Each replica of a CRDT produces a monotonic stream of states 13
25. Lasp Processes • Replicas as monotonic streams
 Each replica of a CRDT produces a monotonic stream of states • Monotonic processes
 Read from one or more input replica streams and produce a single output replica stream 13
26. Lasp Processes • Replicas as monotonic streams
 Each replica of a CRDT produces a monotonic stream of states • Monotonic processes
 Read from one or more input replica streams and produce a single output replica stream • Inflationary reads
 Read operation ensures that we only read inflationary updates to replicas 13
27. Lattice Processing Monotonic Processes 14
28. RA {} P1 F(RA) {} 15
29. RA {} P1 F(RA) {} strict_read({}) 16
30. RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) 17
31. RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1, {a}, {})) 18
32. RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1, {a}, {})) (2, {a}, {}) 19
33. RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1, {a}, {})) (2, {a}, {}) strict_read((1, {a}, {}) (1, {a}, {}) 20
34. RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1, {a}, {})) (2, {a}, {}) strict_read((1, {a}, {}) (1, {a}, {}) (1, {a}, {a}) F((1, {a}, {a})) (2, {a}, {a}) strict_read((1, {a}, {a}) (1, {a}, {a}) 21
35. RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) (1, {a}, {}) (1, {a}, {a}) F((1, {a}, {a})) (2, {a}, {a}) strict_read((1, {a}, {a}) (1, {a}, {a}) 22
36. Example Application Advertisement Counter 23
37. Advertisement Counter • Mobile game platform selling advertisement space
 Advertisements are paid according to a minimum number of impressions 24
38. Advertisement Counter • Mobile game platform selling advertisement space
 Advertisements are paid according to a minimum number of impressions • Clients will go offline
 Clients have limited connectivity and the system still needs to make progress while clients are offline 24
39. Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 25
40. Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Riot Ads Rovio Ads Product Read 50,000 Remove Increment Union 26 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
41. Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Rovio Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 1 Client 27 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
42. Ads ovio Ad ounter 1 ovio Ad ounter 2 Riot Ad ounter 1 Riot Ad ounter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Rovio Ad Counter 1 Ro C Rovio Ad Counter 1 Ro C Rovio Ad Counter 1 Ro C Rovio Ad Counter 1 Ro C Client Side, Sing 28 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
43. Ads Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product move Read Union Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 29 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
44. Ads Contracts Ads Contracts Ads With Contracts Filter Product Read Union Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 30 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
45. Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 31 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
46. Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Riot Ads Rovio Ads Fil Product Read 50,000 Remove Increment Union 32 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
47. Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 33 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
48. Advertisement Counter • Completely monotonic
 Disabling advertisements and contracts are all modeled through monotonic state growth 34
49. Advertisement Counter • Completely monotonic
 Disabling advertisements and contracts are all modeled through monotonic state growth • Divergence
 Divergence is a factor of synchronization period 34
50. Advertisement Counter • Completely monotonic
 Disabling advertisements and contracts are all modeled through monotonic state growth • Divergence
 Divergence is a factor of synchronization period • Arbitrary distribution
 Use of convergent data structures allows computational graph to be arbitrarily distributed 34
51. Client as processes 35
52. Client as processes 35 Server as cluster of nodes in a hash ring
53. Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 36 Server/Client Distribution
54. What if we want to have multiple replication schemes used in a single dataflow graph? 37
55. Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 38 Geo-Replicated Distribution
56. Distribution Abstraction • Previous model assumes Dynamo-style
 Instead, allow nodes in the graph to represent objects that have their own replication scheme 39
57. Distribution Abstraction • Previous model assumes Dynamo-style
 Instead, allow nodes in the graph to represent objects that have their own replication scheme • Each node in the dataflow graph: 39
58. Distribution Abstraction • Previous model assumes Dynamo-style
 Instead, allow nodes in the graph to represent objects that have their own replication scheme • Each node in the dataflow graph: • Replicated state machine 39
59. Distribution Abstraction • Previous model assumes Dynamo-style
 Instead, allow nodes in the graph to represent objects that have their own replication scheme • Each node in the dataflow graph: • Replicated state machine • Dynamo-style “quorum-intersection” 39
60. Distribution Abstraction • Previous model assumes Dynamo-style
 Instead, allow nodes in the graph to represent objects that have their own replication scheme • Each node in the dataflow graph: • Replicated state machine • Dynamo-style “quorum-intersection” • Single value 39
61. Distribution Abstraction • Previous model assumes Dynamo-style
 Instead, allow nodes in the graph to represent objects that have their own replication scheme • Each node in the dataflow graph: • Replicated state machine • Dynamo-style “quorum-intersection” • Single value • Epidemic broadcast for data propagation
 Facilitates distribution of “program state” across nodes in the dataflow graph 39
62. Selective Hearing Semantics 40
63. Two Layer Approach • Lasp 41
64. Two Layer Approach • Lasp • Variables and values are stored on all nodes 41
65. Two Layer Approach • Lasp • Variables and values are stored on all nodes • Read operations: 41
66. Two Layer Approach • Lasp • Variables and values are stored on all nodes • Read operations: • Declare “interest” in a value 41
67. Two Layer Approach • Lasp • Variables and values are stored on all nodes • Read operations: • Declare “interest” in a value • Store a continuation to invoke 
 Predicate function on a lattice position 41
68. Two Layer Approach • Lasp • Variables and values are stored on all nodes • Read operations: • Declare “interest” in a value • Store a continuation to invoke 
 Predicate function on a lattice position • Bind operations (update operations, as well): 41
69. Two Layer Approach • Lasp • Variables and values are stored on all nodes • Read operations: • Declare “interest” in a value • Store a continuation to invoke 
 Predicate function on a lattice position • Bind operations (update operations, as well): • Bind a value on each “interested node”
 This performs a merge with the nodes current value 41
70. Two Layer Approach • Gossip 42
71. Two Layer Approach • Gossip • Weak ordering of broadcast messages
 Variable state and metadata for declare and bind operations 42
72. Two Layer Approach • Gossip • Weak ordering of broadcast messages
 Variable state and metadata for declare and bind operations • Bind operations: 42
73. Two Layer Approach • Gossip • Weak ordering of broadcast messages
 Variable state and metadata for declare and bind operations • Bind operations: • Uniquely identify messages
 Pair of global message identifier and version vector 42
74. Two Layer Approach • Gossip • Weak ordering of broadcast messages
 Variable state and metadata for declare and bind operations • Bind operations: • Uniquely identify messages
 Pair of global message identifier and version vector • Incremented by coordinating replica
 When receiving incoming vector, merge with current vector and advance 42
75. Two Layer Approach • Gossip • Weak ordering of broadcast messages
 Variable state and metadata for declare and bind operations • Bind operations: • Uniquely identify messages
 Pair of global message identifier and version vector • Incremented by coordinating replica
 When receiving incoming vector, merge with current vector and advance • Monotonic state increases per-replica
 Ignore messages that have been subsumed by later messages 42
76. Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Riot Ads Rovio Ads Fil Product Read 50,000 Remove Increment Union 43 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
77. 44 server({#ad{counter=Counter}=Ad, _}, Ads) -> strict_read(Counter, 50000), update(Ads, {remove, Ad}, Ad). Ad Counter Process
78. 44 server({#ad{counter=Counter}=Ad, _}, Ads) -> strict_read(Counter, 50000), update(Ads, {remove, Ad}, Ad). Ad Counter Process • Blocking read of the counter for value 50000
79. 44 server({#ad{counter=Counter}=Ad, _}, Ads) -> strict_read(Counter, 50000), update(Ads, {remove, Ad}, Ad). Ad Counter Process • Blocking read of the counter for value 50000 • Records interest in variable Counter
80. 44 server({#ad{counter=Counter}=Ad, _}, Ads) -> strict_read(Counter, 50000), update(Ads, {remove, Ad}, Ad). Ad Counter Process • Blocking read of the counter for value 50000 • Records interest in variable Counter • Registers a continuation that will invoke when Counter >= 50000 Continuation
81. 44 server({#ad{counter=Counter}=Ad, _}, Ads) -> strict_read(Counter, 50000), update(Ads, {remove, Ad}, Ad). Ad Counter Process • Blocking read of the counter for value 50000 • Records interest in variable Counter • Registers a continuation that will invoke when Counter >= 50000 • Interest in variable Counter removed Continuation
82. 45 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
83. 46 Map Process map(S1, Function, S2) -> map(S1, Function, Bottom(S1), S2). map(S1, Function, Last, S2) -> Continuation = fun(Current) -> Result = compute(Function, Current), bind(S2, Result), map(S1, P, Current, S2) end, read(S1, fun(Current) -> Last < Current, Continuation).
84. 47 Map Process map(S1, Function, S2) -> map(S1, Function, Bottom(S1), S2). map(S1, Function, Last, S2) -> Continuation = fun(Current) -> Result = compute(Function, Current), bind(S2, Result), map(S1, P, Current, S2) end, read(S1, fun(Current) -> Last < Current, Continuation). Invoke Map
85. 48 Map Process map(S1, Function, S2) -> map(S1, Function, Bottom(S1), S2). map(S1, Function, Last, S2) -> Continuation = fun(Current) -> Result = compute(Function, Current), bind(S2, Result), map(S1, P, Current, S2) end, read(S1, fun(Current) -> Last < Current, Continuation). Generate continuation
86. 49 Map Process map(S1, Function, S2) -> map(S1, Function, Bottom(S1), S2). map(S1, Function, Last, S2) -> Continuation = fun(Current) -> Result = compute(Function, Current), bind(S2, Result), map(S1, P, Current, S2) end, read(S1, fun(Current) -> Last < Current, Continuation). Invoke continuation on change
87. What comes next? 50
88. Future Work • Quantitative evaluation
 Evaluation and optimization of the prototype implementation in Erlang with Lasp and Plumtree 51
89. Future Work • Quantitative evaluation
 Evaluation and optimization of the prototype implementation in Erlang with Lasp and Plumtree • Extensions to increase locality
 Local variables that do not get distributed 51
90. Future Work • Quantitative evaluation
 Evaluation and optimization of the prototype implementation in Erlang with Lasp and Plumtree • Extensions to increase locality
 Local variables that do not get distributed • Partial evaluation
 Can we derive optimized distribution models given dataflow execution graphs? 51
91. Conclusion • Successful pairing
 Pairing of a weak delivery model with a model that does not rely on ordering guarantees 52
92. Conclusion • Successful pairing
 Pairing of a weak delivery model with a model that does not rely on ordering guarantees • Distributed epidemic-based runtime
 First implementation of a epidemic broadcast based runtime in a general programming model 52
93. SyncFree is a European research project taking place for 3 years, staring October 2013, and is funded by the European Union, grant agreement n° 609551. 53
94. Thanks! 54 Christopher Meiklejohn @cmeik