and computation • Especially challenging where synchronization is hard • “Internet of Things” Low power, limited memory and connectivity • Mobile Applications Offline operation with replicated, shared state 3
and computation • Especially challenging where synchronization is hard • “Internet of Things” Low power, limited memory and connectivity • Mobile Applications Offline operation with replicated, shared state • How should we manage events generated at the device? 3
events to a centralized location for processing • Most general approach, however expensive Events must be buffered while devices are offline; power requirements for operating the antenna 4
events to a centralized location for processing • Most general approach, however expensive Events must be buffered while devices are offline; power requirements for operating the antenna • Design a distributed algorithm (Directed/Digest Diffusion, TAG) Design an algorithm optimized for program dissemination and collection of results 4
events to a centralized location for processing • Most general approach, however expensive Events must be buffered while devices are offline; power requirements for operating the antenna • Design a distributed algorithm (Directed/Digest Diffusion, TAG) Design an algorithm optimized for program dissemination and collection of results • Least general, however efficient Algorithm can be designed specifically to address unordered delivery, and optimized for minimal state transmission 4
functional programming model over distributed data structures (CRDTs) • Extend our model with new data structures Two new data structures: Pair and Bounded-LWW-Set 6
functional programming model over distributed data structures (CRDTs) • Extend our model with new data structures Two new data structures: Pair and Bounded-LWW-Set • Extend our model with dynamic scope “Dynamic” variables, where each node contains a unique value for a given variable which can be aggregated with a “dynamic” fold operation 6
counters, registers, flags, maps • Strong Eventual Consistency (SEC) Objects that receive the same updates, regardless of order, will reach equivalent state 8
programming model for “eventually consistent” computations • Convergent data structures Primary data abstraction is the CRDT • Enables composition Provides functional composition of CRDTs that preserves the SEC property 15
elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
of events into a local device average • Merge local averages per device Fold local averages across devices into a global average replicated at each device 23
global average. GlobalAverage = declare({counter, counter}, global_average), %% Declare a dynamic variable. Samples = declare_dynamic({bounded_lww_set, 100}), %% Define a local average; computed from the local Bounded-LWW set. LocalAverage = declare_dynamic({counter, counter}), %% Register an event handler with the sensor that is triggered each %% time an event X is triggered at a given timestamp T. EventHandler = fun({X, T} -> update(Samples, {add, x, t}, Actor) end register_event_handler(EventHandler), %% Fold samples using the function `avg' into a local average. fold(Samples, fun avg/2, LocalAverage) %% Fold local average using the function `avg' into a global average. fold_dynamic(LocalAverage, fun sum_pairs/2, GlobalAverage)
global average. GlobalAverage = declare({counter, counter}, global_average), %% Declare a dynamic variable. Samples = declare_dynamic({bounded_lww_set, 100}), %% Define a local average; computed from the local Bounded-LWW set. LocalAverage = declare_dynamic({counter, counter}), %% Register an event handler with the sensor that is triggered each %% time an event X is triggered at a given timestamp T. EventHandler = fun({X, T} -> update(Samples, {add, x, t}, Actor) end register_event_handler(EventHandler), %% Fold samples using the function `avg' into a local average. fold(Samples, fun avg/2, LocalAverage), %% Fold local average using the function `avg' into a global average. fold_dynamic(LocalAverage, fun sum_pairs/2, GlobalAverage)
global average. GlobalAverage = declare({counter, counter}, global_average), %% Declare a dynamic variable. Samples = declare_dynamic({bounded_lww_set, 100}), %% Define a local average; computed from the local Bounded-LWW set. LocalAverage = declare_dynamic({counter, counter}), %% Register an event handler with the sensor that is triggered each %% time an event X is triggered at a given timestamp T. EventHandler = fun({X, T} -> update(Samples, {add, x, t}, Actor) end register_event_handler(EventHandler), %% Fold samples using the function `avg' into a local average. fold(Samples, fun avg/2, LocalAverage), %% Fold local average using the function `avg' into a global average. fold_dynamic(LocalAverage, fun sum_pairs/2, GlobalAverage)
global average. GlobalAverage = declare({counter, counter}, global_average), %% Declare a dynamic variable. Samples = declare_dynamic({bounded_lww_set, 100}), %% Define a local average; computed from the local Bounded-LWW set. LocalAverage = declare_dynamic({counter, counter}), %% Register an event handler with the sensor that is triggered each %% time an event X is triggered at a given timestamp T. EventHandler = fun({X, T} -> update(Samples, {add, x, t}, Actor) end register_event_handler(EventHandler), %% Fold samples using the function `avg' into a local average. fold(Samples, fun avg/2, LocalAverage), %% Fold local average using the function `avg' into a global average. fold_dynamic(LocalAverage, fun sum_pairs/2, GlobalAverage)
global average. GlobalAverage = declare({counter, counter}, global_average), %% Declare a dynamic variable. Samples = declare_dynamic({bounded_lww_set, 100}), %% Define a local average; computed from the local Bounded-LWW set. LocalAverage = declare_dynamic({counter, counter}), %% Register an event handler with the sensor that is triggered each %% time an event X is triggered at a given timestamp T. EventHandler = fun({X, T} -> update(Samples, {add, x, t}, Actor) end register_event_handler(EventHandler), %% Fold samples using the function `avg' into a local average. fold(Samples, fun avg/2, LocalAverage), %% Fold local average using the function `avg' into a global average. fold_dynamic(LocalAverage, fun sum_pairs/2, GlobalAverage)
global average. GlobalAverage = declare({counter, counter}, global_average), %% Declare a dynamic variable. Samples = declare_dynamic({bounded_lww_set, 100}), %% Define a local average; computed from the local Bounded-LWW set. LocalAverage = declare_dynamic({counter, counter}), %% Register an event handler with the sensor that is triggered each %% time an event X is triggered at a given timestamp T. EventHandler = fun({X, T} -> update(Samples, {add, x, t}, Actor) end register_event_handler(EventHandler), %% Fold samples using the function `avg' into a local average. fold(Samples, fun avg/2, LocalAverage), %% Fold local average using the function `avg' into a global average. fold_dynamic(LocalAverage, fun sum_pairs/2, GlobalAverage)
global average. GlobalAverage = declare({counter, counter}, global_average), %% Declare a dynamic variable. Samples = declare_dynamic({bounded_lww_set, 100}), %% Define a local average; computed from the local Bounded-LWW set. LocalAverage = declare_dynamic({counter, counter}), %% Register an event handler with the sensor that is triggered each %% time an event X is triggered at a given timestamp T. EventHandler = fun({X, T} -> update(Samples, {add, x, t}, Actor) end register_event_handler(EventHandler), %% Fold samples using the function `avg' into a local average. fold(Samples, fun avg/2, LocalAverage), %% Fold local average using the function `avg' into a global average. fold_dynamic(LocalAverage, fun sum_pairs/2, GlobalAverage)
Observed-Remove Set, but enforces a maximum number of elements • Objects tagged with local time Each object in the set is tagged with a local time from insertion time 38
Observed-Remove Set, but enforces a maximum number of elements • Objects tagged with local time Each object in the set is tagged with a local time from insertion time • Objects marked “removed” when bound exceeded Use a tombstone to mark objects as removed when performing insertions and merges 38
Observed-Remove Set, but enforces a maximum number of elements • Objects tagged with local time Each object in the set is tagged with a local time from insertion time • Objects marked “removed” when bound exceeded Use a tombstone to mark objects as removed when performing insertions and merges • “Last-Writer-Wins” with Single Writer Single writer removes possible non-determinism inherent with LWW- registers with replicated state 38
across all nodes with a given identifier with their own value that is not replicated • Dynamic Fold operation Combines the value using a merge across all nodes through pairwise synchronization until fixed point is reached 48
and program dissemination with optimizations when aggregations are monotonic; however, not tolerant to some network anomalies • Tiny AGgregation Declarative method for data collection across sensors using SQL-like syntax; however, not a general programming model 54
and program dissemination with optimizations when aggregations are monotonic; however, not tolerant to some network anomalies • Tiny AGgregation Declarative method for data collection across sensors using SQL-like syntax; however, not a general programming model • PVARS Similar to *Lisp parallel variables, where each processor had it’s own value and could apply an operation across nodes 54
prototype implementation in Erlang • Optimizations to reduce metadata Apply known CRDT optimizations to both the fold operation and data structures to reduce space complexity 55
cmeiklejohn
yujisoftware
javiergs
kakao
cer
satoshi256kbyte
tnagao7
tsoshiro
manfredsteyer
masaki12
shpigford
bcantrill
geeforr
eileencodes
jakevdp
tenderlove
addyosmani
samanthasiow
skipperchong
csswizardry
jmmastey
