and computation • Especially challenging where synchronization is hard • “Internet of Things” Low power, limited memory and connectivity • Mobile Applications Offline operation with replicated, shared state 3
and computation • Especially challenging where synchronization is hard • “Internet of Things” Low power, limited memory and connectivity • Mobile Applications Offline operation with replicated, shared state • How should we manage events generated at the device? 3
events to a centralized location for processing • Most general approach, however expensive Events must be buffered while devices are offline; power requirements for operating the antenna 4
events to a centralized location for processing • Most general approach, however expensive Events must be buffered while devices are offline; power requirements for operating the antenna • Design a distributed algorithm (Directed/Digest Diffusion, TAG) Design an algorithm optimized for program dissemination and collection of results 4
events to a centralized location for processing • Most general approach, however expensive Events must be buffered while devices are offline; power requirements for operating the antenna • Design a distributed algorithm (Directed/Digest Diffusion, TAG) Design an algorithm optimized for program dissemination and collection of results • Least general, however efficient Algorithm can be designed specifically to address unordered delivery, and optimized for minimal state transmission 4
functional programming model over distributed data structures (CRDTs) • Extend our model with new data structures Two new data structures: Pair and Bounded-LWW-Set 6
functional programming model over distributed data structures (CRDTs) • Extend our model with new data structures Two new data structures: Pair and Bounded-LWW-Set • Extend our model with dynamic scope “Dynamic” variables, where each node contains a unique value for a given variable which can be aggregated with a “dynamic” fold operation 6
counters, registers, flags, maps • Strong Eventual Consistency (SEC) Objects that receive the same updates, regardless of order, will reach equivalent state 8
programming model for “eventually consistent” computations • Convergent data structures Primary data abstraction is the CRDT • Enables composition Provides functional composition of CRDTs that preserves the SEC property 15
elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
of events into a local device average • Merge local averages per device Fold local averages across devices into a global average replicated at each device 23
global average. GlobalAverage = declare({counter, counter}, global_average), %% Declare a dynamic variable. Samples = declare_dynamic({bounded_lww_set, 100}), %% Define a local average; computed from the local Bounded-LWW set. LocalAverage = declare_dynamic({counter, counter}), %% Register an event handler with the sensor that is triggered each %% time an event X is triggered at a given timestamp T. EventHandler = fun({X, T} -> update(Samples, {add, x, t}, Actor) end register_event_handler(EventHandler), %% Fold samples using the function `avg' into a local average. fold(Samples, fun avg/2, LocalAverage) %% Fold local average using the function `avg' into a global average. fold_dynamic(LocalAverage, fun sum_pairs/2, GlobalAverage)
global average. GlobalAverage = declare({counter, counter}, global_average), %% Declare a dynamic variable. Samples = declare_dynamic({bounded_lww_set, 100}), %% Define a local average; computed from the local Bounded-LWW set. LocalAverage = declare_dynamic({counter, counter}), %% Register an event handler with the sensor that is triggered each %% time an event X is triggered at a given timestamp T. EventHandler = fun({X, T} -> update(Samples, {add, x, t}, Actor) end register_event_handler(EventHandler), %% Fold samples using the function `avg' into a local average. fold(Samples, fun avg/2, LocalAverage), %% Fold local average using the function `avg' into a global average. fold_dynamic(LocalAverage, fun sum_pairs/2, GlobalAverage)
global average. GlobalAverage = declare({counter, counter}, global_average), %% Declare a dynamic variable. Samples = declare_dynamic({bounded_lww_set, 100}), %% Define a local average; computed from the local Bounded-LWW set. LocalAverage = declare_dynamic({counter, counter}), %% Register an event handler with the sensor that is triggered each %% time an event X is triggered at a given timestamp T. EventHandler = fun({X, T} -> update(Samples, {add, x, t}, Actor) end register_event_handler(EventHandler), %% Fold samples using the function `avg' into a local average. fold(Samples, fun avg/2, LocalAverage), %% Fold local average using the function `avg' into a global average. fold_dynamic(LocalAverage, fun sum_pairs/2, GlobalAverage)
global average. GlobalAverage = declare({counter, counter}, global_average), %% Declare a dynamic variable. Samples = declare_dynamic({bounded_lww_set, 100}), %% Define a local average; computed from the local Bounded-LWW set. LocalAverage = declare_dynamic({counter, counter}), %% Register an event handler with the sensor that is triggered each %% time an event X is triggered at a given timestamp T. EventHandler = fun({X, T} -> update(Samples, {add, x, t}, Actor) end register_event_handler(EventHandler), %% Fold samples using the function `avg' into a local average. fold(Samples, fun avg/2, LocalAverage), %% Fold local average using the function `avg' into a global average. fold_dynamic(LocalAverage, fun sum_pairs/2, GlobalAverage)
global average. GlobalAverage = declare({counter, counter}, global_average), %% Declare a dynamic variable. Samples = declare_dynamic({bounded_lww_set, 100}), %% Define a local average; computed from the local Bounded-LWW set. LocalAverage = declare_dynamic({counter, counter}), %% Register an event handler with the sensor that is triggered each %% time an event X is triggered at a given timestamp T. EventHandler = fun({X, T} -> update(Samples, {add, x, t}, Actor) end register_event_handler(EventHandler), %% Fold samples using the function `avg' into a local average. fold(Samples, fun avg/2, LocalAverage), %% Fold local average using the function `avg' into a global average. fold_dynamic(LocalAverage, fun sum_pairs/2, GlobalAverage)
global average. GlobalAverage = declare({counter, counter}, global_average), %% Declare a dynamic variable. Samples = declare_dynamic({bounded_lww_set, 100}), %% Define a local average; computed from the local Bounded-LWW set. LocalAverage = declare_dynamic({counter, counter}), %% Register an event handler with the sensor that is triggered each %% time an event X is triggered at a given timestamp T. EventHandler = fun({X, T} -> update(Samples, {add, x, t}, Actor) end register_event_handler(EventHandler), %% Fold samples using the function `avg' into a local average. fold(Samples, fun avg/2, LocalAverage), %% Fold local average using the function `avg' into a global average. fold_dynamic(LocalAverage, fun sum_pairs/2, GlobalAverage)
global average. GlobalAverage = declare({counter, counter}, global_average), %% Declare a dynamic variable. Samples = declare_dynamic({bounded_lww_set, 100}), %% Define a local average; computed from the local Bounded-LWW set. LocalAverage = declare_dynamic({counter, counter}), %% Register an event handler with the sensor that is triggered each %% time an event X is triggered at a given timestamp T. EventHandler = fun({X, T} -> update(Samples, {add, x, t}, Actor) end register_event_handler(EventHandler), %% Fold samples using the function `avg' into a local average. fold(Samples, fun avg/2, LocalAverage), %% Fold local average using the function `avg' into a global average. fold_dynamic(LocalAverage, fun sum_pairs/2, GlobalAverage)
Observed-Remove Set, but enforces a maximum number of elements • Objects tagged with local time Each object in the set is tagged with a local time from insertion time 38
Observed-Remove Set, but enforces a maximum number of elements • Objects tagged with local time Each object in the set is tagged with a local time from insertion time • Objects marked “removed” when bound exceeded Use a tombstone to mark objects as removed when performing insertions and merges 38
Observed-Remove Set, but enforces a maximum number of elements • Objects tagged with local time Each object in the set is tagged with a local time from insertion time • Objects marked “removed” when bound exceeded Use a tombstone to mark objects as removed when performing insertions and merges • “Last-Writer-Wins” with Single Writer Single writer removes possible non-determinism inherent with LWW- registers with replicated state 38
across all nodes with a given identifier with their own value that is not replicated • Dynamic Fold operation Combines the value using a merge across all nodes through pairwise synchronization until fixed point is reached 48
and program dissemination with optimizations when aggregations are monotonic; however, not tolerant to some network anomalies • Tiny AGgregation Declarative method for data collection across sensors using SQL-like syntax; however, not a general programming model 54
and program dissemination with optimizations when aggregations are monotonic; however, not tolerant to some network anomalies • Tiny AGgregation Declarative method for data collection across sensors using SQL-like syntax; however, not a general programming model • PVARS Similar to *Lisp parallel variables, where each processor had it’s own value and could apply an operation across nodes 54
prototype implementation in Erlang • Optimizations to reduce metadata Apply known CRDT optimizations to both the fold operation and data structures to reduce space complexity 55
cmeiklejohn
uhooi
dscs
os1ma
startjapan
yag_ays
satoki
takahirom
dropout009
rakyll
teshi04
guni1192
reverentgeek
sachag
stephaniewalter
lauravandoore
moore
mza
jonyablonski
ammeep
maltzj
cromwellryan
marcelosomers
geeforr
