Slide 1

Slide 1 text

Coordination-Free Computations Christopher Meiklejohn

Slide 2

Slide 2 text

DISTRIBUTED, EVENTUALLY CONSISTENT COMPUTATIONS LASP 2 CHRISTOPHER MEIKLEJOHN (BASHO TECHNOLOGIES, INC.) PETER VAN ROY (UNIVERSITÉ CATHOLIQUE DE LOUVAIN)

Slide 3

Slide 3 text

LASP MOTIVATION 3

Slide 4

Slide 4 text

EXPENSIVE / IMPRACTICAL SYNCHRONIZATION IS 4

Slide 5

Slide 5 text

MOBILE GAMES: SHARED STATE BETWEEN CLIENTS CLIENTS GO OFFLINE 5 http://www.rovio.com/en/news/blog/261/263-million-monthly-active-users-in-december/

Slide 6

Slide 6 text

DISJOINT STATE AGGREGATED UPSTREAM CLIENTS GO OFFLINE “INTERNET OF THINGS”: 6 Gubbi, Jayavardhana, et al. "Internet of Things (IoT): A vision, architectural elements, and future directions." Future Generation Computer Systems 29.7 (2013): 1645-1660.

Slide 7

Slide 7 text

NO TOTAL ORDER: REPLICATED SHARED STATE WITH OFFLINE CLIENTS CLIENTS NEED TO MAKE PROGRESS 7 Gilbert, Seth, and Nancy Lynch. "Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services." ACM SIGACT News 33.2 (2002): 51-59.

Slide 8

Slide 8 text

WALL CLOCKS: UNRELIABLE AT BEST NON-DETERMINISTIC IF USED IN COMPUTATIONS 8 Corbett, James C., et al. "Spanner: Google’s globally distributed database." ACM Transactions on Computer Systems (TOCS) 31.3 (2013): 8.

Slide 9

Slide 9 text

RECONCILED BY USER CONCURRENCY 9

Slide 10

Slide 10 text

RA RB 1 3 2 ? ? set(1) set(2) set(3) 10

Slide 11

Slide 11 text

RA RB 1 3 2 ? ? set(1) set(2) set(3) 11

Slide 12

Slide 12 text

RA RB 1 3 2 ? ? set(1) set(2) set(3) 12

Slide 13

Slide 13 text

RA RB 1 3 2 ? ? set(1) set(2) set(3) 13

Slide 14

Slide 14 text

CRDTs 14

Slide 15

Slide 15 text

DETERMINISTIC RESOLUTION CRDTs PROVIDE 15

Slide 16

Slide 16 text

CRDTs: MAPS, SETS, COUNTERS, REGISTERS, GRAPHS DETERMINISTIC RESOLUTION 16

Slide 17

Slide 17 text

“CORRECT REPLICAS THAT HAVE DELIVERED THE SAME UPDATES HAVE EQUIVALENT STATE” Shapiro, Marc, et al. "Conflict-free replicated data types." Stabilization, Safety, and Security of Distributed Systems. Springer Berlin Heidelberg, 2011. 386-400. MONOTONIC STRONG EVENTUAL CONSISTENCY CRDTs REALIZE 17

Slide 18

Slide 18 text

‘MAX’ REGISTER CRDTs EXAMPLE 18

Slide 19

Slide 19 text

RA RB 1 3 2 3 3 set(1) set(2) set(3) max(2,3) max(2,3) 19

Slide 20

Slide 20 text

‘ORSET’ SET CRDTs EXAMPLE 20

Slide 21

Slide 21 text

RA RB RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) 21

Slide 22

Slide 22 text

RA RB RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) 22

Slide 23

Slide 23 text

RA RB RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) 23

Slide 24

Slide 24 text

RA RB RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) 24

Slide 25

Slide 25 text

FRAGMENTS & PROGRAMS 25

Slide 26

Slide 26 text

{1,2,3} {3,4,5} 26

Slide 27

Slide 27 text

{1,2,3} {3,4,5} Intersection {3} 27

Slide 28

Slide 28 text

{1,2,3} {3,4,5} Intersection {3} Map {9} 28

Slide 29

Slide 29 text

{1,2,3} {3,4,5} Intersection {3} Map {9} 29

Slide 30

Slide 30 text

{1,2,3} {3,4,5} Intersection {3} Map {9} 30

Slide 31

Slide 31 text

{1,2,3} {3,4,5} Intersection {3} Map {9} 31

Slide 32

Slide 32 text

{1,2,3} {3,4,5} Intersection {3} Map {9} {1,2,3} {3,4,5} Intersection {3} Map {9} 32

Slide 33

Slide 33 text

{1,2,3} {3,4,5} Intersection {3} Map {9} {1,2,3} {3,4,5} Intersection {3} Map {9} 33

Slide 34

Slide 34 text

{1,2,3} {3,4,5} Intersection {3} Map {9} {1,2,3} {3,4,5} Intersection {3} Map {9} 34

Slide 35

Slide 35 text

NONTRIVIAL FUNCTION APPLICATION AND DATA COMPOSITION IS 35

Slide 36

Slide 36 text

RA RB RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RC) {2} fun(X) -> 2 end {2} {} 36

Slide 37

Slide 37 text

RA RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RC) {2} fun(X) -> 2 end {2} {} F(RA) {2} fun(X) -> 2 end {2} {2} {2} 37

Slide 38

Slide 38 text

RA RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RC) {2} fun(X) -> 2 end {2} {} F(RA) {2} fun(X) -> 2 end {2} {2} {2} 38

Slide 39

Slide 39 text

RA RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RC) {2} fun(X) -> 2 end {2} {} F(RA) {2} fun(X) -> 2 end {2} {2} {2} 39

Slide 40

Slide 40 text

RA RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RC) {2} fun(X) -> 2 end {2} {} F(RA) {2} fun(X) -> 2 end {2} {2} {2} 40

Slide 41

Slide 41 text

COMPOSITION: USER OBSERVABLE VALUE VS. STATE METADATA MAPPING IS NONTRIVIAL WITHOUT MAPPING METADATA; UNMERGABLE 41 Brown, Russell, et al. "Riak dt map: A composable, convergent replicated dictionary." Proceedings of the First Workshop on Principles and Practice of Eventual Consistency. ACM, 2014. Conway, Neil, et al. "Logic and lattices for distributed programming." Proceedings of the Third ACM Symposium on Cloud Computing. ACM, 2012. Meiklejohn, Christopher. "On the composability of the Riak DT map: expanding from embedded to multi-key structures." Proceedings of the First Workshop on Principles and Practice of Eventual Consistency. ACM, 2014.

Slide 42

Slide 42 text

LASP LANGUAGE 42

Slide 43

Slide 43 text

LASP: DISTRIBUTED RUNTIME IMPLEMENTED AS ERLANG LIBRARY USES RIAK-CORE 43

Slide 44

Slide 44 text

LASP: CRDTS AS STREAMS OF STATE CHANGES CRDTS CONNECTED BY MONOTONIC PROCESSES MANY TO ONE MAPPING OF CRDTS 44

Slide 45

Slide 45 text

PRIMITIVE OPERATIONS: MONOTONIC READ, UPDATE FUNCTIONAL: MAP, FILTER, FOLD SET-THEORETIC: PRODUCT, UNION, INTERSECTION LIFTED TO OPERATE OVER METADATA 45

Slide 46

Slide 46 text

MAP EXAMPLE 46

Slide 47

Slide 47 text

RA RB RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RB) {2} fun(X) -> 2 end {2} {} (2, {b}, {}) (2, {b}, {b}) {} (2, {a, b}, {b}) (2, {a, b}, {b}) F(RC) {2} fun(X) -> 2 end (2, {a, b}, {b}) {2} {} {2} (2, {b}, {}) (2, {b}, {b}) (2, {a, b}, {b}) F(RA) {2} fun(X) -> 2 end (2, {a, b}, {b}) {2} {2} {2} (2, {a}, {}) (2, {a, b}, {}) (2, {a, b}, {b}) 47

Slide 48

Slide 48 text

RA RB RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RB) {2} fun(X) -> 2 end {2} {} (2, {b}, {}) (2, {b}, {b}) {} (2, {a, b}, {b}) (2, {a, b}, {b}) F(RC) {2} fun(X) -> 2 end (2, {a, b}, {b}) {2} {} {2} (2, {b}, {}) (2, {b}, {b}) (2, {a, b}, {b}) F(RA) {2} fun(X) -> 2 end (2, {a, b}, {b}) {2} {2} {2} (2, {a}, {}) (2, {a, b}, {}) (2, {a, b}, {b}) 48

Slide 49

Slide 49 text

RA RB RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RB) {2} fun(X) -> 2 end {2} {} (2, {b}, {}) (2, {b}, {b}) {} (2, {a, b}, {b}) (2, {a, b}, {b}) F(RC) {2} fun(X) -> 2 end (2, {a, b}, {b}) {2} {} {2} (2, {b}, {}) (2, {b}, {b}) (2, {a, b}, {b}) F(RA) {2} fun(X) -> 2 end (2, {a, b}, {b}) {2} {2} {2} (2, {a}, {}) (2, {a, b}, {}) (2, {a, b}, {b}) 49

Slide 50

Slide 50 text

RA RB RC {1} (1, {a}, {}) {1} (1, {b}, {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RB) {2} fun(X) -> 2 end {2} {} (2, {b}, {}) (2, {b}, {b}) {} (2, {a, b}, {b}) (2, {a, b}, {b}) F(RC) {2} fun(X) -> 2 end (2, {a, b}, {b}) {2} {} {2} (2, {b}, {}) (2, {b}, {b}) (2, {a, b}, {b}) F(RA) {2} fun(X) -> 2 end (2, {a, b}, {b}) {2} {2} {2} (2, {a}, {}) (2, {a, b}, {}) (2, {a, b}, {b}) 50

Slide 51

Slide 51 text

ARCHITECTURE 51

Slide 52

Slide 52 text

SHARED VARIABLE STORE LASP STORE 52

Slide 53

Slide 53 text

LEVELDB, BITCASK, ETS LASP BACKENDS 53

Slide 54

Slide 54 text

PROVIDED BY RIAK DT LASP CRDTs 54

Slide 55

Slide 55 text

CENTRALIZED SEMANTICS LASP ARCHITECTURE 55

Slide 56

Slide 56 text

STORE: SHARED VARIABLE STORE PROCESSESS SYNCHRONIZE ON VARIABLES 56

Slide 57

Slide 57 text

DISTRIBUTED SEMANTICS LASP ARCHITECTURE 57

Slide 58

Slide 58 text

STORE: REPLICATED, SHARDED VARIABLE STORE PROCESSESS SYNCHRONIZE ON VARIABLES 58

Slide 59

Slide 59 text

REPLICATED: DISTRIBUTED WITH RIAK CORE QUORUM REQUESTS; ANTI-ENTROPY PROTOCOL 59

Slide 60

Slide 60 text

HYBRID: DISTRIBUTE PROGRAMS; R/W WITH LOCAL STORE CENTRALIZED EXECUTION 60

Slide 61

Slide 61 text

EXAMPLES 61

Slide 62

Slide 62 text

%% Create initial set. {ok, S1} = lasp:declare(Type), %% Add elements to initial set and update. {ok, _} = lasp:update(S1, {add_all, [1,2,3]}, a), %% Create second set. {ok, S2} = lasp:declare(Type), %% Apply map. ok = lasp:map(S1, fun(X) -> X * 2 end, S2),

Slide 63

Slide 63 text

%% Create initial set. {ok, S1} = lasp_core:declare(Type, Store), %% Add elements to initial set and update. {ok, _} = lasp_core:update(S1, {add_all, [1,2,3]}, a, Store), %% Create second set. {ok, S2} = lasp_core:declare(Type, Store), %% Apply map. ok = lasp_core:map(S1, fun(X) -> X * 2 end, S2, Store),

Slide 64

Slide 64 text

AD COUNTER 64

Slide 65

Slide 65 text

AD COUNTER: TRACKS AD IMPRESSIONS PUSHES ADVERTISEMENTS TO THE CLIENT DISABLES AD AT 50,000+ IMPRESSIONS CLIENTS DISPLAY ADS WHEN OFFLINE 65

Slide 66

Slide 66 text

INFORMATION FLOW AD COUNTER 66

Slide 67

Slide 67 text

Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter Riot Ad Counter Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 67

Slide 68

Slide 68 text

Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter Riot Ad Counter Contracts Riot Ads Rovio Ads Produ Read 50,000 Remove Increment Union 68

Slide 69

Slide 69 text

Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter Riot Ad Counter Contracts Ads Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT 69

Slide 70

Slide 70 text

Ads Contracts Ads Contracts Ads With Contracts ds Ads Filter Product Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT 70

Slide 71

Slide 71 text

Ads ntracts Ads Contracts Ads With Contracts Filter Product Read Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT 71

Slide 72

Slide 72 text

Ads Contracts Ads With Contracts Filter Read Operation Maintained CRDT Maintained CRDT Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 72

Slide 73

Slide 73 text

Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter Riot Ad Counter Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 73

Slide 74

Slide 74 text

Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter Riot Ad Counter Contracts Ads Contracts Riot Ads Rovio Ads Product Read 50,000 Remove Increment Union Lasp Operation User-Maintained C Lasp-Maintained C 74

Slide 75

Slide 75 text

Ads Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 75

Slide 76

Slide 76 text

INFORMATION FLOW: MONOTONIC METADATA TO PREVENT DUPLICATE PROPAGATION 76

Slide 77

Slide 77 text

EXAMPLE CODE AD COUNTER 77

Slide 78

Slide 78 text

%% @doc Client process; standard recursive looping server. client(Id, AdsWithContracts, PreviousValue) -> receive view_ad -> %% Get current ad list. {ok, {_, _, AdList0}} = lasp:read(AdsWithContracts, PreviousValue), AdList = riak_dt_orset:value(AdList0), case length(AdList) of 0 -> %% No advertisements left to display; ignore %% message. client(Id, AdsWithContracts, AdList0); _ -> %% Select a random advertisement from the list of %% active advertisements. {#ad{counter=Ad}, _} = lists:nth( random:uniform(length(AdList)), AdList), %% Increment it. {ok, _} = lasp:update(Ad, increment, Id), lager:info("Incremented ad counter: ~p", [Ad]), client(Id, AdsWithContracts, AdList0) end end.

Slide 79

Slide 79 text

%% @doc Server functions for the advertisement counter. After 5 views, %% disable the advertisement. %% server({#ad{counter=Counter}=Ad, _}, Ads) -> %% Blocking threshold read for 5 advertisement impressions. {ok, _} = lasp:read(Counter, 5), %% Remove the advertisement. {ok, _} = lasp:update(Ads, {remove, Ad}, Ad), lager:info("Removing ad: ~p", [Ad]).

Slide 80

Slide 80 text

%% Generate a series of unique identifiers. RovioAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Rovio Ad Identifiers are: ~p", [RovioAdIds]), TriforkAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Trifork Ad Identifiers are: ~p", [TriforkAdIds]), Ids = RovioAdIds ++ TriforkAdIds, lager:info("Ad Identifiers are: ~p", [Ids]), %% Generate Rovio's advertisements. {ok, RovioAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(RovioAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, RovioAdIds), %% Generate Trifork's advertisements. {ok, TriforkAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(TriforkAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, TriforkAdIds), %% Union ads. {ok, Ads} = lasp:declare(?SET), ok = lasp:union(RovioAds, TriforkAds, Ads), %% For each identifier, generate a contract. {ok, Contracts} = lasp:declare(?SET), lists:map(fun(Id) -> {ok, _} = lasp:update(Contracts, {add, #contract{id=Id}}, undefined) end, Ids), %% Compute the Cartesian product of both ads and contracts. {ok, AdsContracts} = lasp:declare(?SET), ok = lasp:product(Ads, Contracts, AdsContracts), %% Filter items by join on item it. {ok, AdsWithContracts} = lasp:declare(?SET), FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) -> Id1 =:= Id2 end, ok = lasp:filter(AdsContracts, FilterFun, AdsWithContracts), %% Launch a series of client processes, each of which is responsible %% for displaying a particular advertisement. %% Generate a OR-set for tracking clients. {ok, Clients} = lasp:declare(?SET), %% Each client takes the full list of ads when it starts, and reads %% from the variable store. lists:map(fun(Id) -> ClientPid = spawn_link(?MODULE, client, [Id, AdsWithContracts, undefined]), {ok, _} = lasp:update(Clients, {add, ClientPid}, undefined) end, lists:seq(1,5)), %% Launch a server process for each advertisement, which will block %% until the advertisement should be disabled. %% Create a OR-set for the server list. {ok, Servers} = lasp:declare(?SET), %% Get the current advertisement list. {ok, {_, _, AdList0}} = lasp:read(AdsWithContracts), AdList = riak_dt_orset:value(AdList0), %% For each advertisement, launch one server for tracking it's %% impressions and wait to disable. lists:map(fun(Ad) -> ServerPid = spawn_link(?MODULE, server, [Ad, Ads]), {ok, _} = lasp:update(Servers, {add, ServerPid}, undefined) end, AdList),

Slide 81

Slide 81 text

%% Generate a series of unique identifiers. RovioAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Rovio Ad Identifiers are: ~p", [RovioAdIds]), TriforkAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Trifork Ad Identifiers are: ~p", [TriforkAdIds]), Ids = RovioAdIds ++ TriforkAdIds, lager:info("Ad Identifiers are: ~p", [Ids]), %% Generate Rovio's advertisements. {ok, RovioAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(RovioAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, RovioAdIds), %% Generate Trifork's advertisements. {ok, TriforkAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(TriforkAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, TriforkAdIds), %% Union ads. {ok, Ads} = lasp:declare(?SET), ok = lasp:union(RovioAds, TriforkAds, Ads), %% For each identifier, generate a contract. {ok, Contracts} = lasp:declare(?SET), lists:map(fun(Id) -> {ok, _} = lasp:update(Contracts, {add, #contract{id=Id}}, undefined) end, Ids), %% Compute the Cartesian product of both ads and contracts. {ok, AdsContracts} = lasp:declare(?SET), ok = lasp:product(Ads, Contracts, AdsContracts), %% Filter items by join on item it. {ok, AdsWithContracts} = lasp:declare(?SET), FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) -> Id1 =:= Id2 end, ok = lasp:filter(AdsContracts, FilterFun, AdsWithContracts), %% Launch a series of client processes, each of which is responsible %% for displaying a particular advertisement. %% Generate a OR-set for tracking clients. {ok, Clients} = lasp:declare(?SET), %% Each client takes the full list of ads when it starts, and reads %% from the variable store. lists:map(fun(Id) -> ClientPid = spawn_link(?MODULE, client, [Id, AdsWithContracts, undefined]), {ok, _} = lasp:update(Clients, {add, ClientPid}, undefined) end, lists:seq(1,5)), %% Launch a server process for each advertisement, which will block %% until the advertisement should be disabled. %% Create a OR-set for the server list. {ok, Servers} = lasp:declare(?SET), %% Get the current advertisement list. {ok, {_, _, AdList0}} = lasp:read(AdsWithContracts), AdList = riak_dt_orset:value(AdList0), %% For each advertisement, launch one server for tracking it's %% impressions and wait to disable. lists:map(fun(Ad) -> ServerPid = spawn_link(?MODULE, server, [Ad, Ads]), {ok, _} = lasp:update(Servers, {add, ServerPid}, undefined) end, AdList),

Slide 82

Slide 82 text

%% Generate a series of unique identifiers. RovioAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Rovio Ad Identifiers are: ~p", [RovioAdIds]), TriforkAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Trifork Ad Identifiers are: ~p", [TriforkAdIds]), Ids = RovioAdIds ++ TriforkAdIds, lager:info("Ad Identifiers are: ~p", [Ids]), %% Generate Rovio's advertisements. {ok, RovioAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(RovioAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, RovioAdIds), %% Generate Trifork's advertisements. {ok, TriforkAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(TriforkAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, TriforkAdIds), %% Union ads. {ok, Ads} = lasp:declare(?SET), ok = lasp:union(RovioAds, TriforkAds, Ads), %% For each identifier, generate a contract. {ok, Contracts} = lasp:declare(?SET), lists:map(fun(Id) -> {ok, _} = lasp:update(Contracts, {add, #contract{id=Id}}, undefined) end, Ids), %% Compute the Cartesian product of both ads and contracts. {ok, AdsContracts} = lasp:declare(?SET), ok = lasp:product(Ads, Contracts, AdsContracts), %% Filter items by join on item it. {ok, AdsWithContracts} = lasp:declare(?SET), FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) -> Id1 =:= Id2 end, ok = lasp:filter(AdsContracts, FilterFun, AdsWithContracts), %% Launch a series of client processes, each of which is responsible %% for displaying a particular advertisement. %% Generate a OR-set for tracking clients. {ok, Clients} = lasp:declare(?SET), %% Each client takes the full list of ads when it starts, and reads %% from the variable store. lists:map(fun(Id) -> ClientPid = spawn_link(?MODULE, client, [Id, AdsWithContracts, undefined]), {ok, _} = lasp:update(Clients, {add, ClientPid}, undefined) end, lists:seq(1,5)), %% Launch a server process for each advertisement, which will block %% until the advertisement should be disabled. %% Create a OR-set for the server list. {ok, Servers} = lasp:declare(?SET), %% Get the current advertisement list. {ok, {_, _, AdList0}} = lasp:read(AdsWithContracts), AdList = riak_dt_orset:value(AdList0), %% For each advertisement, launch one server for tracking it's %% impressions and wait to disable. lists:map(fun(Ad) -> ServerPid = spawn_link(?MODULE, server, [Ad, Ads]), {ok, _} = lasp:update(Servers, {add, ServerPid}, undefined) end, AdList),

Slide 83

Slide 83 text

%% Generate a series of unique identifiers. RovioAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Rovio Ad Identifiers are: ~p", [RovioAdIds]), TriforkAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Trifork Ad Identifiers are: ~p", [TriforkAdIds]), Ids = RovioAdIds ++ TriforkAdIds, lager:info("Ad Identifiers are: ~p", [Ids]), %% Generate Rovio's advertisements. {ok, RovioAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(RovioAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, RovioAdIds), %% Generate Trifork's advertisements. {ok, TriforkAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(TriforkAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, TriforkAdIds), %% Union ads. {ok, Ads} = lasp:declare(?SET), ok = lasp:union(RovioAds, TriforkAds, Ads), %% For each identifier, generate a contract. {ok, Contracts} = lasp:declare(?SET), lists:map(fun(Id) -> {ok, _} = lasp:update(Contracts, {add, #contract{id=Id}}, undefined) end, Ids), %% Compute the Cartesian product of both ads and contracts. {ok, AdsContracts} = lasp:declare(?SET), ok = lasp:product(Ads, Contracts, AdsContracts), %% Filter items by join on item it. {ok, AdsWithContracts} = lasp:declare(?SET), FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) -> Id1 =:= Id2 end, ok = lasp:filter(AdsContracts, FilterFun, AdsWithContracts), %% Launch a series of client processes, each of which is responsible %% for displaying a particular advertisement. %% Generate a OR-set for tracking clients. {ok, Clients} = lasp:declare(?SET), %% Each client takes the full list of ads when it starts, and reads %% from the variable store. lists:map(fun(Id) -> ClientPid = spawn_link(?MODULE, client, [Id, AdsWithContracts, undefined]), {ok, _} = lasp:update(Clients, {add, ClientPid}, undefined) end, lists:seq(1,5)), %% Launch a server process for each advertisement, which will block %% until the advertisement should be disabled. %% Create a OR-set for the server list. {ok, Servers} = lasp:declare(?SET), %% Get the current advertisement list. {ok, {_, _, AdList0}} = lasp:read(AdsWithContracts), AdList = riak_dt_orset:value(AdList0), %% For each advertisement, launch one server for tracking it's %% impressions and wait to disable. lists:map(fun(Ad) -> ServerPid = spawn_link(?MODULE, server, [Ad, Ads]), {ok, _} = lasp:update(Servers, {add, ServerPid}, undefined) end, AdList),

Slide 84

Slide 84 text

%% Generate a series of unique identifiers. RovioAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Rovio Ad Identifiers are: ~p", [RovioAdIds]), TriforkAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Trifork Ad Identifiers are: ~p", [TriforkAdIds]), Ids = RovioAdIds ++ TriforkAdIds, lager:info("Ad Identifiers are: ~p", [Ids]), %% Generate Rovio's advertisements. {ok, RovioAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(RovioAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, RovioAdIds), %% Generate Trifork's advertisements. {ok, TriforkAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(TriforkAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, TriforkAdIds), %% Union ads. {ok, Ads} = lasp:declare(?SET), ok = lasp:union(RovioAds, TriforkAds, Ads), %% For each identifier, generate a contract. {ok, Contracts} = lasp:declare(?SET), lists:map(fun(Id) -> {ok, _} = lasp:update(Contracts, {add, #contract{id=Id}}, undefined) end, Ids), %% Compute the Cartesian product of both ads and contracts. {ok, AdsContracts} = lasp:declare(?SET), ok = lasp:product(Ads, Contracts, AdsContracts), %% Filter items by join on item it. {ok, AdsWithContracts} = lasp:declare(?SET), FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) -> Id1 =:= Id2 end, ok = lasp:filter(AdsContracts, FilterFun, AdsWithContracts), %% Launch a series of client processes, each of which is responsible %% for displaying a particular advertisement. %% Generate a OR-set for tracking clients. {ok, Clients} = lasp:declare(?SET), %% Each client takes the full list of ads when it starts, and reads %% from the variable store. lists:map(fun(Id) -> ClientPid = spawn_link(?MODULE, client, [Id, AdsWithContracts, undefined]), {ok, _} = lasp:update(Clients, {add, ClientPid}, undefined) end, lists:seq(1,5)), %% Launch a server process for each advertisement, which will block %% until the advertisement should be disabled. %% Create a OR-set for the server list. {ok, Servers} = lasp:declare(?SET), %% Get the current advertisement list. {ok, {_, _, AdList0}} = lasp:read(AdsWithContracts), AdList = riak_dt_orset:value(AdList0), %% For each advertisement, launch one server for tracking it's %% impressions and wait to disable. lists:map(fun(Ad) -> ServerPid = spawn_link(?MODULE, server, [Ad, Ads]), {ok, _} = lasp:update(Servers, {add, ServerPid}, undefined) end, AdList),

Slide 85

Slide 85 text

DISTRIBUTION AD COUNTER 85

Slide 86

Slide 86 text

Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter Riot Ad Counter Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 86

Slide 87

Slide 87 text

Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter Riot Ad Counter Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 87

Slide 88

Slide 88 text

Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter Riot Ad Counter Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 88

Slide 89

Slide 89 text

Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter Riot Ad Counter Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 89

Slide 90

Slide 90 text

DISTRIBUTION BOUNDARIES: ARBITRARY ALLOWS COMPOSITION OF ENTIRE SYSTEM 90

Slide 91

Slide 91 text

MATERIALIZED VIEWS 91

Slide 92

Slide 92 text

MATERIALIZED VIEWS: INCREMENTALLY UPDATING, DELTA PROPAGATION SCHEMA BASED OR DYNAMIC; 2I IN RIAK 92

Slide 93

Slide 93 text

DATABASE AS STREAM OF UPDATES 93

Slide 94

Slide 94 text

Riak K1 K2 K3 K1 K2 94

Slide 95

Slide 95 text

Riak K1 K2 K3 K1 K2 RS1 RS2 RS3 95

Slide 96

Slide 96 text

Riak K1 K2 K3 K1 K2 RS1 RS2 RS3 K1 K1 K2 K2 K3 96

Slide 97

Slide 97 text

Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1 K2 K2 K3 N1 N2 N3 97

Slide 98

Slide 98 text

COMPOSE PROGRAMS 98

Slide 99

Slide 99 text

Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1 K2 K2 K3 N1 N2 N3 Lasp Stream Processor Template 99

Slide 100

Slide 100 text

Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1 K2 K2 K3 N1 N2 N3 Lasp Stream Processor Template Lasp All Objects Program 100

Slide 101

Slide 101 text

Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1 K2 K2 K3 N1 N2 N3 Lasp Stream Processor Template Lasp All Objects Program Lasp “Over 65” Filter 101

Slide 102

Slide 102 text

Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1 K2 K2 K3 N1 N2 N3 Lasp Stream Processor Template Lasp All Objects Program Lasp “Over 65” Filter Lasp “Over 80” Filter 102

Slide 103

Slide 103 text

Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1 K2 K2 K3 N1 N2 N3 Lasp Stream Processor Template Lasp All Objects Program Lasp “Over 65” Filter Lasp “Over 80” Filter Lasp “Named Chris” Filter (L) 103

Slide 104

Slide 104 text

DISTRIBUTE APPLICATIONS 104

Slide 105

Slide 105 text

Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1 K2 K2 K3 N1 N2 N3 Lasp “Named Chris” Filter (L) L’’’ L’’ L’ 105

Slide 106

Slide 106 text

MERGE RESULTS (SEC) 106 N1 N2 N3 L’ L’ L’

Slide 107

Slide 107 text

SUM RESULTS 107 RS1 L’ RS2 L’’

Slide 108

Slide 108 text

Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1 K2 K2 K3 N1 N2 N3 Execution of Lasp “Named Chris” Filter (L) L’’’ L’’ L’ 108

Slide 109

Slide 109 text

CACHE: CACHE RESULTS OF MERGE SPECULATIVELY EXECUTE BASED ON DIVERGENCE 109

Slide 110

Slide 110 text

“INTERNET OF THINGS” 110

Slide 111

Slide 111 text

Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1 K2 K2 K3 N1 N2 N3 Execution of Lasp “Named Chris” Filter (L) L’’’ L’’ L’ 111

Slide 112

Slide 112 text

IoT Network K1 K2 K3 K1 K2 S1 S2 S3 U1 U1 U2 U2 U3 Execution of Lasp Temperature Sensor > 90 F L’’’ L’’ L’ 112

Slide 113

Slide 113 text

IoT Network S1 S2 S3 U1 U1 U2 U2 U3 Execution of Lasp Temperature Sensor > 90 F L’’’ L’’ L’ 113

Slide 114

Slide 114 text

IOT: EXECUTE AT THE EDGE WRITE CODE THINKING ABOUT “ALL” DATA 114

Slide 115

Slide 115 text

RELATED WORK 115

Slide 116

Slide 116 text

DISTRIBUTED OZ RELATED WORK 116

Slide 117

Slide 117 text

DERFLOWL RELATED WORK 117

Slide 118

Slide 118 text

BLOOML RELATED WORK 118

Slide 119

Slide 119 text

LVARS RELATED WORK 119

Slide 120

Slide 120 text

D-STREAMS RELATED WORK 120

Slide 121

Slide 121 text

SUMMINGBIRD RELATED WORK 121

Slide 122

Slide 122 text

FUTURE WORK 122

Slide 123

Slide 123 text

INVARIANT PRESERVATION FUTURE WORK 123

Slide 124

Slide 124 text

CAUSAL+ CONSISTENCY FUTURE WORK 124

Slide 125

Slide 125 text

ORSWOT OPTIMIZATION FUTURE WORK 125

Slide 126

Slide 126 text

DELTA STATE-CRDTs FUTURE WORK 126

Slide 127

Slide 127 text

OPERATION-BASED CRDTs FUTURE WORK 127

Slide 128

Slide 128 text

DEFORESTATION FUTURE WORK 128

Slide 129

Slide 129 text

GITHUB.COM/CMEIKLEJOHN/LASP SOURCE 129

Slide 130

Slide 130 text

DERFLOW DISTRIBUTED DETERMINISTIC DATAFLOW PROGRAMMING FOR ERLANG ERLANG WORKSHOP 2014 130

Slide 131

Slide 131 text

LASP A LANGUAGE FOR DISTRIBUTED, EVENTUALLY CONSISTENT COMPUTATIONS WITH CRDTs PAPOC / EUROSYS 2015 131

Slide 132

Slide 132 text

NOVEMBER 4-6, 2015 SAN FRANCISCO, CA RICON 2015 132

Slide 133

Slide 133 text

SYNCFREE IS A EUROPEAN RESEARCH PROJECT TAKING PLACE FOR 3 YEARS, STARING OCTOBER 2013, AND IS FUNDED BY THE EUROPEAN UNION, GRANT AGREEMENT N°609551. SYNCFREE 133

Slide 134

Slide 134 text

Questions? Please remember to evaluate via the GOTO Guide App