Coordination-Free Computations

Coordination-Free Computations

GOTO Chicago 2015
Chicago, IL

3e09fee7b359be847ed5fa48f524a3d3?s=128

Christopher Meiklejohn

May 12, 2015
Tweet

Transcript

  1. Coordination-Free Computations Christopher Meiklejohn

  2. DISTRIBUTED, EVENTUALLY CONSISTENT COMPUTATIONS LASP 2 CHRISTOPHER MEIKLEJOHN (BASHO TECHNOLOGIES,

    INC.) PETER VAN ROY (UNIVERSITÉ CATHOLIQUE DE LOUVAIN)
  3. LASP MOTIVATION 3

  4. EXPENSIVE / IMPRACTICAL SYNCHRONIZATION IS 4

  5. MOBILE GAMES: SHARED STATE BETWEEN CLIENTS CLIENTS GO OFFLINE 5

    http://www.rovio.com/en/news/blog/261/263-million-monthly-active-users-in-december/
  6. DISJOINT STATE AGGREGATED UPSTREAM CLIENTS GO OFFLINE “INTERNET OF THINGS”:

    6 Gubbi, Jayavardhana, et al. "Internet of Things (IoT): A vision, architectural elements, and future directions." Future Generation Computer Systems 29.7 (2013): 1645-1660.
  7. NO TOTAL ORDER: REPLICATED SHARED STATE WITH OFFLINE CLIENTS CLIENTS

    NEED TO MAKE PROGRESS 7 Gilbert, Seth, and Nancy Lynch. "Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services." ACM SIGACT News 33.2 (2002): 51-59.
  8. WALL CLOCKS: UNRELIABLE AT BEST NON-DETERMINISTIC IF USED IN COMPUTATIONS

    8 Corbett, James C., et al. "Spanner: Google’s globally distributed database." ACM Transactions on Computer Systems (TOCS) 31.3 (2013): 8.
  9. RECONCILED BY USER CONCURRENCY 9

  10. RA RB 1 3 2 ? ? set(1) set(2) set(3)

    10
  11. RA RB 1 3 2 ? ? set(1) set(2) set(3)

    11
  12. RA RB 1 3 2 ? ? set(1) set(2) set(3)

    12
  13. RA RB 1 3 2 ? ? set(1) set(2) set(3)

    13
  14. CRDTs 14

  15. DETERMINISTIC RESOLUTION CRDTs PROVIDE 15

  16. CRDTs: MAPS, SETS, COUNTERS, REGISTERS, GRAPHS DETERMINISTIC RESOLUTION 16

  17. “CORRECT REPLICAS THAT HAVE DELIVERED THE SAME UPDATES HAVE EQUIVALENT

    STATE” Shapiro, Marc, et al. "Conflict-free replicated data types." Stabilization, Safety, and Security of Distributed Systems. Springer Berlin Heidelberg, 2011. 386-400. MONOTONIC STRONG EVENTUAL CONSISTENCY CRDTs REALIZE 17
  18. ‘MAX’ REGISTER CRDTs EXAMPLE 18

  19. RA RB 1 3 2 3 3 set(1) set(2) set(3)

    max(2,3) max(2,3) 19
  20. ‘ORSET’ SET CRDTs EXAMPLE 20

  21. RA RB RC {1} (1, {a}, {}) {1} (1, {b},

    {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) 21
  22. RA RB RC {1} (1, {a}, {}) {1} (1, {b},

    {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) 22
  23. RA RB RC {1} (1, {a}, {}) {1} (1, {b},

    {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) 23
  24. RA RB RC {1} (1, {a}, {}) {1} (1, {b},

    {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) 24
  25. FRAGMENTS & PROGRAMS 25

  26. {1,2,3} {3,4,5} 26

  27. {1,2,3} {3,4,5} Intersection {3} 27

  28. {1,2,3} {3,4,5} Intersection {3} Map {9} 28

  29. {1,2,3} {3,4,5} Intersection {3} Map {9} 29

  30. {1,2,3} {3,4,5} Intersection {3} Map {9} 30

  31. {1,2,3} {3,4,5} Intersection {3} Map {9} 31

  32. {1,2,3} {3,4,5} Intersection {3} Map {9} {1,2,3} {3,4,5} Intersection {3}

    Map {9} 32
  33. {1,2,3} {3,4,5} Intersection {3} Map {9} {1,2,3} {3,4,5} Intersection {3}

    Map {9} 33
  34. {1,2,3} {3,4,5} Intersection {3} Map {9} {1,2,3} {3,4,5} Intersection {3}

    Map {9} 34
  35. NONTRIVIAL FUNCTION APPLICATION AND DATA COMPOSITION IS 35

  36. RA RB RC {1} (1, {a}, {}) {1} (1, {b},

    {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RC) {2} fun(X) -> 2 end {2} {} 36
  37. RA RC {1} (1, {a}, {}) {1} (1, {b}, {})

    {} (1, {b}, {b}) {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RC) {2} fun(X) -> 2 end {2} {} F(RA) {2} fun(X) -> 2 end {2} {2} {2} 37
  38. RA RC {1} (1, {a}, {}) {1} (1, {b}, {})

    {} (1, {b}, {b}) {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RC) {2} fun(X) -> 2 end {2} {} F(RA) {2} fun(X) -> 2 end {2} {2} {2} 38
  39. RA RC {1} (1, {a}, {}) {1} (1, {b}, {})

    {} (1, {b}, {b}) {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RC) {2} fun(X) -> 2 end {2} {} F(RA) {2} fun(X) -> 2 end {2} {2} {2} 39
  40. RA RC {1} (1, {a}, {}) {1} (1, {b}, {})

    {} (1, {b}, {b}) {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RC) {2} fun(X) -> 2 end {2} {} F(RA) {2} fun(X) -> 2 end {2} {2} {2} 40
  41. COMPOSITION: USER OBSERVABLE VALUE VS. STATE METADATA MAPPING IS NONTRIVIAL

    WITHOUT MAPPING METADATA; UNMERGABLE 41 Brown, Russell, et al. "Riak dt map: A composable, convergent replicated dictionary." Proceedings of the First Workshop on Principles and Practice of Eventual Consistency. ACM, 2014. Conway, Neil, et al. "Logic and lattices for distributed programming." Proceedings of the Third ACM Symposium on Cloud Computing. ACM, 2012. Meiklejohn, Christopher. "On the composability of the Riak DT map: expanding from embedded to multi-key structures." Proceedings of the First Workshop on Principles and Practice of Eventual Consistency. ACM, 2014.
  42. LASP LANGUAGE 42

  43. LASP: DISTRIBUTED RUNTIME IMPLEMENTED AS ERLANG LIBRARY USES RIAK-CORE 43

  44. LASP: CRDTS AS STREAMS OF STATE CHANGES CRDTS CONNECTED BY

    MONOTONIC PROCESSES MANY TO ONE MAPPING OF CRDTS 44
  45. PRIMITIVE OPERATIONS: MONOTONIC READ, UPDATE FUNCTIONAL: MAP, FILTER, FOLD SET-THEORETIC:

    PRODUCT, UNION, INTERSECTION LIFTED TO OPERATE OVER METADATA 45
  46. MAP EXAMPLE 46

  47. RA RB RC {1} (1, {a}, {}) {1} (1, {b},

    {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RB) {2} fun(X) -> 2 end {2} {} (2, {b}, {}) (2, {b}, {b}) {} (2, {a, b}, {b}) (2, {a, b}, {b}) F(RC) {2} fun(X) -> 2 end (2, {a, b}, {b}) {2} {} {2} (2, {b}, {}) (2, {b}, {b}) (2, {a, b}, {b}) F(RA) {2} fun(X) -> 2 end (2, {a, b}, {b}) {2} {2} {2} (2, {a}, {}) (2, {a, b}, {}) (2, {a, b}, {b}) 47
  48. RA RB RC {1} (1, {a}, {}) {1} (1, {b},

    {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RB) {2} fun(X) -> 2 end {2} {} (2, {b}, {}) (2, {b}, {b}) {} (2, {a, b}, {b}) (2, {a, b}, {b}) F(RC) {2} fun(X) -> 2 end (2, {a, b}, {b}) {2} {} {2} (2, {b}, {}) (2, {b}, {b}) (2, {a, b}, {b}) F(RA) {2} fun(X) -> 2 end (2, {a, b}, {b}) {2} {2} {2} (2, {a}, {}) (2, {a, b}, {}) (2, {a, b}, {b}) 48
  49. RA RB RC {1} (1, {a}, {}) {1} (1, {b},

    {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RB) {2} fun(X) -> 2 end {2} {} (2, {b}, {}) (2, {b}, {b}) {} (2, {a, b}, {b}) (2, {a, b}, {b}) F(RC) {2} fun(X) -> 2 end (2, {a, b}, {b}) {2} {} {2} (2, {b}, {}) (2, {b}, {b}) (2, {a, b}, {b}) F(RA) {2} fun(X) -> 2 end (2, {a, b}, {b}) {2} {2} {2} (2, {a}, {}) (2, {a, b}, {}) (2, {a, b}, {b}) 49
  50. RA RB RC {1} (1, {a}, {}) {1} (1, {b},

    {}) {} (1, {b}, {b}) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) add(1) add(1) remove(1) F(RB) {2} fun(X) -> 2 end {2} {} (2, {b}, {}) (2, {b}, {b}) {} (2, {a, b}, {b}) (2, {a, b}, {b}) F(RC) {2} fun(X) -> 2 end (2, {a, b}, {b}) {2} {} {2} (2, {b}, {}) (2, {b}, {b}) (2, {a, b}, {b}) F(RA) {2} fun(X) -> 2 end (2, {a, b}, {b}) {2} {2} {2} (2, {a}, {}) (2, {a, b}, {}) (2, {a, b}, {b}) 50
  51. ARCHITECTURE 51

  52. SHARED VARIABLE STORE LASP STORE 52

  53. LEVELDB, BITCASK, ETS LASP BACKENDS 53

  54. PROVIDED BY RIAK DT LASP CRDTs 54

  55. CENTRALIZED SEMANTICS LASP ARCHITECTURE 55

  56. STORE: SHARED VARIABLE STORE PROCESSESS SYNCHRONIZE ON VARIABLES 56

  57. DISTRIBUTED SEMANTICS LASP ARCHITECTURE 57

  58. STORE: REPLICATED, SHARDED VARIABLE STORE PROCESSESS SYNCHRONIZE ON VARIABLES 58

  59. REPLICATED: DISTRIBUTED WITH RIAK CORE QUORUM REQUESTS; ANTI-ENTROPY PROTOCOL 59

  60. HYBRID: DISTRIBUTE PROGRAMS; R/W WITH LOCAL STORE CENTRALIZED EXECUTION 60

  61. EXAMPLES 61

  62. %% Create initial set. {ok, S1} = lasp:declare(Type), %% Add

    elements to initial set and update. {ok, _} = lasp:update(S1, {add_all, [1,2,3]}, a), %% Create second set. {ok, S2} = lasp:declare(Type), %% Apply map. ok = lasp:map(S1, fun(X) -> X * 2 end, S2),
  63. %% Create initial set. {ok, S1} = lasp_core:declare(Type, Store), %%

    Add elements to initial set and update. {ok, _} = lasp_core:update(S1, {add_all, [1,2,3]}, a, Store), %% Create second set. {ok, S2} = lasp_core:declare(Type, Store), %% Apply map. ok = lasp_core:map(S1, fun(X) -> X * 2 end, S2, Store),
  64. AD COUNTER 64

  65. AD COUNTER: TRACKS AD IMPRESSIONS PUSHES ADVERTISEMENTS TO THE CLIENT

    DISABLES AD AT 50,000+ IMPRESSIONS CLIENTS DISPLAY ADS WHEN OFFLINE 65
  66. INFORMATION FLOW AD COUNTER 66

  67. Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter

    Riot Ad Counter Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 67
  68. Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter

    Riot Ad Counter Contracts Riot Ads Rovio Ads Produ Read 50,000 Remove Increment Union 68
  69. Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter

    Riot Ad Counter Contracts Ads Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT 69
  70. Ads Contracts Ads Contracts Ads With Contracts ds Ads Filter

    Product Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT 70
  71. Ads ntracts Ads Contracts Ads With Contracts Filter Product Read

    Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT 71
  72. Ads Contracts Ads With Contracts Filter Read Operation Maintained CRDT

    Maintained CRDT Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 72
  73. Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter

    Riot Ad Counter Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 73
  74. Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter

    Riot Ad Counter Contracts Ads Contracts Riot Ads Rovio Ads Product Read 50,000 Remove Increment Union Lasp Operation User-Maintained C Lasp-Maintained C 74
  75. Ads Contracts Ads Contracts Ads With Contracts Riot Ads Rovio

    Ads Filter Product Read 50,000 Remove Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 75
  76. INFORMATION FLOW: MONOTONIC METADATA TO PREVENT DUPLICATE PROPAGATION 76

  77. EXAMPLE CODE AD COUNTER 77

  78. %% @doc Client process; standard recursive looping server. client(Id, AdsWithContracts,

    PreviousValue) -> receive view_ad -> %% Get current ad list. {ok, {_, _, AdList0}} = lasp:read(AdsWithContracts, PreviousValue), AdList = riak_dt_orset:value(AdList0), case length(AdList) of 0 -> %% No advertisements left to display; ignore %% message. client(Id, AdsWithContracts, AdList0); _ -> %% Select a random advertisement from the list of %% active advertisements. {#ad{counter=Ad}, _} = lists:nth( random:uniform(length(AdList)), AdList), %% Increment it. {ok, _} = lasp:update(Ad, increment, Id), lager:info("Incremented ad counter: ~p", [Ad]), client(Id, AdsWithContracts, AdList0) end end.
  79. %% @doc Server functions for the advertisement counter. After 5

    views, %% disable the advertisement. %% server({#ad{counter=Counter}=Ad, _}, Ads) -> %% Blocking threshold read for 5 advertisement impressions. {ok, _} = lasp:read(Counter, 5), %% Remove the advertisement. {ok, _} = lasp:update(Ads, {remove, Ad}, Ad), lager:info("Removing ad: ~p", [Ad]).
  80. %% Generate a series of unique identifiers. RovioAdIds = lists:map(fun(_)

    -> druuid:v4() end, lists:seq(1, 10)), lager:info("Rovio Ad Identifiers are: ~p", [RovioAdIds]), TriforkAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Trifork Ad Identifiers are: ~p", [TriforkAdIds]), Ids = RovioAdIds ++ TriforkAdIds, lager:info("Ad Identifiers are: ~p", [Ids]), %% Generate Rovio's advertisements. {ok, RovioAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(RovioAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, RovioAdIds), %% Generate Trifork's advertisements. {ok, TriforkAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(TriforkAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, TriforkAdIds), %% Union ads. {ok, Ads} = lasp:declare(?SET), ok = lasp:union(RovioAds, TriforkAds, Ads), %% For each identifier, generate a contract. {ok, Contracts} = lasp:declare(?SET), lists:map(fun(Id) -> {ok, _} = lasp:update(Contracts, {add, #contract{id=Id}}, undefined) end, Ids), %% Compute the Cartesian product of both ads and contracts. {ok, AdsContracts} = lasp:declare(?SET), ok = lasp:product(Ads, Contracts, AdsContracts), %% Filter items by join on item it. {ok, AdsWithContracts} = lasp:declare(?SET), FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) -> Id1 =:= Id2 end, ok = lasp:filter(AdsContracts, FilterFun, AdsWithContracts), %% Launch a series of client processes, each of which is responsible %% for displaying a particular advertisement. %% Generate a OR-set for tracking clients. {ok, Clients} = lasp:declare(?SET), %% Each client takes the full list of ads when it starts, and reads %% from the variable store. lists:map(fun(Id) -> ClientPid = spawn_link(?MODULE, client, [Id, AdsWithContracts, undefined]), {ok, _} = lasp:update(Clients, {add, ClientPid}, undefined) end, lists:seq(1,5)), %% Launch a server process for each advertisement, which will block %% until the advertisement should be disabled. %% Create a OR-set for the server list. {ok, Servers} = lasp:declare(?SET), %% Get the current advertisement list. {ok, {_, _, AdList0}} = lasp:read(AdsWithContracts), AdList = riak_dt_orset:value(AdList0), %% For each advertisement, launch one server for tracking it's %% impressions and wait to disable. lists:map(fun(Ad) -> ServerPid = spawn_link(?MODULE, server, [Ad, Ads]), {ok, _} = lasp:update(Servers, {add, ServerPid}, undefined) end, AdList),
  81. %% Generate a series of unique identifiers. RovioAdIds = lists:map(fun(_)

    -> druuid:v4() end, lists:seq(1, 10)), lager:info("Rovio Ad Identifiers are: ~p", [RovioAdIds]), TriforkAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Trifork Ad Identifiers are: ~p", [TriforkAdIds]), Ids = RovioAdIds ++ TriforkAdIds, lager:info("Ad Identifiers are: ~p", [Ids]), %% Generate Rovio's advertisements. {ok, RovioAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(RovioAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, RovioAdIds), %% Generate Trifork's advertisements. {ok, TriforkAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(TriforkAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, TriforkAdIds), %% Union ads. {ok, Ads} = lasp:declare(?SET), ok = lasp:union(RovioAds, TriforkAds, Ads), %% For each identifier, generate a contract. {ok, Contracts} = lasp:declare(?SET), lists:map(fun(Id) -> {ok, _} = lasp:update(Contracts, {add, #contract{id=Id}}, undefined) end, Ids), %% Compute the Cartesian product of both ads and contracts. {ok, AdsContracts} = lasp:declare(?SET), ok = lasp:product(Ads, Contracts, AdsContracts), %% Filter items by join on item it. {ok, AdsWithContracts} = lasp:declare(?SET), FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) -> Id1 =:= Id2 end, ok = lasp:filter(AdsContracts, FilterFun, AdsWithContracts), %% Launch a series of client processes, each of which is responsible %% for displaying a particular advertisement. %% Generate a OR-set for tracking clients. {ok, Clients} = lasp:declare(?SET), %% Each client takes the full list of ads when it starts, and reads %% from the variable store. lists:map(fun(Id) -> ClientPid = spawn_link(?MODULE, client, [Id, AdsWithContracts, undefined]), {ok, _} = lasp:update(Clients, {add, ClientPid}, undefined) end, lists:seq(1,5)), %% Launch a server process for each advertisement, which will block %% until the advertisement should be disabled. %% Create a OR-set for the server list. {ok, Servers} = lasp:declare(?SET), %% Get the current advertisement list. {ok, {_, _, AdList0}} = lasp:read(AdsWithContracts), AdList = riak_dt_orset:value(AdList0), %% For each advertisement, launch one server for tracking it's %% impressions and wait to disable. lists:map(fun(Ad) -> ServerPid = spawn_link(?MODULE, server, [Ad, Ads]), {ok, _} = lasp:update(Servers, {add, ServerPid}, undefined) end, AdList),
  82. %% Generate a series of unique identifiers. RovioAdIds = lists:map(fun(_)

    -> druuid:v4() end, lists:seq(1, 10)), lager:info("Rovio Ad Identifiers are: ~p", [RovioAdIds]), TriforkAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Trifork Ad Identifiers are: ~p", [TriforkAdIds]), Ids = RovioAdIds ++ TriforkAdIds, lager:info("Ad Identifiers are: ~p", [Ids]), %% Generate Rovio's advertisements. {ok, RovioAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(RovioAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, RovioAdIds), %% Generate Trifork's advertisements. {ok, TriforkAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(TriforkAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, TriforkAdIds), %% Union ads. {ok, Ads} = lasp:declare(?SET), ok = lasp:union(RovioAds, TriforkAds, Ads), %% For each identifier, generate a contract. {ok, Contracts} = lasp:declare(?SET), lists:map(fun(Id) -> {ok, _} = lasp:update(Contracts, {add, #contract{id=Id}}, undefined) end, Ids), %% Compute the Cartesian product of both ads and contracts. {ok, AdsContracts} = lasp:declare(?SET), ok = lasp:product(Ads, Contracts, AdsContracts), %% Filter items by join on item it. {ok, AdsWithContracts} = lasp:declare(?SET), FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) -> Id1 =:= Id2 end, ok = lasp:filter(AdsContracts, FilterFun, AdsWithContracts), %% Launch a series of client processes, each of which is responsible %% for displaying a particular advertisement. %% Generate a OR-set for tracking clients. {ok, Clients} = lasp:declare(?SET), %% Each client takes the full list of ads when it starts, and reads %% from the variable store. lists:map(fun(Id) -> ClientPid = spawn_link(?MODULE, client, [Id, AdsWithContracts, undefined]), {ok, _} = lasp:update(Clients, {add, ClientPid}, undefined) end, lists:seq(1,5)), %% Launch a server process for each advertisement, which will block %% until the advertisement should be disabled. %% Create a OR-set for the server list. {ok, Servers} = lasp:declare(?SET), %% Get the current advertisement list. {ok, {_, _, AdList0}} = lasp:read(AdsWithContracts), AdList = riak_dt_orset:value(AdList0), %% For each advertisement, launch one server for tracking it's %% impressions and wait to disable. lists:map(fun(Ad) -> ServerPid = spawn_link(?MODULE, server, [Ad, Ads]), {ok, _} = lasp:update(Servers, {add, ServerPid}, undefined) end, AdList),
  83. %% Generate a series of unique identifiers. RovioAdIds = lists:map(fun(_)

    -> druuid:v4() end, lists:seq(1, 10)), lager:info("Rovio Ad Identifiers are: ~p", [RovioAdIds]), TriforkAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Trifork Ad Identifiers are: ~p", [TriforkAdIds]), Ids = RovioAdIds ++ TriforkAdIds, lager:info("Ad Identifiers are: ~p", [Ids]), %% Generate Rovio's advertisements. {ok, RovioAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(RovioAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, RovioAdIds), %% Generate Trifork's advertisements. {ok, TriforkAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(TriforkAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, TriforkAdIds), %% Union ads. {ok, Ads} = lasp:declare(?SET), ok = lasp:union(RovioAds, TriforkAds, Ads), %% For each identifier, generate a contract. {ok, Contracts} = lasp:declare(?SET), lists:map(fun(Id) -> {ok, _} = lasp:update(Contracts, {add, #contract{id=Id}}, undefined) end, Ids), %% Compute the Cartesian product of both ads and contracts. {ok, AdsContracts} = lasp:declare(?SET), ok = lasp:product(Ads, Contracts, AdsContracts), %% Filter items by join on item it. {ok, AdsWithContracts} = lasp:declare(?SET), FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) -> Id1 =:= Id2 end, ok = lasp:filter(AdsContracts, FilterFun, AdsWithContracts), %% Launch a series of client processes, each of which is responsible %% for displaying a particular advertisement. %% Generate a OR-set for tracking clients. {ok, Clients} = lasp:declare(?SET), %% Each client takes the full list of ads when it starts, and reads %% from the variable store. lists:map(fun(Id) -> ClientPid = spawn_link(?MODULE, client, [Id, AdsWithContracts, undefined]), {ok, _} = lasp:update(Clients, {add, ClientPid}, undefined) end, lists:seq(1,5)), %% Launch a server process for each advertisement, which will block %% until the advertisement should be disabled. %% Create a OR-set for the server list. {ok, Servers} = lasp:declare(?SET), %% Get the current advertisement list. {ok, {_, _, AdList0}} = lasp:read(AdsWithContracts), AdList = riak_dt_orset:value(AdList0), %% For each advertisement, launch one server for tracking it's %% impressions and wait to disable. lists:map(fun(Ad) -> ServerPid = spawn_link(?MODULE, server, [Ad, Ads]), {ok, _} = lasp:update(Servers, {add, ServerPid}, undefined) end, AdList),
  84. %% Generate a series of unique identifiers. RovioAdIds = lists:map(fun(_)

    -> druuid:v4() end, lists:seq(1, 10)), lager:info("Rovio Ad Identifiers are: ~p", [RovioAdIds]), TriforkAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("Trifork Ad Identifiers are: ~p", [TriforkAdIds]), Ids = RovioAdIds ++ TriforkAdIds, lager:info("Ad Identifiers are: ~p", [Ids]), %% Generate Rovio's advertisements. {ok, RovioAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(RovioAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, RovioAdIds), %% Generate Trifork's advertisements. {ok, TriforkAds} = lasp:declare(?SET), lists:map(fun(Id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?COUNTER), %% Add it to the advertisement set. {ok, _} = lasp:update(TriforkAds, {add, #ad{id=Id, counter=CounterId}}, undefined) end, TriforkAdIds), %% Union ads. {ok, Ads} = lasp:declare(?SET), ok = lasp:union(RovioAds, TriforkAds, Ads), %% For each identifier, generate a contract. {ok, Contracts} = lasp:declare(?SET), lists:map(fun(Id) -> {ok, _} = lasp:update(Contracts, {add, #contract{id=Id}}, undefined) end, Ids), %% Compute the Cartesian product of both ads and contracts. {ok, AdsContracts} = lasp:declare(?SET), ok = lasp:product(Ads, Contracts, AdsContracts), %% Filter items by join on item it. {ok, AdsWithContracts} = lasp:declare(?SET), FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) -> Id1 =:= Id2 end, ok = lasp:filter(AdsContracts, FilterFun, AdsWithContracts), %% Launch a series of client processes, each of which is responsible %% for displaying a particular advertisement. %% Generate a OR-set for tracking clients. {ok, Clients} = lasp:declare(?SET), %% Each client takes the full list of ads when it starts, and reads %% from the variable store. lists:map(fun(Id) -> ClientPid = spawn_link(?MODULE, client, [Id, AdsWithContracts, undefined]), {ok, _} = lasp:update(Clients, {add, ClientPid}, undefined) end, lists:seq(1,5)), %% Launch a server process for each advertisement, which will block %% until the advertisement should be disabled. %% Create a OR-set for the server list. {ok, Servers} = lasp:declare(?SET), %% Get the current advertisement list. {ok, {_, _, AdList0}} = lasp:read(AdsWithContracts), AdList = riak_dt_orset:value(AdList0), %% For each advertisement, launch one server for tracking it's %% impressions and wait to disable. lists:map(fun(Ad) -> ServerPid = spawn_link(?MODULE, server, [Ad, Ads]), {ok, _} = lasp:update(Servers, {add, ServerPid}, undefined) end, AdList),
  85. DISTRIBUTION AD COUNTER 85

  86. Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter

    Riot Ad Counter Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 86
  87. Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter

    Riot Ad Counter Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 87
  88. Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter

    Riot Ad Counter Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 88
  89. Ads Rovio Ad Counter Rovio Ad Counter Riot Ad Counter

    Riot Ad Counter Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 Counter1 Counter2 Counter3 89
  90. DISTRIBUTION BOUNDARIES: ARBITRARY ALLOWS COMPOSITION OF ENTIRE SYSTEM 90

  91. MATERIALIZED VIEWS 91

  92. MATERIALIZED VIEWS: INCREMENTALLY UPDATING, DELTA PROPAGATION SCHEMA BASED OR DYNAMIC;

    2I IN RIAK 92
  93. DATABASE AS STREAM OF UPDATES 93

  94. Riak K1 K2 K3 K1 K2 94

  95. Riak K1 K2 K3 K1 K2 RS1 RS2 RS3 95

  96. Riak K1 K2 K3 K1 K2 RS1 RS2 RS3 K1

    K1 K2 K2 K3 96
  97. Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1

    K2 K2 K3 N1 N2 N3 97
  98. COMPOSE PROGRAMS 98

  99. Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1

    K2 K2 K3 N1 N2 N3 Lasp Stream Processor Template 99
  100. Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1

    K2 K2 K3 N1 N2 N3 Lasp Stream Processor Template Lasp All Objects Program 100
  101. Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1

    K2 K2 K3 N1 N2 N3 Lasp Stream Processor Template Lasp All Objects Program Lasp “Over 65” Filter 101
  102. Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1

    K2 K2 K3 N1 N2 N3 Lasp Stream Processor Template Lasp All Objects Program Lasp “Over 65” Filter Lasp “Over 80” Filter 102
  103. Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1

    K2 K2 K3 N1 N2 N3 Lasp Stream Processor Template Lasp All Objects Program Lasp “Over 65” Filter Lasp “Over 80” Filter Lasp “Named Chris” Filter (L) 103
  104. DISTRIBUTE APPLICATIONS 104

  105. Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1

    K2 K2 K3 N1 N2 N3 Lasp “Named Chris” Filter (L) L’’’ L’’ L’ 105
  106. MERGE RESULTS (SEC) 106 N1 N2 N3 L’ L’ L’

  107. SUM RESULTS 107 RS1 L’ RS2 L’’

  108. Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1

    K2 K2 K3 N1 N2 N3 Execution of Lasp “Named Chris” Filter (L) L’’’ L’’ L’ 108
  109. CACHE: CACHE RESULTS OF MERGE SPECULATIVELY EXECUTE BASED ON DIVERGENCE

    109
  110. “INTERNET OF THINGS” 110

  111. Riak K1 K2 K3 K1 K2 RS2 RS3 K1 K1

    K2 K2 K3 N1 N2 N3 Execution of Lasp “Named Chris” Filter (L) L’’’ L’’ L’ 111
  112. IoT Network K1 K2 K3 K1 K2 S1 S2 S3

    U1 U1 U2 U2 U3 Execution of Lasp Temperature Sensor > 90 F L’’’ L’’ L’ 112
  113. IoT Network S1 S2 S3 U1 U1 U2 U2 U3

    Execution of Lasp Temperature Sensor > 90 F L’’’ L’’ L’ 113
  114. IOT: EXECUTE AT THE EDGE WRITE CODE THINKING ABOUT “ALL”

    DATA 114
  115. RELATED WORK 115

  116. DISTRIBUTED OZ RELATED WORK 116

  117. DERFLOWL RELATED WORK 117

  118. BLOOML RELATED WORK 118

  119. LVARS RELATED WORK 119

  120. D-STREAMS RELATED WORK 120

  121. SUMMINGBIRD RELATED WORK 121

  122. FUTURE WORK 122

  123. INVARIANT PRESERVATION FUTURE WORK 123

  124. CAUSAL+ CONSISTENCY FUTURE WORK 124

  125. ORSWOT OPTIMIZATION FUTURE WORK 125

  126. DELTA STATE-CRDTs FUTURE WORK 126

  127. OPERATION-BASED CRDTs FUTURE WORK 127

  128. DEFORESTATION FUTURE WORK 128

  129. GITHUB.COM/CMEIKLEJOHN/LASP SOURCE 129

  130. DERFLOW DISTRIBUTED DETERMINISTIC DATAFLOW PROGRAMMING FOR ERLANG ERLANG WORKSHOP 2014

    130
  131. LASP A LANGUAGE FOR DISTRIBUTED, EVENTUALLY CONSISTENT COMPUTATIONS WITH CRDTs

    PAPOC / EUROSYS 2015 131
  132. NOVEMBER 4-6, 2015 SAN FRANCISCO, CA RICON 2015 132

  133. SYNCFREE IS A EUROPEAN RESEARCH PROJECT TAKING PLACE FOR 3

    YEARS, STARING OCTOBER 2013, AND IS FUNDED BY THE EUROPEAN UNION, GRANT AGREEMENT N°609551. SYNCFREE 133
  134. Questions? Please remember to evaluate via the GOTO Guide App