State and the distributed data structures

In this talk I present the concept of CRDTs, data structures which allow to replicate the state in a distributed system with guaranteed eventual consistency.

Properties of CRDTs (particularly the state-based ones) ensure that despite of failing nodes and network partitions, the state of all replicas will eventually converge to the same value. I try to focus on high-level requirements and caveats rather than code examples and specific implementations, although you’ll find them here as well.

The talk was presented for the first time at ElixirConf EU 2018.

Transcript

1. STATE AND THE DISTRIBUTED DATA STRUCTURES
2. ARKADIUSZ GIL @arkgil @_arkgil
4. STATE
5. DISTRIBUTED STATE
6. MNESIA
7. REPLICATION
8. TRANSACTIONS
9. Guarantees
10. A B C
11. A B C write(k: 1)
12. A B C k: 1 k: 1
13. A B C k: 1 k: 1 k: 1
14. A B C
15. A B C write(k: 1)
16. A B C k: 1
17. A B C k: 1 k: 1
18. A B C k: 1 k: 1 k: 1
19. A B C
20. A B C write(k: 1) write(k: 1)
21. A B C k: 1 k: 1
22. A B C k: 1 k: 1 k: 1
23. A B C k: 1 k: 1 k: 1
24. A B C write(k: 1) write(k: 2)
25. A B C k: 2 k: 1 k: 1
26. A B C k: 2 k: 1 k: 1
27. MNESIA •built on BEAM tools •transactions (AP or CP) •has some flaws..
28. CAN WE DO BETTER?
29. CAN WE DO BETTER DIFFERENT?
30. CRDTs
31. CENTRAL REGIONAL DENTAL TESTING SERVICES 1 1 http://crdts.org
32. CONFLICT-FREE REPLICATED DATA TYPES
33. CRDT •a data type •a set of functions
34. Example type t() :: integer() @spec increment(t()) :: t() @spec decrement(t()) :: t()
35. Example type t() :: Set.t() @spec add(t(), term()) :: t() @spec remove(t(), term()) :: t()
36. EVENTUAL CONSISTENCY + OPTIMISTIC REPLICATION
37. State-based CRDTs
38. DATA TYPE
39. COMPARE
40. compare( , ) = true compare( , ) = false
41. partial order
42. compare( , ) = false compare( , ) = false
43. MERGE
44. merge( , ) =
45. compare( , ) = true compare( , ) = true
46. least upper bound
47. OBSERVATIONS!
48. compare( , ) = true merge( , ) =
49. merge( , ) =
50. join semilattice
51. Example
52. Example @type t() :: Set.t() def compare(s1, s2), do: ... def merge(s1, s2), do: ...
53. Example @type t() :: Set.t() def compare(s1, s2), do: Set.super_or_eq?(s1, s2) def merge(s1, s2), do: ...
54. Example @type t() :: Set.t() def compare(s1, s2), do: Set.super_or_eq?(s1, s2) def merge(s1, s2), do: Set.union(s1, s2)
55. Example merge({1, 2}, {3}) #=> {1, 2, 3} compare({1, 2, 3}, {1, 2}) #=> true compare({1, 2, 3}, {3}) #=> true
56. Protocol
57. A B C
58. A B C
59. A B C
60. A B C
61. A B C
62. A B C
63. A B C
64. merge( , ) =
65. FUNCTIONS
66. FUNCTIONS MUTATORS
67. MUTATORS @spec m(t(), [term()]) :: t()
68. MERGE, COMPARE =/= MUTATOR
69. Example
70. Example def add(s, el), do: Set.union(s, {el}) def remove(s, el), do: Set.subtract(s, {el})
71. {1,2} {1,2} remove(2)
72. {1} {1,2}
73. {1} {1,2} {1} {1,2}
74. {1,2} {1,2}
75. G-SET
76. users time
77. Problem 1 •"bigger" values dominate
78. compare( , ) = true merge( , ) =
79. Problem 1 •"bigger" values dominate •removing things "shrink"
80. compare( , ) = true m( ) =
81. merge( , ) =
82. Removes must inflate!
83. SOLUTION 1 TOMBSTONES
84. TOMBSTONES @type t :: {Set.t(), Set.t()} def add({s, t}, el) do {Set.union(s, {el}), t} end def remove({s, t}, el) do {s, Set.union(t, {el})} end
85. a = {{}, {}} |> add(1) #=> {{1}, {}} b = a |> add(2) #=> {{1, 2}, {}} c = b |> remove(1) #=> {{1, 2}, {1}}
86. a = {{}, {}} |> add(1) #=> {{1}, {}} b = a |> add(2) #=> {{1, 2}, {}} c = b |> remove(1) #=> {{1, 2}, {1}} compare(b, a) = true compare(c, b) = true
87. def compare({s1, t1}, {s2, t2}) do Set.super_or_eq?(s1, s2) and Set.super_or_eq?(t1, t2) end def merge({s1, t1}, {s2, t2}) do {Set.union(s1, s2), Set.union(t1, t2)} end
88. def element?({s, t}, el) do s |> Set.subtract(t) |> Set.element?(el) end
89. s = {{}, {}} element?(s, 1) #=> false element?(s, 2) #=> false
90. s = {{}, {}} |> add(1) #=> {{1}, {}} element?(s, 1) #=> true element?(s, 2) #=> false
91. s = {{}, {}} |> add(1) #=> {{1}, {}} |> add(2) #=> {{1, 2}, {}} element?(s, 1) #=> true element?(s, 2) #=> true
92. s = {{}, {}} |> add(1) #=> {{1}, {}} |> add(2) #=> {{1, 2}, {}} |> remove(1) #=> {{1, 2}, {1}} element?(s, 1) #=> false element?(s, 2) #=> true
93. s = {{}, {}} |> add(1) #=> {{1}, {}} |> add(2) #=> {{1, 2}, {}} |> remove(1) #=> {{1, 2}, {1}} |> add(1) #=> {{1, 2}, {1}} element?(s, 1) #=> false element?(s, 2) #=> true
94. elements can be removed, but never added again
95. 2P-SET
96. SOLUTION 2 CAUSAL CONTEXTS2 2 https://arxiv.org/pdf/1603.01529.pdf
97. Problem 2
98. Problem 2 whole state sent to each replica, multiple times
99. SOLUTION DELTA STATE REPLICATED DATA TYPES2 2 https://arxiv.org/pdf/1603.01529.pdf
100. MUTATOR @spec m(t(), [term()]) :: t() DELTA MUTATOR @spec d(t(), [term()]) :: t()
101. the rule m(state) = merge(state, d(state))
102. Example def add(s, el), do: union(s, {el}) def delta_add(s, el), do: {el}
103. WHY BOTHER?
104. CONFIDENCE
105. PURE FUNCTIONS & DATA STRUCTURES
106. UNDERSTANDING
107. RIAK
108. RIAK CASSANDRA
109. RIAK CASSANDRA DYNAMO
110. RIAK CASSANDRA DYNAMO ANTIDOTE
111. You can use them, too!
112. LASP
113. DECLARE VARIABLE :lasp.declare({"set", :state_gset}, :state_gset)
114. UPDATE VALUE :lasp.update({"set", :state_gset}, {:add, 1}, self())
115. READ VALUE :lasp.query({"set", :state_gset})
116. REFERENCES •A comprehensive study of Convergent and Commutative Replicated Data Types •Conflict-free Replicated Data Types •Delta State Replicated Data Types •Phoenix 1.2 and Beyond •Efficient State-based CRDTs by Delta-Mutation
117. Thank you! @arkgil @_arkgil