Consistency and Riak

Christopher Meiklejohn
Riak Meetup, Paris, February 2015

Transcript

1. Consistency and Riak Christopher Meiklejohn Riak Meetup, Paris, February 2015 @cmeik
2. History
3. Published SOSP 2007; key-value storage system Amazon Dynamo
4. Focused on high-availability and low-latency Amazon Dynamo
5. Collection of distributed systems techniques Amazon Dynamo
6. LinkedIn Voldemort, Facebook Cassandra Amazon Dynamo
7. Released 2009; Apache2 licensed Dynamo clone Basho Riak
8. Riak Architecture
9. Consistent Hashing hash(bucket/key)
10. hash ring
11. tokenize it
12. node 0 node 1 node 2 hash(key)
13. node 0 node 1 node 2 Replicas are stored to the N - 1 contiguous partitions
14. node 0 node 1 node 2 hash(companies/cisco) Replicas are stored to the N - 1 contiguous partitions
15. node 0 node 1 node 2 hash(companies/cisco) Replicas are stored to the N - 1 contiguous partitions
16. node 0 node 1 node 2
17. Scaling out node 0 node 1 node 2 node 3 +
18. Quorum requests N R W PR/PW DW
19. Vector Clocks establish temporality
22. Anatomy of a Request get(users/clay-davis)
23. Anatomy of a Request get(users/clay-davis) client Riak
24. Anatomy of a Request get(users/clay-davis) Get Handler (FSM) client Riak
25. Anatomy of a Request get(users/clay-davis) Get Handler (FSM) client Riak hash(users/clay-davis) == 10, 11, 12
26. Anatomy of a Request get(users/clay-davis) Get Handler (FSM) client Riak hash(users/clay-davis) == 10, 11, 12 Coordinating node Cluster 6 7 8 9 10 11 12 13 14 15 16 The Ring
27. Anatomy of a Request get(users/clay-davis) Get Handler (FSM) client Riak get(users/clay-davis) Coordinating node Cluster 6 7 8 9 10 11 12 13 14 15 16 The Ring
28. Anatomy of a Request get(users/clay-davis) Get Handler (FSM) client Riak Coordinating node Cluster 6 7 8 9 10 11 12 13 14 15 16 The Ring R=2
29. Anatomy of a Request get(users/clay-davis) Get Handler (FSM) client Riak Coordinating node Cluster 6 7 8 9 10 11 12 13 14 15 16 The Ring R=2 obj
30. Anatomy of a Request get(users/clay-davis) Get Handler (FSM) client Riak R=2 obj obj
31. Anatomy of a Request get(users/clay-davis) Get Handler (FSM) client Riak R=2 obj obj
32. Anatomy of a Request get(users/clay-davis) obj
33. Read Repair (Anti-Entropy)
34. replica replica replica
35. replica replica replica X
36. replica replica replica replica replica replica
37. Active Anti-Entropy (self healing clusters)
38. real-time updates persistent non-blocking disk-based
39. merkle tree to track changes coordinated at the vnode level runs as a background process exchange with neighbor vnodes for inconsistencies resolution semantics: trigger read-repair
40. = hashes marked dirty
45. = keys to read-repair
46. Riak and Consistency
47. Riak Object
48. BKey Value
49. Consistent hashing; dynamic membership Data Placement
53. Replication per-value across ring Data Placement
54. Replica Replica Replica
55. Take the form: {Writer, Value, Time} Concurrent writes
56. [{a, v1, t1}] [{b, v1, t2}] [{a, v1, t1}] Concurrent writes
57. [{a, v1, t1}] [{b, v1, t2}] [{a, v1, t1}] [{b, v1, t2}] [{b, v1, t2}] [{b, v1, t2}] Last Writer Wins
58. [{a, v1, t1}] [{b, v1, t2}] [{a, v1, t1}] [[{a, v1, t1}, {b, v1, t2}] [[{a, v1, t1}, {b, v1, t2}] [[{a, v1, t1}, {b, v1, t2}] Allow Mult
59. User specificed Merge
60. Two Approaches
61. Strong Eventual Consistency
62. Designed for convergence; allows divergence Conflict-free Replicated Data Types
63. Solves the Dynamo concurrency anomaly Conflict-free Replicated Data Types
64. The Theory
69. Two flavors: state-based and operation-based Conflict-free Replicated Data Types
70. Counters, Flags, Registers, Sets, Maps, Graphs Conflict-free Replicated Data Types
71. Broadcast update operation Operation-Based CRDTs
72. Commutative; relies on unique delivery Operation-Based CRDTs
73. Apply change locally; propagate entire state State-Based CRDTs
74. State is merged between replicas State-Based CRDTs
75. Set of all states form a bounded join-semilattice State-Based CRDTs
76. Partially ordered set; join operation Bounded Join-Semilattice
77. Associativity: (X · Y) · Z = X · (Y · Z) Bounded Join-Semilattice
78. Commutativity: X · Y = Y · X Bounded Join-Semilattice
79. Idempotence: X · X = X Bounded Join-Semilattice
80. Examples Bounded Join-Semilattice
81. b a c a, b a, c a, b, c Set; merge function: union. b, c
82. 3 5 7 5 7 7 Increasing natural; merge function: max.
83. F F T F T T Booleans; merge function: or.
84. x <= y montone f(x) <= f(y)
85. Examples State-Based Observed-Remove Set
86. [ [{1, a}], [] ] [ [{1, a}], [] ]
87. [ [{1, a}], [] ] [ [{1, a}], [] ] [ [{1, a}], [{1, a}] ] [ [{1, a}], [{1, a}] ]
88. [ [{1, a}], [] ] [ [{1, a}], [] ] [ [{1, a}], [{1, a}] ] [ [{1, a}], [{1, a}] ] [ [{1, a}, {2, a}], [{1, a}] ]
89. [ [{1, a}], [] ] [ [{1, a}], [] ] [ [{1, a}], [{1, a}] ] [ [{1, a}], [{1, a}] ] [ [{1, a}, {2, a}], [{1, a}] ] [ [{1, a}, {2, a}], [{1, a}] ]
90. [ [{1, a}], [] ] [ [{1, a}], [] ]
91. [ [{1, a}], [] ] [ [{1, a}], [] ] [ [{1, a}, {2, b}], [] ]
92. [ [{1, a}], [] ] [ [{1, a}], [] ] [ [{1, a}, {2, b}], [] ] [ [{1, a}], [{1, a}] ]
93. [ [{1, a}], [] ] [ [{1, a}], [] ] [ [{1, a}, {2, b}], [] ] [ [{1, a}], [{1, a}] ] [ [{1, a}, {2, b}], [{1, a}] ]
94. Strong Consistency
95. Provides atomicity and recency Strong Consistency
96. Prohibits partial writes Strong Consistency
97. A A A
98. A A A Val = B
99. A A A Val = B
100. B A A
101. B A A Get Operation with Read Repair
102. B A A Get Operation with Read Repair
103. B A A Get Operation with Read Repair B B
104. Single key atomic operations Strong Consistency
105. Requires read/modify/write cycle (CAS) Strong Consistency
106. Consensus
107. Distributed Consensus The problem of reaching agreement among remote processes is one of the most fundamental problems in distributed computing and is at the core of many algorithms for distributed data processing, distributed file management, and fault-tolerant distributed applications. Fischer, Lynch, Paterson
108. Termination, agreement, validity The Consensus Problem
109. All processes eventually decide on a value Termination
110. All processes decide on the same value Agreement
111. Value decided on had to have been proposed Validity
112. Consensus Algorithms
113. Paxos, ZAB, Raft, etc. Consensus Algorithms
114. Coordinated requests with a chosen leader The Paxos Algorithm
115. Node 1 Node 2 Node 3 N++ prepare(N) promise(N, Vb) promise(N, Vc) Vn = f(Va, Vb, Vc) commit(N, Vn) accept(N)
116. First request Multi-Paxos
117. Node 1 Node 2 Node 3 N++; I = 0 prepare(N, I) promise(N, I, Vb) promise(N, I, Vc) Vn = f(Va, Vb, Vc) commit(N, I, Vn) accept(N, I)
118. Each additional request Multi-Paxos
119. Node 1 Node 2 Node 3 I++ commit(N, I, V) accept(N, I)
120. Ship entire state! Multi-Paxos
121. Riak
122. Key-value store; keys are independent state Riak
123. Multi-Paxos per key; CAS on isolated state Riak
124. Consensus Groups
125. Participants in decisioning; ensembles Consensus Groups
126. Use the preference list! Consensus Groups
127. preflist
132. One ensemble per preference list; ring size Consensus Groups
133. Ensembles
134. election of leader; get/put operations Riak Ensembles
135. read local; refresh, if old Get Operations
136. Node 1 Node 2 Node 3 obj.epoch < epoch get(key) reply(Epochb, Seqb, Valb) Val = latest(Vala, Valb, Valc) Val.epoch = epoch write(Epoch, ++Seq, Val) ack(Epoch, Seq) reply(Epochc, Seqc, Valc)
137. Node 1 Node 2 Node 3 obj.epoch == epoch Reply = local_get(Key)
138. Worst Case: 2 roundtrips / write Get Operations Best Case: 0 roundtrips / write
139. read local; refresh, modify and commit if old Put Operations
140. Node 1 Node 2 Node 3 obj.epoch < epoch get(key) reply(Epochb, Seqb, Valb) Latest = latest(Vala, Valb, Valc) Val = modify(Latest) write(Epoch, ++Seq, Val) ack(Epoch, Seq) reply(Epochc, Seqc, Valc)
141. Node 1 Node 2 Node 3 obj.epoch == epoch Latest = local_get(Key) Val = modify(Latest) write(Epoch, ++Seq, Val) ack(Epoch, Seq)
142. Worst Case: 2 roundtrips / write Put Operations Best Case: 1 roundtrips / write
143. Elect a new leader; start a new epoch Failed Quorums
144. Cluster Membership
145. Use joint consensus from multi paxos Dynamic Membership
146. Existing Ensemble Joining Ensemble riak_01 riak_02 riak_03 riak_07 riak_08 riak_09 [{riak_01}, {riak_02}, {riak_03}] [{riak_07}, {riak_08}, {riak_09}]
147. Joint-Consensus Ensemble [{riak_01}, {riak_02}, {riak_03}, {riak_07}, {riak_08}, {riak_09}]
148. Joint-Consensus Ensemble [{riak_01}, {riak_02}, {riak_03}, {riak_07}, {riak_08}, {riak_09}]
149. New Ensemble riak_07 riak_08 riak_09 [{riak_07}, {riak_08}, {riak_09}]
150. Single-key linearizability; reduced availability Strong Consistency
151. $ riak-admin bucket-type create strongly_consistent \ ‘{"props":{"consistent":true}}' $ riak-admin bucket-type status strongly_consistent $ riak-admin bucket-type activate strongly_consistent Enable strong consistency; http://docs.basho.com/riak/latest/dev/advanced/strong-consistency/
152. Conflict-Free Replicated Data Types Strong Eventual Consistency
153. $ riak-admin bucket-type create maps \ '{"props":{"datatype":"map"}}' $ riak-admin bucket-type create sets \ '{"props":{"datatype":"set"}}' $ riak-admin bucket-type create counters \ ‘{“props":{"datatype":"counter"}}' $ riak-admin bucket-type status maps $ riak-admin bucket-type activate maps Create bucket type for data types; http://docs.basho.com/riak/latest/dev/using/data-types/
154. $ curl -XPOST http://localhost:10018/types/counters/buckets/counters/ datatypes/traffic_tickets \ -H "Content-Type: application/json" \ -d '{"increment": 1}’ $ curl http://localhost:10018/types/counters/buckets/counters/ datatypes/traffic_tickets Operate on counters; http://docs.basho.com/riak/latest/dev/using/data-types/
155. $ curl -XPOST http://localhost:10018/types/sets/buckets/travel/ datatypes/cities \ -H "Content-Type: application/json" \ -d '{"add_all":["Toronto", “Montreal"]}' $ curl -XPOST http://localhost:10018/types/sets/buckets/travel/ datatypes/cities \ -H "Content-Type: application/json" \ -d '{"remove": “Montreal"}' $ curl http://localhost:10018/types/sets/buckets/travel/datatypes/ cities Operate on sets; http://docs.basho.com/riak/latest/dev/using/data-types/
156. $ curl -XPOST http://localhost:10018/types/maps/buckets/customers/ datatypes/ahmed_info \ -H "Content-Type: application/json" \ -d ' { "update": { "first_name_register": "Ahmed", "phone_number_register": "5551234567" } }' $ curl -XPOST http://localhost:8098/types/maps/buckets/customers/ datatypes/ahmed_info \ -H "Content-Type: application/json" \ -d ' { "update": { "annika_info_map": { "update": { "interests_set": { "add": "tango dancing" } } } } } ' Operate on maps; http://docs.basho.com/riak/latest/dev/using/data-types/
157. Questions?