Consistency and Riak

Transcript

1. 1.

   Consistency and Riak Christopher Meiklejohn Riak Meetup, Paris, February 2015

   @cmeik
2. 2.

   History

3. 3.

   Published SOSP 2007; key-value storage system Amazon Dynamo

4. 4.

   Focused on high-availability and low-latency Amazon Dynamo

5. 5.

   Collection of distributed systems techniques Amazon Dynamo

6. 6.

   LinkedIn Voldemort, Facebook Cassandra Amazon Dynamo

7. 7.

   Released 2009; Apache2 licensed Dynamo clone Basho Riak

8. 8.

   Riak Architecture

9. 9.

   Consistent Hashing hash(bucket/key)

10. 10.

    hash ring

11. 11.

    tokenize it

12. 12.

    node 0 node 1 node 2 hash(key)

13. 13.

    node 0 node 1 node 2 Replicas are stored to

    the N - 1 contiguous partitions
14. 14.

    node 0 node 1 node 2 hash(companies/cisco) Replicas are stored

    to the N - 1 contiguous partitions
15. 15.

    node 0 node 1 node 2 hash(companies/cisco) Replicas are stored

    to the N - 1 contiguous partitions
16. 16.

    node 0 node 1 node 2

17. 17.

    Scaling out node 0 node 1 node 2 node 3

    +
18. 18.

    Quorum requests N R W PR/PW DW

19. 19.

    Vector Clocks establish temporality

20. 20.

    None
21. 21.

    None
22. 22.

    Anatomy of a Request get(users/clay-davis)

23. 23.

    Anatomy of a Request get(users/clay-davis) client Riak

24. 24.

    Anatomy of a Request get(users/clay-davis) Get Handler (FSM) client Riak

25. 25.

    Anatomy of a Request get(users/clay-davis) Get Handler (FSM) client Riak

    hash(users/clay-davis) == 10, 11, 12
26. 26.

    Anatomy of a Request get(users/clay-davis) Get Handler (FSM) client Riak

    hash(users/clay-davis) == 10, 11, 12 Coordinating node Cluster 6 7 8 9 10 11 12 13 14 15 16 The Ring
27. 27.

    Anatomy of a Request get(users/clay-davis) Get Handler (FSM) client Riak

    get(users/clay-davis) Coordinating node Cluster 6 7 8 9 10 11 12 13 14 15 16 The Ring
28. 28.

    Anatomy of a Request get(users/clay-davis) Get Handler (FSM) client Riak

    Coordinating node Cluster 6 7 8 9 10 11 12 13 14 15 16 The Ring R=2
29. 29.

    Anatomy of a Request get(users/clay-davis) Get Handler (FSM) client Riak

    Coordinating node Cluster 6 7 8 9 10 11 12 13 14 15 16 The Ring R=2 obj
30. 30.

    Anatomy of a Request get(users/clay-davis) Get Handler (FSM) client Riak

    R=2 obj obj
31. 31.

    Anatomy of a Request get(users/clay-davis) Get Handler (FSM) client Riak

    R=2 obj obj
32. 32.

    Anatomy of a Request get(users/clay-davis) obj

33. 33.

    Read Repair (Anti-Entropy)

34. 34.

    replica replica replica

35. 35.

    replica replica replica X

36. 36.

    replica replica replica replica replica replica

37. 37.

    Active Anti-Entropy (self healing clusters)

38. 38.

    real-time updates persistent non-blocking disk-based

39. 39.

    merkle tree to track changes coordinated at the vnode level

    runs as a background process exchange with neighbor vnodes for inconsistencies resolution semantics: trigger read-repair
40. 40.

    = hashes marked dirty

41. 41.

    None
42. 42.

    None
43. 43.

    None
44. 44.

    None
45. 45.

    = keys to read-repair

46. 46.

    Riak and Consistency

47. 47.

    Riak Object

48. 48.

    BKey Value

49. 49.

    Consistent hashing; dynamic membership Data Placement

50. 50.

    None
51. 51.

    None
52. 52.

    None
53. 53.

    Replication per-value across ring Data Placement

54. 54.

    Replica Replica Replica

55. 55.

    Take the form: {Writer, Value, Time} Concurrent writes

56. 56.

    [{a, v1, t1}] [{b, v1, t2}] [{a, v1, t1}] Concurrent

    writes
57. 57.

    [{a, v1, t1}] [{b, v1, t2}] [{a, v1, t1}] [{b,

    v1, t2}] [{b, v1, t2}] [{b, v1, t2}] Last Writer Wins
58. 58.

    [{a, v1, t1}] [{b, v1, t2}] [{a, v1, t1}] [[{a,

    v1, t1}, {b, v1, t2}] [[{a, v1, t1}, {b, v1, t2}] [[{a, v1, t1}, {b, v1, t2}] Allow Mult
59. 59.

    User specificed Merge

60. 60.

    Two Approaches

61. 61.

    Strong Eventual Consistency

62. 62.

    Designed for convergence; allows divergence Conflict-free Replicated Data Types

63. 63.

    Solves the Dynamo concurrency anomaly Conflict-free Replicated Data Types

64. 64.

    The Theory

65. 65.

    None
66. 66.

    None
67. 67.

    None
68. 68.

    None
69. 69.

    Two flavors: state-based and operation-based Conflict-free Replicated Data Types

70. 70.

    Counters, Flags, Registers, Sets, Maps, Graphs Conflict-free Replicated Data Types

71. 71.

    Broadcast update operation Operation-Based CRDTs

72. 72.

    Commutative; relies on unique delivery Operation-Based CRDTs

73. 73.

    Apply change locally; propagate entire state State-Based CRDTs

74. 74.

    State is merged between replicas State-Based CRDTs

75. 75.

    Set of all states form a bounded join-semilattice State-Based CRDTs

76. 76.

    Partially ordered set; join operation Bounded Join-Semilattice

77. 77.

    Associativity: (X · Y) · Z = X · (Y

    · Z) Bounded Join-Semilattice
78. 78.

    Commutativity: X · Y = Y · X Bounded Join-Semilattice

79. 79.

    Idempotence: X · X = X Bounded Join-Semilattice

80. 80.

    Examples Bounded Join-Semilattice

81. 81.

    b a c a, b a, c a, b, c

    Set; merge function: union. b, c
82. 82.

    3 5 7 5 7 7 Increasing natural; merge function:

    max.
83. 83.

    F F T F T T Booleans; merge function: or.

84. 84.

    x <= y montone f(x) <= f(y)

85. 85.

    Examples State-Based Observed-Remove Set

86. 86.

    [ [{1, a}], [] ] [ [{1, a}], [] ]

87. 87.

    [ [{1, a}], [] ] [ [{1, a}], [] ]

    [ [{1, a}], [{1, a}] ] [ [{1, a}], [{1, a}] ]
88. 88.

    [ [{1, a}], [] ] [ [{1, a}], [] ]

    [ [{1, a}], [{1, a}] ] [ [{1, a}], [{1, a}] ] [ [{1, a}, {2, a}], [{1, a}] ]
89. 89.

    [ [{1, a}], [] ] [ [{1, a}], [] ]

    [ [{1, a}], [{1, a}] ] [ [{1, a}], [{1, a}] ] [ [{1, a}, {2, a}], [{1, a}] ] [ [{1, a}, {2, a}], [{1, a}] ]
90. 90.

    [ [{1, a}], [] ] [ [{1, a}], [] ]

91. 91.

    [ [{1, a}], [] ] [ [{1, a}], [] ]

    [ [{1, a}, {2, b}], [] ]
92. 92.

    [ [{1, a}], [] ] [ [{1, a}], [] ]

    [ [{1, a}, {2, b}], [] ] [ [{1, a}], [{1, a}] ]
93. 93.

    [ [{1, a}], [] ] [ [{1, a}], [] ]

    [ [{1, a}, {2, b}], [] ] [ [{1, a}], [{1, a}] ] [ [{1, a}, {2, b}], [{1, a}] ]
94. 94.

    Strong Consistency

95. 95.

    Provides atomicity and recency Strong Consistency

96. 96.

    Prohibits partial writes Strong Consistency

97. 97.

    A A A

98. 98.

    A A A Val = B

99. 99.

    A A A Val = B

100. 100.

     B A A

101. 101.

     B A A Get Operation with Read Repair

102. 102.

     B A A Get Operation with Read Repair

103. 103.

     B A A Get Operation with Read Repair B B

104. 104.

     Single key atomic operations Strong Consistency

105. 105.

     Requires read/modify/write cycle (CAS) Strong Consistency

106. 106.

     Consensus

107. 107.

     Distributed Consensus The problem of reaching agreement among remote processes

     is one of the most fundamental problems in distributed computing and is at the core of many algorithms for distributed data processing, distributed file management, and fault-tolerant distributed applications. Fischer, Lynch, Paterson
108. 108.

     Termination, agreement, validity The Consensus Problem

109. 109.

     All processes eventually decide on a value Termination

110. 110.

     All processes decide on the same value Agreement

111. 111.

     Value decided on had to have been proposed Validity

112. 112.

     Consensus Algorithms

113. 113.

     Paxos, ZAB, Raft, etc. Consensus Algorithms

114. 114.

     Coordinated requests with a chosen leader The Paxos Algorithm

115. 115.

     Node 1 Node 2 Node 3 N++ prepare(N) promise(N, Vb)

     promise(N, Vc) Vn = f(Va, Vb, Vc) commit(N, Vn) accept(N)
116. 116.

     First request Multi-Paxos

117. 117.

     Node 1 Node 2 Node 3 N++; I = 0

     prepare(N, I) promise(N, I, Vb) promise(N, I, Vc) Vn = f(Va, Vb, Vc) commit(N, I, Vn) accept(N, I)
118. 118.

     Each additional request Multi-Paxos

119. 119.

     Node 1 Node 2 Node 3 I++ commit(N, I, V)

     accept(N, I)
120. 120.

     Ship entire state! Multi-Paxos

121. 121.

     Riak

122. 122.

     Key-value store; keys are independent state Riak

123. 123.

     Multi-Paxos per key; CAS on isolated state Riak

124. 124.

     Consensus Groups

125. 125.

     Participants in decisioning; ensembles Consensus Groups

126. 126.

     Use the preference list! Consensus Groups

127. 127.

     preflist

128. 128.

     None
129. 129.

     None
130. 130.

     None
131. 131.

     None
132. 132.

     One ensemble per preference list; ring size Consensus Groups

133. 133.

     Ensembles

134. 134.

     election of leader; get/put operations Riak Ensembles

135. 135.

     read local; refresh, if old Get Operations

136. 136.

     Node 1 Node 2 Node 3 obj.epoch < epoch get(key)

     reply(Epochb, Seqb, Valb) Val = latest(Vala, Valb, Valc) Val.epoch = epoch write(Epoch, ++Seq, Val) ack(Epoch, Seq) reply(Epochc, Seqc, Valc)
137. 137.

     Node 1 Node 2 Node 3 obj.epoch == epoch Reply

     = local_get(Key)
138. 138.

     Worst Case: 2 roundtrips / write Get Operations Best Case:

     0 roundtrips / write
139. 139.

     read local; refresh, modify and commit if old Put Operations

140. 140.

     Node 1 Node 2 Node 3 obj.epoch < epoch get(key)

     reply(Epochb, Seqb, Valb) Latest = latest(Vala, Valb, Valc) Val = modify(Latest) write(Epoch, ++Seq, Val) ack(Epoch, Seq) reply(Epochc, Seqc, Valc)
141. 141.

     Node 1 Node 2 Node 3 obj.epoch == epoch Latest

     = local_get(Key) Val = modify(Latest) write(Epoch, ++Seq, Val) ack(Epoch, Seq)
142. 142.

     Worst Case: 2 roundtrips / write Put Operations Best Case:

     1 roundtrips / write
143. 143.

     Elect a new leader; start a new epoch Failed Quorums

144. 144.

     Cluster Membership

145. 145.

     Use joint consensus from multi paxos Dynamic Membership

146. 146.

     Existing Ensemble Joining Ensemble riak_01 riak_02 riak_03 riak_07 riak_08 riak_09

     [{riak_01}, {riak_02}, {riak_03}] [{riak_07}, {riak_08}, {riak_09}]
147. 147.

     Joint-Consensus Ensemble [{riak_01}, {riak_02}, {riak_03}, {riak_07}, {riak_08}, {riak_09}]

148. 148.

     Joint-Consensus Ensemble [{riak_01}, {riak_02}, {riak_03}, {riak_07}, {riak_08}, {riak_09}]

149. 149.

     New Ensemble riak_07 riak_08 riak_09 [{riak_07}, {riak_08}, {riak_09}]

150. 150.

     Single-key linearizability; reduced availability Strong Consistency

151. 151.

     $ riak-admin bucket-type create strongly_consistent \ ‘{"props":{"consistent":true}}' $ riak-admin bucket-type

     status strongly_consistent $ riak-admin bucket-type activate strongly_consistent Enable strong consistency; http://docs.basho.com/riak/latest/dev/advanced/strong-consistency/
152. 152.

     Conflict-Free Replicated Data Types Strong Eventual Consistency

153. 153.

     $ riak-admin bucket-type create maps \ '{"props":{"datatype":"map"}}' $ riak-admin bucket-type

     create sets \ '{"props":{"datatype":"set"}}' $ riak-admin bucket-type create counters \ ‘{“props":{"datatype":"counter"}}' $ riak-admin bucket-type status maps $ riak-admin bucket-type activate maps Create bucket type for data types; http://docs.basho.com/riak/latest/dev/using/data-types/
154. 154.

     $ curl -XPOST http://localhost:10018/types/counters/buckets/counters/ datatypes/traffic_tickets \ -H "Content-Type: application/json" \

     -d '{"increment": 1}’ $ curl http://localhost:10018/types/counters/buckets/counters/ datatypes/traffic_tickets Operate on counters; http://docs.basho.com/riak/latest/dev/using/data-types/
155. 155.

     $ curl -XPOST http://localhost:10018/types/sets/buckets/travel/ datatypes/cities \ -H "Content-Type: application/json" \

     -d '{"add_all":["Toronto", “Montreal"]}' $ curl -XPOST http://localhost:10018/types/sets/buckets/travel/ datatypes/cities \ -H "Content-Type: application/json" \ -d '{"remove": “Montreal"}' $ curl http://localhost:10018/types/sets/buckets/travel/datatypes/ cities Operate on sets; http://docs.basho.com/riak/latest/dev/using/data-types/
156. 156.

     $ curl -XPOST http://localhost:10018/types/maps/buckets/customers/ datatypes/ahmed_info \ -H "Content-Type: application/json" \

     -d ' { "update": { "first_name_register": "Ahmed", "phone_number_register": "5551234567" } }' $ curl -XPOST http://localhost:8098/types/maps/buckets/customers/ datatypes/ahmed_info \ -H "Content-Type: application/json" \ -d ' { "update": { "annika_info_map": { "update": { "interests_set": { "add": "tango dancing" } } } } } ' Operate on maps; http://docs.basho.com/riak/latest/dev/using/data-types/
157. 157.

     Questions?