Riak -- Seattle Scalability

Presentation I about Riak/future-of-Riak at the Seattle Scalability meetup in August 2013. Note: slides are designed for presentation and miss context without me talking.

Transcript

1. Joseph Blomstedt (@jtuple) Basho Technologies Seattle Scalability Meetup August 2013
2. 2 Joe Blomstedt
3. 3 Senior Engineer
4. 4 joe@basho.com
5. 5 @jtuple
8. 8 Open Source
9. 9 Open Source Apache 2 github.com/basho
11. 11 Multi-DC Replication Support
12. Riak Basics 12
14. 14 Key / Value Secondary Indexes Full Text Search Map/Reduce Counters
15. 15 Key / Value Secondary Indexes Full Text Search Map/Reduce Counters
16. 16 PUT GET DELETE
17. 17 users/joe PUT
18. 18 users/joe GET
19. 19 users/joe DELETE
20. 20 Key / Value Secondary Indexes Full Text Search Map/Reduce Counters
21. 21 user/1234 PUT {first: “John”, last: “Doe”, age: 40}
22. 22 user/1234 PUT {first: “John”, last: “Doe”, age: 40} first_bin: “John” last_bin: “Doe” age_int: 40
23. 23 pics/pic42 PUT animal_bin: cat cuteness_int: 10
24. 24 Tagged Fields Exact/Range Single Index Streaming Pagination
25. 25 Key / Value Secondary Indexes Full Text Search Map/Reduce Counters
26. 26 Automatic Indexing JSON XML Text Custom
27. 27 Querying Exact Match Wildcard Range
28. 28 Querying Grouping Prefix matching Term Boosting
29. 29 Querying AND OR NOT
30. 30 Key / Value Secondary Indexes Full Text Search Map/Reduce Counters
31. 31 Use Cases Filtering Aggregation Extraction
32. 32 M/R Input Known Keys 2nd Index Query Full Text Query
33. 33 Distributed Mappers: data nodes Reducers: coordinator
34. 34 Key / Value Secondary Indexes Full Text Search Map/Reduce Counters
35. 35 Why do people use Riak?
36. Big Data 36
37. Big Data 37 Scalable Distributed
38. Big Data 38 5 nodes 100 nodes
39. Critical Data 39
40. Critical Data 40 Big Medium small
41. Fault Tolerance 41
42. Availability 42
43. CAP Theorem 43
44. 44 Partition-tolerance Consistency Availability
45. 45 Partition-tolerance Consistency Availability
46. 46 Partition-tolerance Consistency Availability CP AP
47. 47 Partition-tolerance Consistency Availability CP AP
48. 48 C/P Strict Quorum A/P Sloppy Quorum A/P
49. 49 C/P Strict Quorum A/P Sloppy Quorum A/P
50. 50 Node 1 Node 2 Node 3 Node 4 Node 5 client client client
51. 51 Node 1 Node 2 Node 3 Node 4 Node 5 client client client
52. 52 Node 1 Node 2 Node 3 Node 4 Node 5 client client client
53. 53 Node 1 Node 2 Node 3 Node 4 Node 5 client client client
54. 54 C/P Strict Quorum A/P Sloppy Quorum A/P
55. 55 Node 1 Node 2 Node 3 Node 4 Node 5 client client client
56. 56 Node 1 Node 2 Node 3 Node 4 Node 5 client client client
57. 57 Node 1 Node 2 Node 3 Node 4 Node 5 client client client
58. 58 Node 1 Node 2 Node 3 Node 4 Node 5 client client client
59. 59 C/P Strict Quorum A/P Sloppy Quorum A/P
60. 60 Node 1 Node 2 Node 3 Node 4 Node 5 client client client
61. 61 Node 1 Node 2 Node 3 Node 4 Node 5 client client client
62. 62 Node 1 Node 2 Node 3 Node 4 Node 5 client client client
63. 63 Node 1 Node 2 Node 3 Node 4 Node 5 client client client
64. 64 Node 1 Node 2 Node 3 Node 4 Node 5 client client client client client
65. Eventual Consistency 65
66. 66 A A A
67. 67 A A A
68. 68 A A A B
69. 69 A A A B
70. 70 A A A B B B B
71. 71 A A A
72. 72 A A A B C
73. 73 A A A B C
74. 74 A A A B {B,C} {B,C} {B,C} C
75. 75 Write Once Immutable Last Write Wins Business Rules CRDTs/Monotonicity
76. 76 Write Once Immutable Last Write Wins Business Rules CRDTs/Monotonicity
77. 77 Write Once Immutable Last Write Wins Business Rules CRDTs/Monotonicity
78. 78 Write Once Immutable Last Write Wins Business Rules CRDTs/Monotonicity
79. 79 Write Once Immutable Last Write Wins Business Rules CRDTs/Monotonicity
80. 80 Write Once Immutable Last Write Wins Business Rules CRDTs/Monotonicity
81. 81 Write Once Immutable Last Write Wins Business Rules “Set Union”
82. 82 {1} {1} {1}
83. 83 {1} {1} {1}
84. 84 {1} {1} {1} {1,2} {1,3} {1,2,4}
85. 85 {1} {1} {1} {1,2} {1,3} {1,2,4}
86. 86 {1} {1} {1} {1,2} {1,2,3,4} {1,2,3,4} {1,2,3,4} {1,3} {1,2,4}
87. Clustering 87
88. 88 Consistent Hashing Sloppy Quorums Hinted Handoff
89. 89 0 2160
90. 90 0 2160
92. 92 Node 2 Node 1 Node 3
93. 93 Node 2 Node 1 Node 3 Node 4
94. 94 Node 2 Node 1 Node 3 Node 4 SHA1(Key)
95. 95 Node 2 Node 1 Node 3 Node 4 SHA1(Key) primary replicas
96. 96 Node 2 Node 1 Node 3 Node 4 primary + fallback replicas
97. 97 Node 2 Node 1 Node 3 Node 4 handoff
98. Self Healing 98
99. 99 Here Today != Here Tomorrow
100. 100 Read Repair
101. 101 Active Anti Entropy
102. 102 Node 2 Node 1 Node 3 Node 4 AAE exchange
103. 103 Active Anti Entropy Realtime Persistent Rebuilt Weekly
104. Upcoming 104
105. 105 Data Types (CRDTs) Strong Consistency (C/P) Solr Integration
106. CRDTs 106
109. 109 Counters Sets Maps
110. 110 Counters Sets Maps
111. 111 1 1 1
112. 112 1 1 1
113. 113 1 1 1 2 4 +1 +3
114. 114 1 1 1 2 4 +1 +3
115. 115 1 1 1 2 {2,4} {2,4} {2,4} 4 +1 +3
116. 116 (a,1) (a,1) (a,1)
117. 117 (a,1) (a,1) (a,1)
118. 118 (a,1) +1 +3 (a,1) (a,1) (a,2) (a,1),(b,3)
119. 119 (a,1) +1 +3 (a,1) (a,1) (a,2) (a,1),(b,3)
120. 120 (a,1) +1 +3 (a,1) (a,1) (a,2) (a,1),(b,3) (a,2),(b,3) (a,2),(b,3) (a,1),(b,3)
121. 121 (a,1) +1 +3 (a,1) (a,1) (a,2) (a,1),(b,3) (a,2),(b,3) (a,2),(b,3) (a,1),(b,3) =5
122. 122 Counters Sets Maps
123. 123 Counters Sets Maps
124. Consistency 124
125. 125 A/P or C/P Per Bucket Choice
126. 126 Single Key Optimistic CAS Counters/Sets/Maps
127. 127 Multi Paxos Disjoint Ensembles Scalable
128. Node 2 Node 1 Node 3
129. 129 1 2 3 4 5 6 7 123
130. 130 1 2 3 4 5 6 7 123 234
131. 131 1 2 3 4 5 6 7 123 234 345
132. 132 1 2 3 4 5 6 7 123 234 345 456
133. 133 1 2 3 4 5 6 7 123 234 345 456 567...
134. 134 1 2 3 4 5 6 7 123 Ensembles 234 345 456 567...
135. 135 Ensemble Indepedent Paxos Group Leader Election Serializability
136. 136 write 2 1 1 1 write 2 leader
137. 137 write 2 2 2 2 write 2 leader 2 X
138. 138 3 3 3 write 3 leader 3
139. 139 3 3 3 leader
140. 140 +1 1 1 1 +1 leader
141. 141 2 2 2 +1 leader 2
142. 142 3 3 3 leader 3
143. 143 3 3 3 leader
144. Riak + Solr 144
145. 145 Next Generation Search Full Distributed Solr Embedded in Riak Active Anti Entropy
146. End 146
147. 147 We hiring!
150. ricon.io
151. Questions? 151