Meet Riak with Justin Pease

Riak is an open source, scalable, fault-tolerant distributed database.

In this fast-paced introduction to Riak, Basho Director of Client Services Justin Pease discusses what Riak offers, some of the concepts upon which it is built, and why it should possibly be of interest to you.

Many of the concepts covered are broadly applicable to any distributed software.

Transcript

1. basho Meet Riak: A Distributed Database.
2. basho
3. basho So what is Riak?
4. basho Riak is an open source, highly scalable, fault-tolerant distributed database.
5. basho Riak is an open source, highly scalable, fault-tolerant distributed database. What does that mean?
6. basho Riak is an open source, highly scalable, fault-tolerant distributed database. Why might you care?
7. basho First, a little foundational information.
8. basho Dynamo
9. basho Riak is based upon Amazon’s Dynamo paper. http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html
10. basho Dynamo Inspired: Masterless, peer-coordinated replication
11. basho Dynamo Inspired: Consistent Hashing
12. basho Dynamo Inspired: Quorum reads & writes
13. basho Dynamo Inspired: Anti-entropy: read-repair, hinted handoff
14. basho Dynamo* powers S3 and the Amazon shopping cart, among other things. *Not to be confused with DynamoDB
15. basho Key-Value Store
16. basho Anyone use a key-value store?
17. basho At its core Riak is a key-value store.
18. basho A key-value store is like a big hashtable datastore = Hash.new => {}
19. basho You associate your data with a meaningful key, and store them together in the database.
20. basho Put datastore[:foo] = :bar => {:foo=>:bar}
21. basho Requests for the key will return the associated data.
22. basho Get datastore[:foo] => :bar
23. basho You can remove data.
24. basho Delete datastore.delete(:foo) => {}
25. basho Simple.
26. basho The data can be a binary: audio, video, images.
27. basho The data can be structured text: json, xml, etc.
28. basho The stored object also have structured metadata. (Vector Clock, Content-Type, Links, custom content)
29. basho Riak does provide extras on top of K/V. (MapReduce, Links, Full-text Search, Secondary Indices)
30. basho CAP Theorem
31. basho How many are familiar with the CAP Theorem?
32. basho CAP is not a feature.
33. basho CAP is not an option to include or exclude.
34. basho The CAP theorem is a description of characteristics inherent to all distributed systems.
35. basho In 2000, Dr. Eric Brewer* suggested that in a distributed system there is a tension between these characteristics: Consistency Availability Partition Tolerance * Dr. Eric Brewer is on Basho’s Board of Directors.
36. basho In 2002, Seth Gilbert and Nancy Lynch of MIT published a proof of Brewer’s conjecture, that at any given moment in time you can only guarantee 2 of the 3 characteristics.
37. basho What does Consistency, Availability and Partition Tolerance mean?
38. basho Consistency
39. basho Consistency means a request for a piece of data is guaranteed to always return the last written value.
40. basho The thought of sacrificing consistency seems to make some people nervous.
41. basho opposite(consistency) ≠ random data
42. basho opposite(consistency) ≠ lack of data integrity
43. basho Sacrificing consistency means that some reads may not get the most current data.
44. basho “I could never sacrifice consistency in my application!”
45. basho In some scenarios any inconsistency is unacceptable.
46. basho In many scenarios it’s a perfectly acceptable trade-off.
47. basho You may already be doing so.
48. basho Do you use DNS?
49. basho Ever had to wait for a DNS change to propagate?
50. basho Amazingly the internet continues to function.
51. basho Anyone have a master / slave database setup?
52. basho Is replication ever out of sync between the master and slave?
53. basho You’ve already sacrificed consistency.
54. basho Anyone use a caching layer?
55. basho Does that cache ever serve data that may not be the absolute latest available?
56. basho You’ve already sacrificed consistency.
57. basho Riak is an “eventually consistent” system.
58. basho This means Riak is designed to tolerate temporary inconsistency.
59. basho Riak provides means to automatically move towards consistency.
60. basho “Eventual” often means seconds, or even milliseconds.
61. basho Why sacrifice consistency at all?
62. basho Consistency has a cost: Availability. (Example: table or row locks blocking writes)
63. basho In a distributed system the cost of consistency increases. (Multiple nodes, network latency)
64. basho If we trade-off a measure of consistency - then - in exchange we can increase availability.
65. basho Availability
66. basho Requests will continue to complete, even when portions of the distributed system are down.
67. basho Availability is not the same thing as 100% uptime.
68. basho Availability is not the same thing as 100% uptime. (If all your nodes are inaccessible, requests will not complete.)
69. basho Writes are always accepted. No incoming data is lost.
70. basho Reads will always return a response. That response may be a 404.
71. basho Partition Tolerance
72. basho Can your network guarantee zero problems? (Network partitions, latency, lost messages, etc.)
73. basho Failure must be expected. (More on that later)
74. basho Partition Tolerance: Not on the table for trading. (More info, google: “coda hale partition tolerance”)
75. basho The CAP Decision
76. basho We want all 3 all the time.
77. basho Unfortunately we don’t always get what we want.
78. basho The CAP theorem informs us that trade-offs will be required in any distributed system.
79. basho Of note, it does not require committing to two.
80. basho It’s pick two at any given moment in time.
81. basho But when a trade-off must be made, what will you sacrifice and when?
82. basho Whatever distributed system you may use, this is relevant, and a question you should have an answer for.
83. basho Enough theory...
84. basho Let’s focus on Riak
85. basho Riak is an open source, highly scalable, fault-tolerant distributed database.
86. basho Riak is an open source, highly scalable, fault-tolerant distributed database.
87. basho License: Apache 2 Open Source
88. basho Source Code: github.com/basho/riak Open Source
89. basho Packages: basho.com Open Source
90. basho Open Source Documentation: wiki.basho.com (Also check out Mathias Meyer’s Riak Handbook)
91. basho Open Source Great community: basho.com/community IRC, mailing list, etc.
92. basho Riak is an open source, highly scalable, fault-tolerant distributed database.
93. basho What does that mean? Highly Scalable
94. basho Scalable is not a synonym for fast. Highly Scalable
95. basho A scalable system may be fast. Highly Scalable
96. basho Speed alone does not assure scalability. Highly Scalable
97. basho Highly Scalable Scaling is about understanding and predicting the cost of growth.
98. basho Highly Scalable It means being able to answer specific questions.
99. basho Highly Scalable What will it cost to grow capacity 2X? (“Capacity” may be write-throughput, storage capacity, processing power, etc.)
100. basho Highly Scalable Riak is “Linearly Scalable”.
101. basho Highly Scalable Why might you care?
102. basho Highly Scalable Linear scalability makes predicting scaling costs easy.
103. basho Highly Scalable With Riak, you get X% “capacity” with X% the nodes. (Where “capacity” equals write-throughput, storage capacity, and processing power.)
104. basho Highly Scalable: Voxer Voxer is a walkie-talkie/push-to-talk app for iOS & Android.
105. basho Highly Scalable: Voxer #1 social networking app in iOS & Android app stores (December 2011)
106. basho Highly Scalable: Voxer They needed to increase capacity 10X
107. basho Highly Scalable: Voxer This was achieved by increasing their node count 10X
108. basho Highly Scalable: Voxer 1 billion operations per day
109. basho Highly Scalable: Voxer Not too shabby.
110. basho Highly Scalable What does it look like from an ops perspective?
111. basho Highly Scalable Another real example from a customer.
112. basho Highly Scalable You have a cluster.
113. basho Highly Scalable They had a new node provisioned.
114. basho Highly Scalable You have a cluster.
115. basho Highly Scalable On an existing node, add the new one: $ riak-admin join node_foo@192.168.10.223
116. basho Highly Scalable Data will redistribute.
117. basho
118. basho Highly Scalable Until the cluster is rebalanced.
119. basho
120. basho Highly Scalable Invisible to end users.
121. basho Highly Scalable In this case, we saw: 6ms variance for 99th% (32 to 38ms) 0.68s variance for 100th% (0.12s to 0.8s)
122. basho Riak is an open source, highly scalable, fault-tolerant distributed database.
123. basho Fault Tolerant What does that mean?
124. basho Fault Tolerant It’s not the same as reliable.
125. basho Fault Tolerant Reliable is good.
126. basho Fault Tolerant Reliable means unlikely to fail.
127. basho Fault Tolerant Everything fails.
128. basho Fault Tolerant Even reliable things.
129. basho Fault Tolerant Reliability doesn’t help you during a failure.
130. basho Fault Tolerant Fault tolerance means planning on failures.
131. basho Fault Tolerant Why might you care?
132. basho Fault Tolerant Reduce the cost of the inevitable failures.
133. basho Fault Tolerant Everything fails.
134. basho Fault Tolerant You will face failure. Repeatedly.
135. basho Fault Tolerant Even with the...
136. basho The Cloud
137. basho Uptime! (It’s always sunny, and the universe loves you!)
138. basho Scalability! (Now with 100% more sparkles and rainbows!!)
139. basho
140. basho
141. basho
142. basho
143. basho Suddenly...
144. basho Defective NAS enclosures are cooking your drives.
145. basho The NOC guy accidentally yanked the wrong cable(s).
146. basho The upstream network provider is experiencing an outage.
147. basho
148. basho
149. basho
150. basho Fault Tolerant Riak is designed to transparently survive node failures, network failures, and software failures.
151. basho Fault Tolerant If a node is down, another takes over responsibility for it.
152. basho Fault Tolerant Writes that the failed node was supposed to own are temporarily assigned to a substitute node.
153. basho Fault Tolerant Reads requests for data the failed node was responsible for will trigger a replica being sent to the substitute node.
154. basho Fault Tolerant Your success & sanity will be affected by ability to gracefully handle failures.
155. basho Fault Tolerant Riak’s fault tolerance is largely contributed to by...
156. basho Riak is an open source, highly scalable, fault-tolerant distributed database.
157. basho What does that mean? Distributed
158. basho Distributed That you are talking over a network protocol.
159. basho Distributed In itself, not very interesting.
160. basho Distributed Riak is also decentralized and homogenous.
161. basho Distributed Why might you care?
162. basho Distributed Decentralization and homogeneity provide some highly desirable consequences.
163. basho Distributed No master or otherwise “special” nodes.
164. basho Distributed All nodes can accept requests.
165. basho Distributed This means no single point of failure.
166. basho Distributed When a node fails, it doesn’t matter which one it is.
167. basho Distributed The failure of individual nodes generally means reduced capacity, not service disruption.
168. basho Those are some of the high level benefits that Riak offers.
169. basho CAP Tuning
170. basho Distributed systems are subject to the CAP Theorem.
171. basho The CAP theorem informs us that trade-offs will be required in any distributed system.
172. basho Some distributed systems will make a choice for you.
173. basho Many DBMS choose to guarantee consistency by use of locks (row, table, etc).
174. basho By definition, while locked, it is not available.
175. basho Riak favors availability, but gives you control.
176. basho Control to decide what and when trade-offs make sense in your application.
177. basho Riak allows this fine tuning of behavior via N, R, W
178. basho N = Number of replicas across the cluster. (Cluster-wide setting)
179. basho N = Number of replicas across the cluster. (Ideally across N distinct servers, but not necessarily... e.g. you have less than N servers.)
180. basho N = Number of replicas across the cluster. (Rule of thumb: A cluster of at least N+1 nodes)
181. basho R = Number of replicas which must return. (Tunable per request)
182. basho W = Number of replicas for write to be successful (Tunable per request)
183. basho Built for the Web
184. basho Speaks HTTP by default (Play with it via curl) Does HTTP/REST well (see webmachine) Plays well with existing HTTP infrastructure (Reverse proxy caches, load balancers, web servers)
185. basho Riak is interesting software
186. basho You should try it out
187. basho But, should you build your app on Riak?
188. basho Maybe, maybe not.
189. basho Don’t use Riak if...
190. basho Don’t use Riak if... You don’t want to tradeoff familiar query patterns
191. basho Don’t use Riak if... You don’t want to tradeoff familiar hiring patterns
192. basho Don’t use Riak if... Incomplete understanding of your data model
193. basho Don’t use Riak if... Everything is fine on a single server database
194. basho Don’t use Riak if... You have no problems with your current database
195. basho Seriously consider Riak if...
196. basho Seriously consider Riak if... Data availability outages are expensive
197. basho Seriously consider Riak if... You need to be able to scale predictably
198. basho Seriously consider Riak if... You need your latency to be predictable
199. basho Seriously consider Riak if... You’re willing to learn new ways to structure & query data
200. basho Seriously consider Riak if... You’re willing to accept eventual consistency
201. basho Seriously consider Riak if... You’re willing to train devs to use a new type of database
202. basho Successful Use Cases
203. basho Successful Use Cases Social Networking: Yammer, Clipboard, GitHub
204. basho Successful Use Cases Casual Gaming: Mochi Media
205. basho Successful Use Cases Medical Records: Trifork/Denmark
206. basho Successful Use Cases Mobile Apps: Voxer, Bump
207. basho Successful Use Cases Caching: Posterous
208. basho Successful Use Cases Session Storage
209. basho Conclusion
210. basho Riak is not right for every project.
211. basho But if you need a highly scalable, fault tolerant, distributed database... you should check it out.
212. basho People are doing awesome things with Riak.
213. basho
214. basho Client libraries make it pretty easy to try out.
215. basho Basho Developed Erlang Javascript Java PHP Python Ruby
216. basho Community Developed C C++ Clojure Common Lisp Go Haskell .NET Perl Smalltalk etc.
217. basho We have a fast track at wiki.basho.com.
218. basho Riak 1.1.1 was released March 7, 2012.
219. basho Includes Riak Control
220. basho Demo
221. basho Give it a spin.
222. basho Any questions?
223. basho Looking for work? I’m looking for software developers who are passionate about customer service.
224. basho Thank you! twitter: @j_pease email: jpease@basho.com