Riak Use Cases - Dissecting The Solutions To Hard Problems

Presented on November 29, 2012, by Basho Hacker Chris Molozian. This is a great slide deck for anyone looking to get a better impression of Riak, where it fits in the database space, and how people are using it in production.

1. Riak Use Cases Dissecting the Solutions to Hard Problems NoSQL Roadshow - Amsterdam 29 / 11 / 2012 Friday, 30 November 12
2. $ Friday, 30 November 12
3. whoami $ Friday, 30 November 12
4. whoami $ Name: Chris Molozian Title: Client Services Engineer Company: Basho Technologies Email: cmolozian@basho.com Friday, 30 November 12
5. $ Friday, 30 November 12
6. whoami $ Friday, 30 November 12
7. whoami $ Name: Matthew Revell $ Title: Community Manager Company: Basho Technologies Twitter: @matthewrevell Friday, 30 November 12
8. whoami $ Name: Matthew Revell $ ./presentation Title: Community Manager Company: Basho Technologies Twitter: @matthewrevell Friday, 30 November 12
9. NoSQL "departs from the relational model altogether; it should therefore have been called more appropriately 'NoREL' " ~ Carlo Strozzi Friday, 30 November 12
10. NoSQL • Divided into a (growing) list of categories (the more exotic ones include Multivalue and Tuple stores) • All are “optimized” for record storage • Arguably the largest categories are: "departs from the relational model altogether; it should therefore have been called more appropriately 'NoREL' " ~ Carlo Strozzi Friday, 30 November 12
11. NoSQL • Divided into a (growing) list of categories (the more exotic ones include Multivalue and Tuple stores) • All are “optimized” for record storage • Arguably the largest categories are: Graph "departs from the relational model altogether; it should therefore have been called more appropriately 'NoREL' " ~ Carlo Strozzi Friday, 30 November 12
12. NoSQL • Divided into a (growing) list of categories (the more exotic ones include Multivalue and Tuple stores) • All are “optimized” for record storage • Arguably the largest categories are: Graph Key-Value "departs from the relational model altogether; it should therefore have been called more appropriately 'NoREL' " ~ Carlo Strozzi Friday, 30 November 12
13. NoSQL • Divided into a (growing) list of categories (the more exotic ones include Multivalue and Tuple stores) • All are “optimized” for record storage • Arguably the largest categories are: Graph Key-Value "departs from the relational model altogether; it should therefore have been called more appropriately 'NoREL' " ~ Carlo Strozzi Document Friday, 30 November 12
14. Graph Friday, 30 November 12
15. Graph • Data is represented using: Friday, 30 November 12
16. Graph • Data is represented using: • Nodes - an entity of some kind (i.e. User) Friday, 30 November 12
17. Graph • Data is represented using: • Nodes - an entity of some kind (i.e. User) • Edges - the relationship between nodes Friday, 30 November 12
18. Graph • Data is represented using: • Nodes - an entity of some kind (i.e. User) • Edges - the relationship between nodes • Use when the important data is in the edges Friday, 30 November 12
19. Graph • Data is represented using: • Nodes - an entity of some kind (i.e. User) • Edges - the relationship between nodes • Use when the important data is in the edges Node Node Node Edge Edge Edge Friday, 30 November 12
20. Document Friday, 30 November 12
21. Document • Data is represented using: • Documents - A !exible collection of k/v pairs ("elds) • Use when most queries are not primary key Friday, 30 November 12
22. Document • Data is represented using: • Documents - A !exible collection of k/v pairs ("elds) • Use when most queries are not primary key Document UserID (key) : test_user Name : Chris Job : Engineer Friday, 30 November 12
23. Key-Value Friday, 30 November 12
24. Key-Value • Data is represented using: • Record - A key/value pair • Use when most queries are primary key or you can denormalize the data problem to k/v pair Friday, 30 November 12
25. Key-Value • Data is represented using: • Record - A key/value pair • Use when most queries are primary key or you can denormalize the data problem to k/v pair key value namespace key value key value Friday, 30 November 12
26. Friday, 30 November 12
27. What is Riak? Friday, 30 November 12
28. What is Riak? • Distributed key/value store + extras Friday, 30 November 12
29. What is Riak? • Distributed key/value store + extras • Advanced query features Friday, 30 November 12
30. What is Riak? • Distributed key/value store + extras • Advanced query features • Pre/Post commit hooks Friday, 30 November 12
31. What is Riak? • Distributed key/value store + extras • Advanced query features • Pre/Post commit hooks • Multiple storage engines Friday, 30 November 12
32. What is Riak? • Distributed key/value store + extras • Advanced query features • Pre/Post commit hooks • Multiple storage engines • Scales Linearly + Fault Tolerant Friday, 30 November 12
33. What is Riak? • Distributed key/value store + extras • Advanced query features • Pre/Post commit hooks • Multiple storage engines • Scales Linearly + Fault Tolerant • Open Source (Apache 2.0) Friday, 30 November 12
34. What is Riak? • Distributed key/value store + extras • Advanced query features • Pre/Post commit hooks • Multiple storage engines • Scales Linearly + Fault Tolerant • Open Source (Apache 2.0) • Written in Erlang/OTP Friday, 30 November 12
35. Tunable Consistency Friday, 30 November 12
36. Tunable Consistency • We haven’t solved CAP; no one has Friday, 30 November 12
37. Tunable Consistency • We haven’t solved CAP; no one has • With Riak, you tune the CAP values: Friday, 30 November 12
38. Tunable Consistency • We haven’t solved CAP; no one has • With Riak, you tune the CAP values: • N: number of instances of your data Friday, 30 November 12
39. Tunable Consistency • We haven’t solved CAP; no one has • With Riak, you tune the CAP values: • N: number of instances of your data • R: number of nodes Riak reads from Friday, 30 November 12
40. Tunable Consistency • We haven’t solved CAP; no one has • With Riak, you tune the CAP values: • N: number of instances of your data • R: number of nodes Riak reads from • W: number of nodes Riak writes to, before optional further replication Friday, 30 November 12
41. Tunable Consistency • We haven’t solved CAP; no one has • With Riak, you tune the CAP values: • N: number of instances of your data • R: number of nodes Riak reads from • W: number of nodes Riak writes to, before optional further replication • Per cluster, per bucket or per operation Friday, 30 November 12
42. Con!ict Resolution (1) Friday, 30 November 12
43. Con!ict Resolution (1) • Concurrent actors modifying the same data cause data divergence. Friday, 30 November 12
44. Con!ict Resolution (1) • Concurrent actors modifying the same data cause data divergence. • Riak provides two solutions to manage this: Friday, 30 November 12
45. Con!ict Resolution (1) • Concurrent actors modifying the same data cause data divergence. • Riak provides two solutions to manage this: • Last Write Wins Naive approach but works for some use cases Friday, 30 November 12
46. Con!ict Resolution (1) • Concurrent actors modifying the same data cause data divergence. • Riak provides two solutions to manage this: • Last Write Wins Naive approach but works for some use cases • Vector Clocks Retain “sibling” copies of data for merging Friday, 30 November 12
47. Con!ict Resolution (2) Friday, 30 November 12
48. Con!ict Resolution (2) node node node node node App App App LB VMs Riak Cluster Friday, 30 November 12
49. Con!ict Resolution (2) node node node node node App App App LB VMs Riak Cluster Application layer timestamps, with siblings Friday, 30 November 12
50. Con!ict Resolution (3) Friday, 30 November 12
51. Con!ict Resolution (3) node node node node node LB Riak Cluster App App App App App Friday, 30 November 12
52. Con!ict Resolution (3) Application layer business logic, with siblings node node node node node LB Riak Cluster App App App App App Friday, 30 November 12
53. Sibling Handling Friday, 30 November 12
54. Sibling Handling "We don't ever do conflict resolution by picking a random sibling." Friday, 30 November 12
55. Sibling Handling "We don't ever do conflict resolution by picking a random sibling." "For an array property, we often take the union of all values in all siblings. This works great for array properties that we only ever add to." Friday, 30 November 12
56. Sibling Handling "We don't ever do conflict resolution by picking a random sibling." "For an array property, we often take the union of all values in all siblings. This works great for array properties that we only ever add to." "We often take the maximum sibling value or the minimum sibling value, depending on the semantics of that attribute." Friday, 30 November 12
57. Sibling Handling "We don't ever do conflict resolution by picking a random sibling." "For an array property, we often take the union of all values in all siblings. This works great for array properties that we only ever add to." "We often take the maximum sibling value or the minimum sibling value, depending on the semantics of that attribute." ~ Myron Marston, SEOMoz Friday, 30 November 12
58. • HTTP REST or optimised binary interface (PB) • O#cial Basho supported: • Community: C#, C/C++, Haskell, Clojure, Scala, Go, PHP and many others Client Libraries Friday, 30 November 12
59. Riak Use Cases Friday, 30 November 12
60. Riak Use Cases • Reliability, !exibility, scalability Friday, 30 November 12
61. Riak Use Cases • Reliability, !exibility, scalability • Session Data Friday, 30 November 12
62. Riak Use Cases • Reliability, !exibility, scalability • Session Data • Serving Advertising Friday, 30 November 12
63. Riak Use Cases • Reliability, !exibility, scalability • Session Data • Serving Advertising • Log and Sensor Data Friday, 30 November 12
64. Riak Use Cases • Reliability, !exibility, scalability • Session Data • Serving Advertising • Log and Sensor Data • Content Addressable Storage (CAS) Friday, 30 November 12
65. Riak Use Cases • Reliability, !exibility, scalability • Session Data • Serving Advertising • Log and Sensor Data • Content Addressable Storage (CAS) • Private Cloud [S3 API] - Riak CS Friday, 30 November 12
66. Riak Use Cases • Reliability, !exibility, scalability • Session Data • Serving Advertising • Log and Sensor Data • Content Addressable Storage (CAS) • Private Cloud [S3 API] - Riak CS • Wherever low latency increases revenue Friday, 30 November 12
67. Rovio is an industry-changing entertainment media company based in Finland, and the creator of the globally successful Angry Birds franchise. Friday, 30 November 12
68. • Store Game Session data in Riak A per-user collection of game “states”. Rovio is an industry-changing entertainment media company based in Finland, and the creator of the globally successful Angry Birds franchise. Friday, 30 November 12
69. • Store Game Session data in Riak A per-user collection of game “states”. • Synchronization of user’s data across gaming devices. Rovio is an industry-changing entertainment media company based in Finland, and the creator of the globally successful Angry Birds franchise. Friday, 30 November 12
70. • Store Game Session data in Riak A per-user collection of game “states”. • Synchronization of user’s data across gaming devices. • Buckets: Account - Keyed by user_id Rovio is an industry-changing entertainment media company based in Finland, and the creator of the globally successful Angry Birds franchise. Friday, 30 November 12
71. An Enterprise Social Network that brings together employees, content, conversations, and business data in a single location. Friday, 30 November 12
72. • Store “Noti"cations” in Riak A per-user sorted set of events with calls to action. An Enterprise Social Network that brings together employees, content, conversations, and business data in a single location. Friday, 30 November 12
73. • Store “Noti"cations” in Riak A per-user sorted set of events with calls to action. • Data types consist of: Cursor, Item List, Items {id: 41626118990497, timestamp: 1300845012, category: “likes- message”, properties: {liker_id: 97238, [... etc]} } An Enterprise Social Network that brings together employees, content, conversations, and business data in a single location. Friday, 30 November 12
74. • Store “Noti"cations” in Riak A per-user sorted set of events with calls to action. • Data types consist of: Cursor, Item List, Items {id: 41626118990497, timestamp: 1300845012, category: “likes- message”, properties: {liker_id: 97238, [... etc]} } • Buckets: Cursor - Keyed by user_id + cursor_name Stream - Keyed by user_id An Enterprise Social Network that brings together employees, content, conversations, and business data in a single location. Friday, 30 November 12
75. • Store “Noti"cations” in Riak A per-user sorted set of events with calls to action. • Data types consist of: Cursor, Item List, Items {id: 41626118990497, timestamp: 1300845012, category: “likes- message”, properties: {liker_id: 97238, [... etc]} } • Buckets: Cursor - Keyed by user_id + cursor_name Stream - Keyed by user_id • SOA - known as “Streamie” An Enterprise Social Network that brings together employees, content, conversations, and business data in a single location. Friday, 30 November 12
76. SEOmoz is the world’s most popular provider of SEO software. Their easy to use tools and tutorials make search engine optimization accessible to everyone. Friday, 30 November 12
77. • Ranking collections of web documents SEOmoz is the world’s most popular provider of SEO software. Their easy to use tools and tutorials make search engine optimization accessible to everyone. Friday, 30 November 12
78. • Ranking collections of web documents • Data types consist of: Subscription(s), Ranking List, Ranking History, Recent Ranking Report... (etc) SEOmoz is the world’s most popular provider of SEO software. Their easy to use tools and tutorials make search engine optimization accessible to everyone. Friday, 30 November 12
79. • Ranking collections of web documents • Data types consist of: Subscription(s), Ranking List, Ranking History, Recent Ranking Report... (etc) • Buckets: Ranking List - Keyed by engine+locale+keyword+URL_fragment Subscription - Keyed by user_campaign SEOmoz is the world’s most popular provider of SEO software. Their easy to use tools and tutorials make search engine optimization accessible to everyone. Friday, 30 November 12
80. Other Users Friday, 30 November 12
81. Other Users • High Availability Environments (i.e. Health Care) Friday, 30 November 12
82. Other Users • High Availability Environments (i.e. Health Care) • Content Addressable Storage as a Service (like a private Dropbox cloud) Friday, 30 November 12
83. Other Users • High Availability Environments (i.e. Health Care) • Content Addressable Storage as a Service (like a private Dropbox cloud) • Oil/Gas Rig Environment Logging Friday, 30 November 12
84. Other Users • High Availability Environments (i.e. Health Care) • Content Addressable Storage as a Service (like a private Dropbox cloud) • Oil/Gas Rig Environment Logging • Web Gaming Platforms Friday, 30 November 12
85. Other Users • High Availability Environments (i.e. Health Care) • Content Addressable Storage as a Service (like a private Dropbox cloud) • Oil/Gas Rig Environment Logging • Web Gaming Platforms • Product Catalog (and other Retail use cases) Friday, 30 November 12
86. With any Use Case Consider 3 Things Friday, 30 November 12
87. With any Use Case Consider 3 Things • Query Patterns Friday, 30 November 12
88. With any Use Case Consider 3 Things • Query Patterns • Inter-connectivity of your data (how much can it be denormalised) Friday, 30 November 12
89. With any Use Case Consider 3 Things • Query Patterns • Inter-connectivity of your data (how much can it be denormalised) • Polyglot solution (SOA + Database) (no single database "ts every problem) Friday, 30 November 12
90. With any Use Case Consider 3 Things • Query Patterns • Inter-connectivity of your data (how much can it be denormalised) • Polyglot solution (SOA + Database) (no single database "ts every problem) Understand your data access patterns and you’ll be able to choose the right database for you Friday, 30 November 12
91. Basho EMEA Friday, 30 November 12
92. Questions? Chris Molozian, cmolozian@basho.com Matthew Revell, mrevell@basho.com Friday, 30 November 12
93. Want to know more? We will come and give a Riak tech talk at your organisation or group: bit.ly/RiakTechTalk Friday, 30 November 12