Data Modeling for Scale with Riak Data Types

Your application has gotten big, you know you need to start scaling horizontally. You can't afford downtime because competition is tight and users will jump ship if they have a bad experience, but you also can't sacrifice the rapid development process you're used to. You're looking at scalable NoSQL databases like Riak, but the key-value model is too difficult, especially when you have to deal with eventual consistency too. What can you do?

Luckily, Riak 2.0 now includes a number of rich data types that give you a familiar interface to counters, sets, maps, registers, and boolean flags, while handling all the ugliness of eventual consistency for you. Instead of teaching your application how to resolve conflicts for every new type of data, you can simply manipulate the data types to fit your application and quit worrying. In this talk, I'll briefly introduce the theory and implementation behind Riak's data types (also known as CRDTs), and walk through some example applications that use them in popular languages. You can have your horizontally-scalable cake and eat it too!

Transcript

1. Data Modeling for Scale with Riak Data Types Sean Cribbs @seancribbs #riak #datatypes GlueCon 2014
2. I work for Basho We make Visit our booth
3. Riak is Eventually Consistent
4. Eventual Consistency Replicated Loose coordination Convergence 1 2 3
5. ✔Fault-tolerant ✔Highly available ✔Low-latency Eventual is Good
6. Consistency? 1 2 3 B A No clear winner! Throw one out? Keep both?
7. Consistency? 1 2 3 B A No clear winner! Throw one out? Keep both? Cassandra
8. Consistency? 1 2 3 B A No clear winner! Throw one out? Keep both? Cassandra Riak
9. Con!icts! A! B!
10. Semantic Resolution • Your app knows the domain - use business rules to resolve • Amazon Dynamo’s shopping cart
11. Semantic Resolution • Your app knows the domain - use business rules to resolve • Amazon Dynamo’s shopping cart “Ad hoc approaches have proven brittle and error-prone”
12. Convergent Replicated Data Types
13. Convergent Replicated Data Types useful abstractions
14. Convergent Replicated Data Types multiple independent copies useful abstractions
15. Convergent Replicated Data Types multiple independent copies resolves automatically toward a single value useful abstractions
16. How CRDTs Work • A partially-ordered set of values • A merge function • An identity value • In!ation operations
17. How CRDTs Work • A partially-ordered set of values • A merge function • An identity value • In!ation operations What CRDTs Enable • Consistency without coordination • Fluent, rich interaction with data
18. This research is supported in part by European FP7 project 609 551 SyncFree http://syncfree.lip6.fr/ (2013--2016).
19. by @joedevivo
20. Forget CRDTs Do Data Modeling
21. Data Modeling for Riak • Identify needs for both read and write • Design around key as index • Denormalize relationships if possible • Weigh data size against coherence
22. Riak Data Types
23. Riak Data Types increment decrement Counter :: int
24. Riak Data Types increment decrement Counter :: int add* remove Set :: { bytes }
25. Riak Data Types increment decrement Counter :: int add* remove Set :: { bytes } remove update* Map :: bytes → DT
26. Riak Data Types increment decrement Counter :: int add* remove Set :: { bytes } remove update* Map :: bytes → DT
27. Riak Data Types increment decrement Counter :: int add* remove Set :: { bytes } assign Register :: bytes remove update* Map :: bytes → DT
28. Riak Data Types increment decrement Counter :: int add* remove Set :: { bytes } enable* disable Flag :: boolean assign Register :: bytes remove update* Map :: bytes → DT
29. MADDATA
30. Counters
31. Ad Network • Impressions - when someone sees an ad • Click-through - when someone clicks on an ad • Hourly rollups ad-metrics/<campaign>/<type>-<hour>
32. $ riak-admin bucket-type create ad-metrics \ '{"props":{"datatype":"counter"}}' ad-metrics created $ riak-admin bucket-type activate ad-metrics ad-metrics has been activated $ riak-admin bucket-type list ad-metrics (active) Ad Network
33. from riak import RiakClient from rogersads import RIAK_CONFIG from time import strftime client = RiakClient(**RIAK_CONFIG) metrics = client.bucket_type('ad-metrics') def record_metric(campaign, metric_type): key = metric_type + strftime('-%Y%m%d-%H') counter = metrics.bucket(campaign).new(key) counter.increment() counter.store() Ad Network
34. Ad Network 0 750 1500 2250 3000 9 10 11 12 13 14 15 16
35. Sets
36. • RSVPs - guest lists • Connections - friends lists per-user • Likes - expressing interest PartyOn
37. $ riak-admin bucket-type create partyon-sets \ '{"props":{"datatype":"set"}}' partyon-sets created $ riak-admin bucket-type activate partyon-sets partyon-sets has been activated $ riak-admin bucket-type list partyon-sets (active) PartyOn
38. • RSVPs partyon-sets/rsvps/<eventid> • Connections partyon-sets/friends/<userid> • Likes partyon-sets/likes/<eventid> PartyOn
39. from riak.datatypes import Set sets = client.bucket_type('partyon-sets') rsvps = sets.bucket('rsvps') friends = sets.bucket('friends') likes = sets.bucket('likes') PartyOn
40. def rsvp_get(event): return rsvps.get(event) # Returns a Set def rsvp_add(event, user): guests = rsvps.new(event) guests.add(user) guests.store(return_body=True) return guests.context def rsvp_remove(event, user, context): guests = Set(rsvps, event, context=context) guests.remove(user) guests.store() PartyOn
41. Maps (and the rest)
42. GameNet • User pro"les - demographic data users/<userid>/profile • Achievements - trophies per game users/<userid>/trophies • Game state - progress and stats users/<userid>/<gameid>
43. $ riak-admin bucket-type create users \ '{"props":{"datatype":"map"}}' users created $ riak-admin bucket-type activate users users has been activated $ riak-admin bucket-type list users (active) GameNet
44. users = client.bucket_type('users') def update_profile(user, fields): profile = users.bucket(user).get('profile') for field in fields: if field in USER_FLAGS: if fields[field]: profile.flags[field].enable() else: profile.flags[field].disable() else: value = fields[field] profile.registers[field].assign(value) profile.store() GameNet
45. def add_trophy(user, game, trophy): trophies = users.bucket(user).get('trophies') trophies.sets[game].add(trophy) trophies.store() def get_trophies(user, game): trophies = users.bucket(user).get('trophies') return trophies.sets[game].value GameNet
46. def build_structure(user, game, structure, gold, wood, stone): gamestate = users.bucket(user).get(game) gamestate.sets['structures'].add(structure) gamestate.counters['gold'].decrement(gold) gamestate.counters['wood'].decrement(wood) gamestate.counters['stone'].decrement(stone) gamestate.store(return_body=True) return gamestate.value GameNet
47. Bene"ts • Richer interactions, familiar types • Write mutations, not state • No merge function to write • Same reliability and predictability of vanilla Riak
48. Caveats • Value size still matters • Updates not idempotent • Cross-key atomicity not possible (yet)
49. Future • Riak 2.0 due out this summer - betas available now! • Richer querying, lighter storage requirements, more types