Data Modeling for Scale with Riak Data Types (QCon NY)

Eventually consistent databases like Riak give you horizontal scale, high availability and predictable latency, but can make data modeling more difficult. Riak 2.0 breaks from this pattern by introducing rich data types that are familiar to developers but handle eventual consistency for you.

In this talk, we'll briefly discuss the theory behind these new datatypes and then walk through some example applications that use them in popular languages.

Transcript

1. Data Modeling for Scale with Riak Data Types Sean Cribbs @seancribbs #riak #datatypes QCon NYC 2014
2. I work for Basho We make Visit our booth!
3. Riak is Eventually Consistent key-value + indexes + search + MapReduce
4. Eventual Consistency Replicated Loose coordination Convergence 1 2 3
5. ✔Fault-tolerant ✔Highly available ✔Low-latency Eventual is Good
6. Consistency? 1 2 3 B A No clear winner! Throw one out? Keep both?
7. Consistency? 1 2 3 B A No clear winner! Throw one out? Keep both? Cassandra
8. Consistency? 1 2 3 B A No clear winner! Throw one out? Keep both? Cassandra Riak
9. Con!icts! A! B!
10. Semantic Resolution • Your app knows the domain - use business rules to resolve • Amazon Dynamo’s shopping cart
11. Semantic Resolution • Your app knows the domain - use business rules to resolve • Amazon Dynamo’s shopping cart “Ad hoc approaches have proven brittle and error-prone”
12. Convergent Replicated Data Types
13. Convergent Replicated Data Types useful abstractions
14. Convergent Replicated Data Types multiple independent copies useful abstractions
15. Convergent Replicated Data Types multiple independent copies resolves automatically toward a single value useful abstractions
16. How CRDTs Work • A partially-ordered set of values • A merge function • An identity value • In!ation operations
17. How CRDTs Work • A partially-ordered set of values • A merge function • An identity value • In!ation operations What CRDTs Enable • Consistency without coordination • Fluent, rich interaction with data
18. This research is supported in part by European FP7 project 609 551 SyncFree http://syncfree.lip6.fr/ (2013--2016).
19. by @joedevivo
20. Forget CRDTs Do Data Modeling
21. Data Modeling for Riak • Identify needs for both read and write • Design around key as index • Denormalize relationships if possible • Weigh data size against coherence
22. Riak Data Types
23. Riak Data Types increment decrement Counter :: int
24. Riak Data Types increment decrement Counter :: int add* remove Set :: { bytes }
25. Riak Data Types increment decrement Counter :: int add* remove Set :: { bytes } remove update* Map :: bytes → DT
26. Riak Data Types increment decrement Counter :: int add* remove Set :: { bytes } remove update* Map :: bytes → DT
27. Riak Data Types increment decrement Counter :: int add* remove Set :: { bytes } assign Register :: bytes remove update* Map :: bytes → DT
28. Riak Data Types increment decrement Counter :: int add* remove Set :: { bytes } enable* disable Flag :: boolean assign Register :: bytes remove update* Map :: bytes → DT
29. MADDATA
30. Counters
31. Ad Network • Impressions - when someone sees an ad • Click-through - when someone clicks on an ad • Hourly rollups ad-metrics/<campaign>/<type>-<hour>
32. $ riak-admin bucket-type create ad-metrics \ '{"props":{"datatype":"counter"}}' ad-metrics created $ riak-admin bucket-type activate ad-metrics ad-metrics has been activated $ riak-admin bucket-type list ad-metrics (active) Ad Network
33. from riak import RiakClient from rogersads import RIAK_CONFIG from time import strftime client = RiakClient(**RIAK_CONFIG) metrics = client.bucket_type('ad-metrics') def record_metric(campaign, metric_type): key = metric_type + strftime('-%Y%m%d-%H') counter = metrics.bucket(campaign).new(key) counter.increment() counter.store() Ad Network
34. Ad Network 0 750 1500 2250 3000 9 10 11 12 13 14 15 16
35. Sets
36. • RSVPs - guest lists • Connections - friends lists per-user • Likes - expressing interest PartyOn
37. $ riak-admin bucket-type create partyon-sets \ '{"props":{"datatype":"set"}}' partyon-sets created $ riak-admin bucket-type activate partyon-sets partyon-sets has been activated $ riak-admin bucket-type list partyon-sets (active) PartyOn
38. • RSVPs partyon-sets/rsvps/<eventid> • Connections partyon-sets/friends/<userid> • Likes partyon-sets/likes/<eventid> PartyOn
39. from riak.datatypes import Set sets = client.bucket_type('partyon-sets') rsvps = sets.bucket('rsvps') friends = sets.bucket('friends') likes = sets.bucket('likes') PartyOn
40. def rsvp_get(event): return rsvps.get(event) # Returns a Set def rsvp_add(event, user): guests = rsvps.new(event) guests.add(user) guests.store(return_body=True) return guests.context def rsvp_remove(event, user, context): guests = Set(rsvps, event, context=context) guests.remove(user) guests.store() PartyOn
41. Maps (and the rest)
42. GameNet • User pro"les - demographic data users/profiles/<userid> • Achievements - trophies per game users/trophies/<userid> • Game state - progress and stats users/<gameid>/<userid>
43. $ riak-admin bucket-type create users \ '{"props":{"datatype":"map"}}' users created $ riak-admin bucket-type activate users users has been activated $ riak-admin bucket-type list users (active) GameNet
44. users = client.bucket_type('users') def update_profile(user, fields): profile = users.bucket('profiles').get(user) for field in fields: if field in USER_FLAGS: if fields[field]: profile.flags[field].enable() else: profile.flags[field].disable() else: value = fields[field] profile.registers[field].assign(value) profile.store() GameNet
45. def add_trophy(user, game, trophy): trophies = users.bucket('trophies').get(user) trophies.sets[game].add(trophy) trophies.store() def get_trophies(user, game): trophies = users.bucket('trophies').get(user) return trophies.sets[game].value GameNet
46. def build_structure(user, game, structure, gold, wood, stone): gamestate = users.bucket(game).get(user) gamestate.sets['structures'].add(structure) gamestate.counters['gold'].decrement(gold) gamestate.counters['wood'].decrement(wood) gamestate.counters['stone'].decrement(stone) gamestate.store(return_body=True) return gamestate.value GameNet
47. client.create_search_index('asteroids') users.bucket('asteroids').set_property('search_index', 'asteroids') def find_asteroids_opponents(min_score=0): query = "score_counter:[{} to *]".format(min_score) results = client.fulltext_search( 'asteroids', query, fl=['userid_register', 'score_counter']) return results['docs'] GameNet
48. Bene"ts • Richer interactions, familiar types • Write mutations, not state • No merge function to write • Same reliability and predictability of vanilla Riak
49. Caveats • Value size still matters • Updates not idempotent (yet) • Cross-key atomicity not possible (yet)
50. Future • Riak 2.0 due out this summer - betas available now! • Richer querying, lighter storage requirements, more types