Opaque No More: Riak's New Data Types

Since the beginning, Riak has been a plain-Jane key-value store that pretty much didn't care what you stored in it.

That changed for the first time in Riak 1.4, when we introduced a new convergent data type, the counter, exposing operations other than just "set" and "get".

We're going even further for Riak 2.0 by adding more convergent datatypes, a whole new API for accessing them from client applications, and garbage collection to keep them tight and small.

Transcript

1. Opaque No More: Riak’s New Data Types Sean Cribbs Software Engineer
2. Sam Elliott Intern Extraordinarie Russell Brown CRDT Mastermind
3. Riak today Distributed, replicated, fault-tolerant, ops-friendly
4. Riak today Mostly plain key-value
5. Riak today Great for immutable or low-contention data
6. Riak today Conflicts? You’re on your own
7. Riak today Riak 1.4 introduced distributed counters
8. CRDTs Convergent Replicated Data Types aka “Commutative...” and “Conflict-Free...”
9. CRDTs Lots of maths to manipulate and resolve divergent data
10. Why? Eventual consistency is hard to reason about
11. Why? Writing conflict resolvers yourself is tedious and error-prone
12. Why? Opaque BLOBs sometimes just aren’t enough
13. Why? Databases should store and manipulate data
14. “You’ve got to give the people what they want.” Ben Folds, “Rock Star” Photo by BERKLEE COLLEGE OF MUSIC
15. Riak’s New Data Types but first...
16. Do you even know how Vector Clocks work?
17. Vector Clock actually... Version Vectors. http://haslab.wordpress.com/2011/07/08/version-vectors-are-not-vector-clocks/
18. Version Vector An alternative for wall-clock time; logical time
19. Version Vector List of “dots”: (Actor, Count) pairs
20. Version Vector Actions (updates) increment the actor’s count
21. Version Vector Merge by taking all actors, maximum count for each
22. Version Vector aka “G-Counter” from the CRDT literature
23. Counters Essentially same as 1.4 counters; increment and decrement ops
24. Counters [ Actor → (Increments, Decrements) ]
25. Counters increment Actor who receives operation adds to its count (same for decrements) [ Actor → (Increments, Decrements) ]
26. Counters Size: O(Actors)
27. Sets Lots of implementation options: G-Set, 2P-Set, OR-Set, U-Set
28. Observed-Remove Sets [ Element → (Adds, Removes) ]
29. OR Set add Create unique token, insert into element’s “Adds” [ Element → (Adds, Removes) ]
30. OR Set remove Copy all “Adds” into the “Removes” [ Element → (Adds, Removes) ]
31. OR Set merge Union “Adds”, Union “Removes” [ Element → (Adds, Removes) ]
32. OR Sets { Add | Remove } 㱺 Add wins
33. OR Sets Size: O(Adds + Removes) VERY space-inefficient, we can do better
34. ORSWOTs Observed-Remove Sets Without Tombstones
35. ORSWOTs Version Vector ⊠ [ Element → [ Dot ] ]
36. ORSWOT add Increment Version Vector Merge updated Dot with the Element’s Dots Version Vector ⊠ [ Element → [ Dot ] ]
37. ORSWOT remove Version Vector unchanged; drop removed elements Version Vector ⊠ [ Element → [ Dot ] ]
38. Merging ORSWOTs Element in both: merge dots Element differs: Drop if Dots dominated by other Version Vector Keep otherwise Merge Version Vector Version Vector ⊠ [ Element → [ Dot ] ]
39. ORSWOTs Size: O(Actors + Elements) About 165× SMALLER! 3> orset_bm:start(10000, riak_dt_orset). Results are {99,3464858} in 9051.647 ok 4> orset_bm:start(10000, riak_dt_orswot). Results are {99,21005} in 7206.505 ok 5> 3464858 / 21005. 164.95396334206143
40. Maps Not much in the literature yet Designed this summer during Sam’s internship
41. Maps Behavior is like an ORSWOT of Keys with CRDT Values
42. Maps Version Vector ⊠ [ (Name, ValueType) → ([Dot], ValueCRDT) ]
43. Maps update Apply type-specific update to ValueCRDT Increment Version Vector Insert Dot into value’s Dot list Version Vector ⊠ [ (Name, ValueType) → ([Dot], ValueCRDT) ]
44. Maps add Like update, but insert the “bottom” / empty value if not present Version Vector ⊠ [ (Name, ValueType) → ([Dot], ValueCRDT) ]
45. Maps remove Like ORSWOT remove, drop the key-value pair Version Vector ⊠ [ (Name, ValueType) → ([Dot], ValueCRDT) ]
46. Maps merge Field in both: Merge Dots, merge Values Field in only one: Drop when dominated by other Version Vector Keep otherwise
47. Maps values Counter, Set, Register, Flag, Map
48. Registers Opaque binary (Value, Timestamp) Last-Write Wins aka “Some Write Loses”
49. Flags Boolean value (Version Vector, Boolean) { Enable | Disable } 㱺 Enable
50. Garbage Collection Photo by Forest
51. Garbage CRDTs are monotonically increasing in size
52. Garbage Our types make no tombstones, but Actor IDs grow without bound
53. Garbage We can’t prune Actor IDs optimistically like in Riak Objects
54. Garbage Collection NOT in 2.0! But here’s what we have planned...
55. Lazy Consensus If Garbage(DataType) > Threshold Gossip to compute GLB Wait for commit, keeping GC info with CRDT
56. Lazy Consensus (Epoch, GLBSet, CommittedSet, GLB) 1a. Increment Epoch (one replica) 1b. Compute intersection into GLB, add self to GLBSet 2. When all are present in GLBSet, add self to CommittedSet 3. When all are present in CommittedSet, everyone collects
57. Lazy Consensus Problems: - Have to keep GC info until committed (grows!) - Who should the participants be?
58. Handoff Counters Automatic removal of expired Actors on Handoff http://arxiv.org/abs/1307.3207
60. Client API Currently only PB, HTTP coming soon
61. Client API Requires a Bucket Type with ‘datatype’ = map | set | counter ‘allow_mult’=true Manual right now, will be easier soon.
62. Client Operations %% Fetch {ok, DataType} = riakc_pb_socket:fetch_type(Pid, {Type, Bucket}, Key). %% Update riakc_pb_socket:update_type(Pid, {Type, Bucket}, Key, Op). %% Modify riakc_pb_socket:modify_type(Pid, Fun, {Type, Bucket}, Key, Options).
63. Client CRDT Behavior -callback new() -> datatype(). -callback new(Value::term(), context()) -> datatype(). -callback value(datatype()) -> term(). -callback dirty_value(datatype()) -> term(). -callback to_op(datatype()) -> update(term()). -callback is_type(datatype()) -> boolean(). -callback type() -> typename().
64. Client Counters %% Create one Counter = riakc_counter:new(), %% Increment / Decrement Counter1 = riakc_counter:increment(Counter), Counter2 = riakc_counter:decrement(5, Counter), %% Values 0 = riakc_counter:value(Counter2), %% Immutable -4 = riakc_counter:dirty_value(Counter2), %% Extract op Op = riakc_counter:to_op(Counter2).
65. Client Sets %% Create Set = riakc_set:new(), %% Add / remove Set1 = riakc_set:add_element(<<"sean">>, Set), Set2 = riakc_set:del_element(<<"russell">>, Set1), %% Values [] = riakc_set:value(Set2), [<<"sean">>] = riakc_set:dirty_value(Set2), %% Extract op Op = riakc_set:to_op(Set2).
66. Client Maps %% Create Map = riakc_map:new(), %% Add / remove fields Map1 = riakc_map:add({<<"emails">>, set}, Map), Map2 = riakc_map:erase({<<"id">>, register}, Map1), %% Update fields Map3 = riakc_map:update({<<"confirmed">>, flag}, fun riakc_flag:enable/1, Map2), %% Values [] = riakc_map:value(Map3), [{{<<"confirmed">>, flag}, true}, {{<<"emails">>, set}, []}] = riakc_map:dirty_value(Map3).
67. The Context If you’re going to remove, fetch first to get the context - Empty fallbacks - Missing elements to be removed Additions/modifications don’t need context!
68. SyncFree INRIA, Universidade Nova de Lisboa, Université Catholique de Louvain, Koç Üniversitesi, Technische Universität Kaiserslautern, Basho, Rovio, Trifork European FP7 project 609551 SyncFree http://syncfree.lip6.fr/ (2013-2016)
69. Thanks! @seancribbs sean@basho.com