CRDTs: An UPDATE (or just a PUT)

Riak 2.0's Data Types, presented at RICON West 2013, in SF.

A Few links for the interested:
- Dotted Version Vector Sets, a nice explanation: https://github.com/ricardobcl/Dotted-Version-Vectors#readme
- Our datatype repo: https://github.com/basho/riak_dt
- Comprehensive study of Convergent and Commutative Replicated Data Types (tech report): http://hal.upmc.fr/inria-00555588
- Optimised OR-Set (paper): http://arxiv.org/abs/1210.3368
- Dotted Version Vectors (paper): http://arxiv.org/abs/1011.5808
- Semilattices (wiki): https://en.wikipedia.org/wiki/Semilattice

Transcript

1. CRDTs
   An UPDATE (or maybe just a PUT)
   

   View Slide

2. Engineer, Bashtronaut
   Sam Elliott
   

   View Slide

3. Not Only Me
   Russell Brown
   Sean Cribbs
   Chris Meiklejohn
   Carlos Baquero (UMinho)
   

   View Slide

4. CRDTs
   Co-what Replicated Data Types?!
   

   View Slide

5. CRDTs
   Data Types
   

   View Slide

6. This Talk
   Usecases, Interface & Implementation
   

   View Slide

7. Usecases
   What can I do with them?
   

   View Slide

8. Counters
   Since 1.4
   

   View Slide

9. Counters
   Increment & Decrement
   

   View Slide

10. Counters
    Events!
    

    View Slide

11. Burger Counter
    

    View Slide

12. Sets
    New in 2.0!
    

    View Slide

13. Sets
    Add & Remove Elements
    

    View Slide

14. Sets
    Shopping Basket!
    

    View Slide

15. Shopping Cart
    

    View Slide

16. Maps
    New in 2.0!
    

    View Slide

17. Maps
    Values are CRDTs
    

    View Slide

18. Maps
    Values are Counters
    Sets
    

    View Slide

19. Maps
    Values are Counters
    Sets
    Registers
    Booleans
    

    View Slide

20. Maps
    Values are Counters
    Sets
    Registers
    Booleans
    Maps
    

    View Slide

21. Maps
    For all your Big Data™ needs!
    

    View Slide

22. Address Book
    

    View Slide

23. Interface
    How can I do that with them?
    

    View Slide

24. Counters
    Updates NOT Idempotent
    (Sorry Pat)
    

    View Slide

25. Counters
    Size: O(Actors)
    

    View Slide

26. Sets
    { Add | Remove } ~> Add Wins
    

    View Slide

27. Sets
    Size: O(Operations) ?
    

    View Slide

28. Sets
    Size: O(Actors + Elements)
    

    View Slide

29. Maps
    Entries keyed by Name & Type
    

    View Slide

30. Maps
    { Update | Remove } ~> Update Wins
    

    View Slide

31. Maps
    Size: O(Actors + Entries)
    

    View Slide

32. Registers
    Last Write Wins
    

    View Slide

33. Registers
    Some Write Loses
    

    View Slide

34. Booleans
    { Enable | Disable } ~> Enable
    

    View Slide

35. Booleans
    Size: O(Actors)
    

    View Slide

36. Operations
    PB & HTTP
    

    View Slide

37. Increment(1)
    Add(“Cuttlefish”)
    Update(“name”,
    register,
    Set(“Sam Elliott”))
    

    View Slide

38. Operations
    Atomic
    

    View Slide

39. Operations
    Unordered
    

    View Slide

40. Operations
    Causality
    

    View Slide

41. Operations
    Context Required for Correct Removal
    

    View Slide

42. Bucket Types
    Required for All Using 2.0 API
    

    View Slide

43. Bucket Types
    Counters from 1.4 API Use Default
    

    View Slide

44. Not in 2.0
    2i, Custom Metadata
    

    View Slide

45. Not in 2.0
    YZ Extractors
    

    View Slide

46. Not in 2.0
    Type-level Queries
    

    View Slide

47. Implementation
    How do they work underneath?
    

    View Slide

48. CRDTs
    Convergent Replicated Data Types
    

    View Slide

49. Semilattices
    S
    ≤
    ∨
    ⊥
    Values
    Partial Order
    Join
    Bottom
    

    View Slide

50. Counters
    PN-Counters
    

    View Slide

51. PN-Counters
    [ Node → ( Increments, Decrements ) ]
    

    View Slide

52. PN-Counters: Inc
    Increase Increments for Current Node
    

    View Slide

53. PN-Counters: Dec
    Increase Decrements for Current Node
    

    View Slide

54. PN-Counters
    Merge: Pairwise Maximum per-Node
    

    View Slide

55. Sets
    OR-Set
    

    View Slide

56. OR-Set
    [ Element → ( Tokens , Tombstones ) ]
    

    View Slide

57. Sets
    ORSWOT: Without Tombstones!
    

    View Slide

58. ORSWOT
    VV ⊠ [ Element → [ Dot ] ]
    

    View Slide

59. ORSWOT: Add
    Increment VV
    Add Dot to Element’s Dot
    

    View Slide

60. ORSWOT: Remove
    Leave VV
    Erase Element & Dots
    

    View Slide

61. ORSWOT: Merge
    Element In Both:
    Merge Dots
    Element In Only One:
    Drop when Dominated by Other’s VV
    Keep Otherwise
    

    View Slide

62. Maps
    OR-Map
    

    View Slide

63. Maps
    VV ⊠ ( Field , Type ) → ( [ Dot ] ,
    Value )
    

    View Slide

64. Map: Update
    Apply Update to Value
    Increment VV
    Add Dot to Field’s Dots
    

    View Slide

65. Map: Add
    Like Update, Inserting ⊥
    

    View Slide

66. Map: Remove
    Leave VV
    Erase Field & Dots
    

    View Slide

67. Map: Merge
    Field In Both:
    Merge Dots
    Field In Only One:
    Drop when Dominated by Other’s VV
    Keep Otherwise
    

    View Slide

68. Causality
    Context & CRDTs
    

    View Slide

69. Causality
    Concurrent Remove
    

    View Slide

70. Concurrent Remove
    add(a) remove(a)
    {a,b}
    {a,b} {a,b}
    ?
    ?
    

    View Slide

71. C. Remove: What We Want
    add(a) remove(a)
    {a,b}
    {a,b}
    {a,b}
    {a,b} {a,b}
    Sᵢ₊₁
    Sᵢ
    

    View Slide

72. C. Remove: The Incorrect Solution
    add(a) remove(a)
    {a,b}
    {a,b}
    {b}
    {a,b} {a,b}
    Sᵢ₊₁
    Sᵢ
    Sᵢ₊₁ = op(Sᵢ)
    Sᵢ
    

    View Slide

73. C. Remove: The Correct Solution
    add(a) remove(a)
    {a,b}
    {a,b}
    {a,b}
    {a,b} {a,b}
    Sᵢ₊₁
    Sᵢ
    Sᵢ₊₁ = opᵤ(merge( Sᵢ , opᵣ() ) )
    

    View Slide

74. Causality
    Split Brain
    

    View Slide

75. Split Brain
    remove(a)
    {a,b} {b}
    ?
    ?
    (eventually)
    {a,b}
    

    View Slide

76. Split Brain: What We Want
    remove(a)
    {a,b} {b}
    {b}
    {b}
    (eventually)
    {a,b}
    

    View Slide

77. Split Brain: The Incorrect Solution
    remove(a)
    {a,b} {b}
    {a,b}
    {b}
    Error
    Sᵢ₊₁ = op(Sᵢ)
    (eventually)
    {a,b}
    Sᵢ₊₁
    Sᵢ
    

    View Slide

78. Split Brain: The Correct Solution
    remove(a)
    {b}
    {a,b}
    {b}
    (eventually)
    {a,b}
    {a,b}
    Sᵢ₊₁ = opᵤ(merge( Sᵢ , opᵣ() ) )
    Sᵢ₊₁
    Sᵢ
    

    View Slide

79. QuickCheck
    Is Awesome!
    

    View Slide

80. QuickCheck
    Generalised Checks for CRDTs in riak_dt
    

    View Slide

81. SyncFree
    CRDTs, European Style
    Funded by
    the European Union
    

    View Slide

82. SyncFree
    Dead Actor Pruning
    Funded by
    the European Union
    

    View Slide

83. SyncFree
    Useful Invariants
    Funded by
    the European Union
    

    View Slide

84. References
    I’m a Real Academic!
    

    View Slide

85. http://hal.upmc.fr/inria-00555588
    

    View Slide

86. http://arxiv.org/abs/1210.3368
    

    View Slide

87. http://arxiv.org/abs/1011.5808
    

    View Slide

88. https://github.com/basho/riak_dt
    

    View Slide

89. Recap
    Halp, I forgot something!
    

    View Slide

90. CRDTs
    Data Types
    

    View Slide

91. Sets & Maps
    In 2.0
    

    View Slide

92. Sets & Maps
    “If in doubt, Data is Kept”
    

    View Slide

93. Operations
    Don’t forget Context for Removes
    

    View Slide

94. No 2i, Metadata
    Yet
    

    View Slide

95. Bucket Types
    Except 1.4-Style Counters
    

    View Slide

96. No More Siblings!
    !!!!!!!!111111ONEONEONE
    

    View Slide

97. Magical
    Unless You Know How They Work
    (Sorry)
    

    View Slide

98. Question Time!
    Sam Elliott: sam.elliott@basho.com
    riak-users@lists.basho.com
    riak-core@lists.basho.com
    

    View Slide