CRDTs: An UPDATE (or just a PUT)

334a587ef1901e2d48dfd15f90522496?s=47 Sam Elliott
October 30, 2013

CRDTs: An UPDATE (or just a PUT)

Riak 2.0's Data Types, presented at RICON West 2013, in SF.

A Few links for the interested:
- Dotted Version Vector Sets, a nice explanation: https://github.com/ricardobcl/Dotted-Version-Vectors#readme
- Our datatype repo: https://github.com/basho/riak_dt
- Comprehensive study of Convergent and Commutative Replicated Data Types (tech report): http://hal.upmc.fr/inria-00555588
- Optimised OR-Set (paper): http://arxiv.org/abs/1210.3368
- Dotted Version Vectors (paper): http://arxiv.org/abs/1011.5808
- Semilattices (wiki): https://en.wikipedia.org/wiki/Semilattice

334a587ef1901e2d48dfd15f90522496?s=128

Sam Elliott

October 30, 2013
Tweet

Transcript

  1. CRDTs An UPDATE (or maybe just a PUT)

  2. Engineer, Bashtronaut Sam Elliott

  3. Not Only Me Russell Brown Sean Cribbs Chris Meiklejohn Carlos

    Baquero (UMinho)
  4. CRDTs Co-what Replicated Data Types?!

  5. CRDTs Data Types

  6. This Talk Usecases, Interface & Implementation

  7. Usecases What can I do with them?

  8. Counters Since 1.4

  9. Counters Increment & Decrement

  10. Counters Events!

  11. Burger Counter

  12. Sets New in 2.0!

  13. Sets Add & Remove Elements

  14. Sets Shopping Basket!

  15. Shopping Cart

  16. Maps New in 2.0!

  17. Maps Values are CRDTs

  18. Maps Values are Counters Sets

  19. Maps Values are Counters Sets Registers Booleans

  20. Maps Values are Counters Sets Registers Booleans Maps

  21. Maps For all your Big Data™ needs!

  22. Address Book

  23. Interface How can I do that with them?

  24. Counters Updates NOT Idempotent (Sorry Pat)

  25. Counters Size: O(Actors)

  26. Sets { Add | Remove } ~> Add Wins

  27. Sets Size: O(Operations) ?

  28. Sets Size: O(Actors + Elements)

  29. Maps Entries keyed by Name & Type

  30. Maps { Update | Remove } ~> Update Wins

  31. Maps Size: O(Actors + Entries)

  32. Registers Last Write Wins

  33. Registers Some Write Loses

  34. Booleans { Enable | Disable } ~> Enable

  35. Booleans Size: O(Actors)

  36. Operations PB & HTTP

  37. Increment(1) Add(“Cuttlefish”) Update(“name”, register, Set(“Sam Elliott”))

  38. Operations Atomic

  39. Operations Unordered

  40. Operations Causality

  41. Operations Context Required for Correct Removal

  42. Bucket Types Required for All Using 2.0 API

  43. Bucket Types Counters from 1.4 API Use Default

  44. Not in 2.0 2i, Custom Metadata

  45. Not in 2.0 YZ Extractors

  46. Not in 2.0 Type-level Queries

  47. Implementation How do they work underneath?

  48. CRDTs Convergent Replicated Data Types

  49. Semilattices S ≤ ∨ ⊥ Values Partial Order Join Bottom

  50. Counters PN-Counters

  51. PN-Counters [ Node → ( Increments, Decrements ) ]

  52. PN-Counters: Inc Increase Increments for Current Node

  53. PN-Counters: Dec Increase Decrements for Current Node

  54. PN-Counters Merge: Pairwise Maximum per-Node

  55. Sets OR-Set

  56. OR-Set [ Element → ( Tokens , Tombstones ) ]

  57. Sets ORSWOT: Without Tombstones!

  58. ORSWOT VV ⊠ [ Element → [ Dot ] ]

  59. ORSWOT: Add Increment VV Add Dot to Element’s Dot

  60. ORSWOT: Remove Leave VV Erase Element & Dots

  61. ORSWOT: Merge Element In Both: Merge Dots Element In Only

    One: Drop when Dominated by Other’s VV Keep Otherwise
  62. Maps OR-Map

  63. Maps VV ⊠ ( Field , Type ) → (

    [ Dot ] , Value )
  64. Map: Update Apply Update to Value Increment VV Add Dot

    to Field’s Dots
  65. Map: Add Like Update, Inserting ⊥

  66. Map: Remove Leave VV Erase Field & Dots

  67. Map: Merge Field In Both: Merge Dots Field In Only

    One: Drop when Dominated by Other’s VV Keep Otherwise
  68. Causality Context & CRDTs

  69. Causality Concurrent Remove

  70. Concurrent Remove add(a) remove(a) {a,b} {a,b} {a,b} ? ?

  71. C. Remove: What We Want add(a) remove(a) {a,b} {a,b} {a,b}

    {a,b} {a,b} Sᵢ₊₁ Sᵢ
  72. C. Remove: The Incorrect Solution add(a) remove(a) {a,b} {a,b} {b}

    {a,b} {a,b} Sᵢ₊₁ Sᵢ Sᵢ₊₁ = op(Sᵢ) Sᵢ
  73. C. Remove: The Correct Solution add(a) remove(a) {a,b} {a,b} {a,b}

    {a,b} {a,b} Sᵢ₊₁ Sᵢ Sᵢ₊₁ = opᵤ(merge( Sᵢ , opᵣ() ) )
  74. Causality Split Brain

  75. Split Brain remove(a) {a,b} {b} ? ? (eventually) {a,b}

  76. Split Brain: What We Want remove(a) {a,b} {b} {b} {b}

    (eventually) {a,b}
  77. Split Brain: The Incorrect Solution remove(a) {a,b} {b} {a,b} {b}

    Error Sᵢ₊₁ = op(Sᵢ) (eventually) {a,b} Sᵢ₊₁ Sᵢ
  78. Split Brain: The Correct Solution remove(a) {b} {a,b} {b} (eventually)

    {a,b} {a,b} Sᵢ₊₁ = opᵤ(merge( Sᵢ , opᵣ() ) ) Sᵢ₊₁ Sᵢ
  79. QuickCheck Is Awesome!

  80. QuickCheck Generalised Checks for CRDTs in riak_dt

  81. SyncFree CRDTs, European Style Funded by the European Union

  82. SyncFree Dead Actor Pruning Funded by the European Union

  83. SyncFree Useful Invariants Funded by the European Union

  84. References I’m a Real Academic!

  85. http://hal.upmc.fr/inria-00555588

  86. http://arxiv.org/abs/1210.3368

  87. http://arxiv.org/abs/1011.5808

  88. https://github.com/basho/riak_dt

  89. Recap Halp, I forgot something!

  90. CRDTs Data Types

  91. Sets & Maps In 2.0

  92. Sets & Maps “If in doubt, Data is Kept”

  93. Operations Don’t forget Context for Removes

  94. No 2i, Metadata Yet

  95. Bucket Types Except 1.4-Style Counters

  96. No More Siblings! !!!!!!!!111111ONEONEONE

  97. Magical Unless You Know How They Work (Sorry)

  98. Question Time! Sam Elliott: sam.elliott@basho.com riak-users@lists.basho.com riak-core@lists.basho.com