$30 off During Our Annual Pro Sale. View Details »

CRDTs: An UPDATE (or just a PUT)

Sam Elliott
October 30, 2013

CRDTs: An UPDATE (or just a PUT)

Riak 2.0's Data Types, presented at RICON West 2013, in SF.

A Few links for the interested:
- Dotted Version Vector Sets, a nice explanation: https://github.com/ricardobcl/Dotted-Version-Vectors#readme
- Our datatype repo: https://github.com/basho/riak_dt
- Comprehensive study of Convergent and Commutative Replicated Data Types (tech report): http://hal.upmc.fr/inria-00555588
- Optimised OR-Set (paper): http://arxiv.org/abs/1210.3368
- Dotted Version Vectors (paper): http://arxiv.org/abs/1011.5808
- Semilattices (wiki): https://en.wikipedia.org/wiki/Semilattice

Sam Elliott

October 30, 2013
Tweet

More Decks by Sam Elliott

Other Decks in Technology

Transcript

  1. CRDTs
    An UPDATE (or maybe just a PUT)

    View Slide

  2. Engineer, Bashtronaut
    Sam Elliott

    View Slide

  3. Not Only Me
    Russell Brown
    Sean Cribbs
    Chris Meiklejohn
    Carlos Baquero (UMinho)

    View Slide

  4. CRDTs
    Co-what Replicated Data Types?!

    View Slide

  5. CRDTs
    Data Types

    View Slide

  6. This Talk
    Usecases, Interface & Implementation

    View Slide

  7. Usecases
    What can I do with them?

    View Slide

  8. Counters
    Since 1.4

    View Slide

  9. Counters
    Increment & Decrement

    View Slide

  10. Counters
    Events!

    View Slide

  11. Burger Counter

    View Slide

  12. Sets
    New in 2.0!

    View Slide

  13. Sets
    Add & Remove Elements

    View Slide

  14. Sets
    Shopping Basket!

    View Slide

  15. Shopping Cart

    View Slide

  16. Maps
    New in 2.0!

    View Slide

  17. Maps
    Values are CRDTs

    View Slide

  18. Maps
    Values are Counters
    Sets

    View Slide

  19. Maps
    Values are Counters
    Sets
    Registers
    Booleans

    View Slide

  20. Maps
    Values are Counters
    Sets
    Registers
    Booleans
    Maps

    View Slide

  21. Maps
    For all your Big Data™ needs!

    View Slide

  22. Address Book

    View Slide

  23. Interface
    How can I do that with them?

    View Slide

  24. Counters
    Updates NOT Idempotent
    (Sorry Pat)

    View Slide

  25. Counters
    Size: O(Actors)

    View Slide

  26. Sets
    { Add | Remove } ~> Add Wins

    View Slide

  27. Sets
    Size: O(Operations) ?

    View Slide

  28. Sets
    Size: O(Actors + Elements)

    View Slide

  29. Maps
    Entries keyed by Name & Type

    View Slide

  30. Maps
    { Update | Remove } ~> Update Wins

    View Slide

  31. Maps
    Size: O(Actors + Entries)

    View Slide

  32. Registers
    Last Write Wins

    View Slide

  33. Registers
    Some Write Loses

    View Slide

  34. Booleans
    { Enable | Disable } ~> Enable

    View Slide

  35. Booleans
    Size: O(Actors)

    View Slide

  36. Operations
    PB & HTTP

    View Slide

  37. Increment(1)
    Add(“Cuttlefish”)
    Update(“name”,
    register,
    Set(“Sam Elliott”))

    View Slide

  38. Operations
    Atomic

    View Slide

  39. Operations
    Unordered

    View Slide

  40. Operations
    Causality

    View Slide

  41. Operations
    Context Required for Correct Removal

    View Slide

  42. Bucket Types
    Required for All Using 2.0 API

    View Slide

  43. Bucket Types
    Counters from 1.4 API Use Default

    View Slide

  44. Not in 2.0
    2i, Custom Metadata

    View Slide

  45. Not in 2.0
    YZ Extractors

    View Slide

  46. Not in 2.0
    Type-level Queries

    View Slide

  47. Implementation
    How do they work underneath?

    View Slide

  48. CRDTs
    Convergent Replicated Data Types

    View Slide

  49. Semilattices
    S



    Values
    Partial Order
    Join
    Bottom

    View Slide

  50. Counters
    PN-Counters

    View Slide

  51. PN-Counters
    [ Node → ( Increments, Decrements ) ]

    View Slide

  52. PN-Counters: Inc
    Increase Increments for Current Node

    View Slide

  53. PN-Counters: Dec
    Increase Decrements for Current Node

    View Slide

  54. PN-Counters
    Merge: Pairwise Maximum per-Node

    View Slide

  55. Sets
    OR-Set

    View Slide

  56. OR-Set
    [ Element → ( Tokens , Tombstones ) ]

    View Slide

  57. Sets
    ORSWOT: Without Tombstones!

    View Slide

  58. ORSWOT
    VV ⊠ [ Element → [ Dot ] ]

    View Slide

  59. ORSWOT: Add
    Increment VV
    Add Dot to Element’s Dot

    View Slide

  60. ORSWOT: Remove
    Leave VV
    Erase Element & Dots

    View Slide

  61. ORSWOT: Merge
    Element In Both:
    Merge Dots
    Element In Only One:
    Drop when Dominated by Other’s VV
    Keep Otherwise

    View Slide

  62. Maps
    OR-Map

    View Slide

  63. Maps
    VV ⊠ ( Field , Type ) → ( [ Dot ] ,
    Value )

    View Slide

  64. Map: Update
    Apply Update to Value
    Increment VV
    Add Dot to Field’s Dots

    View Slide

  65. Map: Add
    Like Update, Inserting ⊥

    View Slide

  66. Map: Remove
    Leave VV
    Erase Field & Dots

    View Slide

  67. Map: Merge
    Field In Both:
    Merge Dots
    Field In Only One:
    Drop when Dominated by Other’s VV
    Keep Otherwise

    View Slide

  68. Causality
    Context & CRDTs

    View Slide

  69. Causality
    Concurrent Remove

    View Slide

  70. Concurrent Remove
    add(a) remove(a)
    {a,b}
    {a,b} {a,b}
    ?
    ?

    View Slide

  71. C. Remove: What We Want
    add(a) remove(a)
    {a,b}
    {a,b}
    {a,b}
    {a,b} {a,b}
    Sᵢ₊₁
    Sᵢ

    View Slide

  72. C. Remove: The Incorrect Solution
    add(a) remove(a)
    {a,b}
    {a,b}
    {b}
    {a,b} {a,b}
    Sᵢ₊₁
    Sᵢ
    Sᵢ₊₁ = op(Sᵢ)
    Sᵢ

    View Slide

  73. C. Remove: The Correct Solution
    add(a) remove(a)
    {a,b}
    {a,b}
    {a,b}
    {a,b} {a,b}
    Sᵢ₊₁
    Sᵢ
    Sᵢ₊₁ = opᵤ(merge( Sᵢ , opᵣ() ) )

    View Slide

  74. Causality
    Split Brain

    View Slide

  75. Split Brain
    remove(a)
    {a,b} {b}
    ?
    ?
    (eventually)
    {a,b}

    View Slide

  76. Split Brain: What We Want
    remove(a)
    {a,b} {b}
    {b}
    {b}
    (eventually)
    {a,b}

    View Slide

  77. Split Brain: The Incorrect Solution
    remove(a)
    {a,b} {b}
    {a,b}
    {b}
    Error
    Sᵢ₊₁ = op(Sᵢ)
    (eventually)
    {a,b}
    Sᵢ₊₁
    Sᵢ

    View Slide

  78. Split Brain: The Correct Solution
    remove(a)
    {b}
    {a,b}
    {b}
    (eventually)
    {a,b}
    {a,b}
    Sᵢ₊₁ = opᵤ(merge( Sᵢ , opᵣ() ) )
    Sᵢ₊₁
    Sᵢ

    View Slide

  79. QuickCheck
    Is Awesome!

    View Slide

  80. QuickCheck
    Generalised Checks for CRDTs in riak_dt

    View Slide

  81. SyncFree
    CRDTs, European Style
    Funded by
    the European Union

    View Slide

  82. SyncFree
    Dead Actor Pruning
    Funded by
    the European Union

    View Slide

  83. SyncFree
    Useful Invariants
    Funded by
    the European Union

    View Slide

  84. References
    I’m a Real Academic!

    View Slide

  85. http://hal.upmc.fr/inria-00555588

    View Slide

  86. http://arxiv.org/abs/1210.3368

    View Slide

  87. http://arxiv.org/abs/1011.5808

    View Slide

  88. https://github.com/basho/riak_dt

    View Slide

  89. Recap
    Halp, I forgot something!

    View Slide

  90. CRDTs
    Data Types

    View Slide

  91. Sets & Maps
    In 2.0

    View Slide

  92. Sets & Maps
    “If in doubt, Data is Kept”

    View Slide

  93. Operations
    Don’t forget Context for Removes

    View Slide

  94. No 2i, Metadata
    Yet

    View Slide

  95. Bucket Types
    Except 1.4-Style Counters

    View Slide

  96. No More Siblings!
    !!!!!!!!111111ONEONEONE

    View Slide

  97. Magical
    Unless You Know How They Work
    (Sorry)

    View Slide

  98. View Slide