Riak CRDTs

Riak CRDTs

Gordon Guthrie talks on CRDTs May 28th 2015 Meetup.

B4c5e3be1957239def212083b3f50916?s=128

Datageeks Paris

June 01, 2015
Tweet

Transcript

  1. Convergent Replicated Data Types in Riak 2.0 Gordon Guthrie Senior

    Software Engineer - Basho
  2. What is Riak?

  3. What is it for? • Riak is a highly-available, low-latency

    KV store • it is an open-source implementation of the Dynamo paper from Amazon • Dynamo underpins the Amazon shopping cart
  4. Why does it matter? • availability matters • latency matters

    • Amazon found every 100ms of latency cost them 1% in sales • Google found an extra 0.5 seconds in search page generation time dropped traffic by 20%
  5. Riak Availability

  6. The Ring

  7. Mapped To Machines Physical Machine 1 Physical Machine 2 Physical

    Machine 3 Physical Machine 4 Physical Machine 5
  8. Machines Go Down Physical Machine 1 Physical Machine 2 Physical

    Machine 3 Physical Machine 4 Physical Machine 5
  9. And Come Back Physical Machine 1 Physical Machine 2 Physical

    Machine 3 Physical Machine 4 Physical Machine 5
  10. but that’s not enough That’s how the data survives a

    box dying
  11. What happens if the network partitions

  12. CAP Theorem

  13. The Cap Theorem Pick 2 Consistency Availability Partition! Tolerance Riak

  14. Partitions Happen • because Riak is a highly-available data store

    it will continue to store your data • but your data will not be consistent
  15. AP Physical Machine 1 Physical Machine 2 Physical Machine 3

    Physical Machine 4 Physical Machine 5 Partition Val 1 Val 2
  16. And Come Back Physical Machine 1 Physical Machine 2 Physical

    Machine 3 Physical Machine 4 Physical Machine 5
  17. Now you have siblings Physical Machine 1 Physical Machine 2

    Physical Machine 3 Physical Machine 4 Physical Machine 5 {Val 1, Val 2}
  18. Siblings the developer’s problem that Riak 2.0 is trying to

    fix
  19. Sibling Resolution • Developers hate siblings • Developers hate resolving

    siblings • Developers play Timestamp Roulette and go for Last Write Wins • but with intercontinental lag, NTP failure, clock skew and latency sometimes Timestamps kill your data
  20. Google’s View • “Designing applications to cope with concurrency anomalies

    in their data is very error-prone, time- consuming, and ultimately not worth the performance gains.” • “We have a lot of experience with eventual consistency systems at Google.” • “We find developers spend a significant fraction of their time building extremely complex and error-prone mechanisms to cope with eventual consistency”
  21. The Partition Cycle Consistent Available Available Partitioned Not Consistent Available

    (Eventually) Consistent Available partitioned unpartitioned Developer! Hell Developer! Heaven CRDTs
  22. CRDTs Consistent Replicated Data Types

  23. Understanding CRDTs • Not going to go into a lot

    of detail • Based on the idea of Lamport Vector Clocks • A Lamport Vector clock allows an ‘actor’ in a distributed data system to say 2 things: • this is when I changed this data • and when I changed it I knew all the things every other actor had done upto these times • Its a vector clock because it has a separate clock for every actor
  24. Where Are The Actors? • Actors can be clients-side or

    server-side or both • Riak is implementing an architecture that allows users to use server-side CRDTs • to avoid the risk of ‘actor’ explosion
  25. Client Side Actors • Alice’s Laptop • Alice’s phone •

    Alice’s iPad • Alice’s daughter’s iPad (her battery is flat) • Bob’s new laptop • Bob’s old laptop • etc, etc
  26. None
  27. How They Work • Lets take a simple example with

    two actions: • add an item to a shopping cart • empty the shopping cart • these actions don’t commute: • add item then clear gives an empty cart • empty cart then add item gives a cart with one item in it
  28. How They Work Consistent Available Available Partitioned Not Consistent Available

    (Eventually) Consistent Available partitioned unpartitioned CRDTs Clear Add! Item [{a, }] [{b, }] {[{a, }], [{b, }]} The ‘actor’ clearing the shopping cart didn’t know there was anything in the cart Therefore there should be something in the cart after merging the two updates
  29. How They Work Consistent Available Available Partitioned Not Consistent Available

    (Eventually) Consistent Available partitioned unpartitioned CRDTs Clear Add! Item [{a, }] [{a, },{b, }] {[{a, }], [{a, },{b, }]} The ‘actor’ clearing the shopping cart knew there was something in the cart Therefore there should be nothing in the cart after merging the two updates
  30. Scary Maths Stuff • Joined Semi-Lattices that have a Bottom

    and a Least Upper Bound! • that have defined properties involving Associativity, Commutativity, Idempotency • with a Merge Function
  31. References Your essential CRDT reading list http://christophermeiklejohn.com/crdt/2014/07/22/readings-in-crdts.html

  32. CRDTs in Riak 2.0

  33. Native Eventually Consistent Data Types • Flags • Registers •

    Counters • Sets • Maps • can contain Flags, Registers, Counter and Maps • the basis of composable data types
  34. Flags • Can have one of two values: • enabled

    • disabled • Operations: • enable • disable • Examples - use like Booleans • has a tweet been sent? • has the customer signed up to a pricing plan?
  35. Registers • Named Binaries that have contents which have a

    value • Operations: • store new value • Examples • store “My New Post” in the field “Blog Post Title”
  36. Counters • Contain a number which can be incremented or

    decremented • Operations • increment • decrement • Examples • the number of “likes” in Facebook • the number of Twitter followers
  37. Sets • Collections of unique binaries • Operations • add

    an element • remove an element • add a list of elements • remove a list of elements • Examples • shopping cart
  38. Maps • Maps are like hash maps • Operations •

    add a named Flag, Register, Counter, Set or Map • remove a named Flag, Register, Counter, Set or Map • pass through an op for an element in the Map • Example • the Map is used to compose the data model
  39. Questions? Fin