Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Riak, Ripple, and Rails

Riak, Ripple, and Rails

Designing Apps for Eventual Consistency

Bryce "BonzoESC" Kerley

June 14, 2012
Tweet

More Decks by Bryce "BonzoESC" Kerley

Other Decks in Programming

Transcript

  1. Riak, Ripple, and Rails Designing Apps for Eventual Consistency Orlando

    Ruby Users Group June 14, 2012 Thursday, June 14, 12
  2. Riak is Distributed r1n1 r1n2 r1n3 r1n4 r1n5 Nodes are

    joined into a cluster Thursday, June 14, 12
  3. Riak is Distributed r1n1 r1n2 r1n3 r1n4 r1n5 Nodes are

    joined into a cluster All nodes equal Thursday, June 14, 12
  4. Riak is Distributed r1n1 r1n2 r1n3 r1n4 r1n5 Nodes are

    joined into a cluster All nodes equal Self-sharding Thursday, June 14, 12
  5. Riak is Distributed r1n1 r1n2 r1n3 r1n4 r1n5 Nodes are

    joined into a cluster All nodes equal Self-sharding Generalizable structure for distributed software Thursday, June 14, 12
  6. Riak is Distributed r1n1 r1n2 r1n3 r1n4 r1n5 Nodes are

    joined into a cluster All nodes equal Self-sharding Generalizable structure for distributed software http://git.io/riak_core Thursday, June 14, 12
  7. Riak is Fault-Tolerant Thursday, June 14, 12 Riak objects are

    distributed on a “ring,” shown here as a “pizza,” and split into 2^n slices; strictly speaking, five doesn’t count, but it makes the diagrams easier.
  8. Riak is Fault-Tolerant r1n1 r1n2 r1n3 r1n4 r1n5 Thursday, June

    14, 12 We could store each partition only on a single node, but that’s not fault tolerant; if you lose a node, you lose 1/N your data.
  9. Riak is Fault-Tolerant r1n1 r1n2 r1n3 r1n4 r1n5 Thursday, June

    14, 12 Instead, we put each partition on multiple nodes, three by default.
  10. Riak is Fault-Tolerant r1n1 r1n2 r1n3 r1n4 r1n5 Thursday, June

    14, 12 So when r1n3 blows up, spring, turquoise, and blueberry are still available on two other nodes.
  11. Riak is Fault-Tolerant r1n1 r1n2 r1n3 r1n4 r1n5 Thursday, June

    14, 12 So when r1n3 blows up, spring, turquoise, and blueberry are still available on two other nodes.
  12. Riak is Fault-Tolerant r1n1 r1n2 r1n3 r1n4 r1n5 Thursday, June

    14, 12 And when a new node replaces it, it can bring itself back online based on other replicas.
  13. Riak is Fault-Tolerant r1n1 r1n2 r1n3 r1n4 r1n5 Thursday, June

    14, 12 And when a new node replaces it, it can bring itself back online based on other replicas.
  14. Key-Value Buckets containing keys Keys can be given names or

    auto-named /riak/orug/YwEFtEC8NUUiEUjuY4mW8WfqtNd Thursday, June 14, 12
  15. Search Full text Structure Aware Post-commit hook Painful Re-indexing Thursday,

    June 14, 12 Structure aware: it knows how to index fields in JSON, XML, and Erlang terms, besides plain text That it only indexes on save means that if you have to re-index, it has to scan each value
  16. Secondary Indexes Requires LevelDB backend Indexes specified as metadata My

    Favorite Thursday, June 14, 12 You can (and for my HIPAA-sensitive project, we do) index at-rest encrypted data this way One re-indexing caveat: you have to re-specify indexes when you update a record
  17. Map-Reduce Not Really Querying Aggregate and Filter Results Thursday, June

    14, 12 You can (and for my HIPAA-sensitive project, we do) index at-rest encrypted data this way One re-indexing caveat: you have to re-specify indexes when you update a record
  18. Query Design owner_id zombie_id hideout_id eaten_at size Brains Thursday, June

    14, 12 Let’s start with a zombie-themed example. There’s a bunch of brains wandering around, they get eaten by zombies. We want to easily query a few things about them.
  19. Query Design owner_id zombie_id zombie_id-hideout_id zombie_id-eaten_at hideout-eaten_at owner_id zombie_id hideout_id

    eaten_at size Brains Thursday, June 14, 12 We want to find a given brain by owner (i.e. who got their brain eaten), the zombie that ate it, find what brains a zombie ate where, find what brains a zombie ate on a given day, and what days a given hideout gets raided on.
  20. owner_id zombie_id hideout_id eaten_at size Brains range query multiple records

    one record Query Design owner_id zombie_id zombie_id-hideout_id zombie_id-eaten_at hideout-eaten_at Thursday, June 14, 12
  21. class Brain include Ripple::Document property :owner_key, String, presence: true, index:

    true property :zombie_key, String, index: true property :hideout_id, Integer property :eaten_at, Time one :owner, using: :stored_key one :zombie, using: :stored_key index :zombie_hideout, String do "#{zombie_key}-#{hideout_id}" end index :zombie_eaten, String do "#{zombie_key}-#{eaten_at}" end index :hideout_eaten, String do owner_id zombie_id hideout_id eaten_at size Brains Query Design Thursday, June 14, 12 For the sake of discussion, Hideouts are stored with ActiveRecord
  22. class Brain include Ripple::Document property :owner_key, String, presence: true, index:

    true property :zombie_key, String, index: true property :hideout_id, Integer property :eaten_at, Time one :owner, using: :stored_key one :zombie, using: :stored_key index :zombie_hideout, String do "#{zombie_key}-#{hideout_id}" end index :zombie_eaten, String do "#{zombie_key}-#{eaten_at}" end index :hideout_eaten, String do "#{hideout_id}-#{eaten_at}" end def hideout Hideout.find hideout_id end def hideout=(hideout) owner_id zombie_id hideout_id eaten_at size Brains Query Design Thursday, June 14, 12 Cardinality is important in multi-field indexes. If we wanted to group by zombie_key, these would be fine
  23. index :zombie_hideout, String do "#{zombie_key}-#{hideout_id}" end index :zombie_eaten, String do

    "#{zombie_key}-#{eaten_at}" end index :hideout_eaten, String do "#{hideout_id}-#{eaten_at}" end def hideout Hideout.find hideout_id end def hideout=(hideout) self.hideout_id = hideout.id end end owner_id zombie_id hideout_id eaten_at size Brains Query Design Thursday, June 14, 12 For the sake of discussion, Hideouts are stored with ActiveRecord
  24. owner_id zombie_id hideout_id eaten_at size Brains Query Design Thursday, June

    14, 12 For the sake of discussion, Hideouts are stored with ActiveRecord
  25. owner_id zombie_id hideout_id eaten_at size Brains Query Design Thursday, June

    14, 12 We’re going to load all our brains into “bs”
  26. bs = Brain.find_by_index '$bucket', '_' #=> [<Brain:Xo0FJfY8yDHcY5tqQ4icHFL8Vkp owner_key="…" zombie_key="…" hideout_id=5

    eaten_at=2012-06-14 03:39:30 UTC>] bs.first.indexes_for_persistence #=> {"owner_key_bin"=>#<Set: {"WuKuWyyi6YPdDGgHcoUuQmu3xQ9"}>, "zombie_key_bin"=>#<Set: {"IzlkD8XbH3efwkXj5n8tAQzzCZp"}>, "zombie_hideout_bin"=>#<Set: {"IzlkD8XbH3efwkXj5n8tAQzzCZp-5"}>, "zombie_eaten_bin"=>#<Set: owner_id zombie_id hideout_id eaten_at size Brains Query Design Thursday, June 14, 12 We’re going to load all our brains into “bs”
  27. bs = Brain.find_by_index '$bucket', '_' #=> [<Brain:Xo0FJfY8yDHcY5tqQ4icHFL8Vkp owner_key="…" zombie_key="…" hideout_id=5

    eaten_at=2012-06-14 03:39:30 UTC>] bs.first.indexes_for_persistence #=> {"owner_key_bin"=>#<Set: {"WuKuWyyi6YPdDGgHcoUuQmu3xQ9"}>, "zombie_key_bin"=>#<Set: {"IzlkD8XbH3efwkXj5n8tAQzzCZp"}>, "zombie_hideout_bin"=>#<Set: {"IzlkD8XbH3efwkXj5n8tAQzzCZp-5"}>, "zombie_eaten_bin"=>#<Set: owner_id zombie_id hideout_id eaten_at size Brains Query Design $bucket is a special 2i Thursday, June 14, 12 We’re going to load all our brains into “bs”
  28. bs = Brain.find_by_index '$bucket', '_' #=> [<Brain:Xo0FJfY8yDHcY5tqQ4icHFL8Vkp owner_key="…" zombie_key="…" hideout_id=5

    eaten_at=2012-06-14 03:39:30 UTC>] bs.first.indexes_for_persistence #=> {"owner_key_bin"=>#<Set: {"WuKuWyyi6YPdDGgHcoUuQmu3xQ9"}>, "zombie_key_bin"=>#<Set: {"IzlkD8XbH3efwkXj5n8tAQzzCZp"}>, "zombie_hideout_bin"=>#<Set: {"IzlkD8XbH3efwkXj5n8tAQzzCZp-5"}>, "zombie_eaten_bin"=>#<Set: owner_id zombie_id hideout_id eaten_at size Brains Query Design Thursday, June 14, 12 We’re going to load all our brains into “bs”
  29. #=> [<Brain:Xo0FJfY8yDHcY5tqQ4icHFL8Vkp owner_key="…" zombie_key="…" hideout_id=5 eaten_at=2012-06-14 03:39:30 UTC>] bs.first.indexes_for_persistence #=>

    {"owner_key_bin"=>#<Set: {"WuKuWyyi6YPdDGgHcoUuQmu3xQ9"}>, "zombie_key_bin"=>#<Set: {"IzlkD8XbH3efwkXj5n8tAQzzCZp"}>, "zombie_hideout_bin"=>#<Set: {"IzlkD8XbH3efwkXj5n8tAQzzCZp-5"}>, "zombie_eaten_bin"=>#<Set: {"IzlkD8XbH3efwkXj5n8tAQzzCZp-0000000000133 9645170"}>, "hideout_eaten_bin"=>#<Set: {"5-00000000001339645170"}>} Brain.find_by_index 'hideout_eaten', ow zom hid eat siz Query Design Thursday, June 14, 12 These are the five indexes
  30. owner_id zombie_id hideout_id eaten_at size Brains Query Design Thursday, June

    14, 12 We use a range query to find all the brains that got eaten at hideout 5 between 100 and 200, and we see that 170 counts
  31. {"IzlkD8XbH3efwkXj5n8tAQzzCZp-5"}>, "zombie_eaten_bin"=>#<Set: {"IzlkD8XbH3efwkXj5n8tAQzzCZp-0000000000133 9645170"}>, "hideout_eaten_bin"=>#<Set: {"5-00000000001339645170"}>} Brain.find_by_index 'hideout_eaten', ("5-00000000001339645100".."5-0000000000133 9645200")

    #=> [<Brain:Xo0FJfY8yDHcY5tqQ4icHFL8Vkp owner_key="…" zombie_key="…" hideout_id=5 eaten_at=2012-06-14 03:39:30 UTC>] Brain.find_by_index 'hideout_eaten', ("5-00000000001339645100".."5-0000000000133 9645150") #=> [] owner_id zombie_id hideout_id eaten_at size Brains Query Design Thursday, June 14, 12 We use a range query to find all the brains that got eaten at hideout 5 between 100 and 200, and we see that 170 counts
  32. owner_id zombie_id hideout_id eaten_at size Brains Query Design Thursday, June

    14, 12 And there wasn’t anything between 100 and 150; 170 is outside of that range.
  33. ("5-00000000001339645100".."5-0000000000133 9645200") #=> [<Brain:Xo0FJfY8yDHcY5tqQ4icHFL8Vkp owner_key="…" zombie_key="…" hideout_id=5 eaten_at=2012-06-14 03:39:30 UTC>]

    Brain.find_by_index 'hideout_eaten', ("5-00000000001339645100".."5-0000000000133 9645150") #=> [] owner_id zombie_id hideout_id eaten_at size Brains Query Design Thursday, June 14, 12 And there wasn’t anything between 100 and 150; 170 is outside of that range.
  34. owner_id zombie_id hideout_id eaten_at size Brains Query Design Thursday, June

    14, 12 And there wasn’t anything between 100 and 150; 170 is outside of that range.
  35. Hooking Up Rails Add ripple to Gemfile Add config/ripple.yml Put

    Ripple-backed models in app/models Thursday, June 14, 12
  36. Hooking Up Rails Add ripple to Gemfile source 'https://rubygems.org' gem

    'rails', '3.2.5' gem 'sqlite3' gem 'ripple', '~> 1.0.0.beta2' gem 'haml' Thursday, June 14, 12
  37. Hooking Up Rails Put Ripple-backed models in app/models class Zombie

    include Ripple::Document end Thursday, June 14, 12
  38. Photos http://flic.kr/p/5NfKi2 - Satellite Beach http://flic.kr/p/9kTSCe - USF Bull http://flic.kr/p/53K5JV

    - Coconut Grove http://flic.kr/p/4r3Vjk - Steven Bristol Thursday, June 14, 12