Share
Embed
Riak, Ripple, and Rails
Designing Apps for Eventual Consistency
Bryce "BonzoESC" Kerley
June 14, 2012
Transcript
87 slides
-
Riak, Ripple, and Rails Designing Apps for Eventual Consistency Orlando
Ruby Users Group June 14, 2012 Thursday, June 14, 12 -
Riak is Distributed r1n1 r1n2 r1n3 r1n4 r1n5 Nodes are
joined into a cluster Thursday, June 14, 12 -
Riak is Distributed r1n1 r1n2 r1n3 r1n4 r1n5 Nodes are
joined into a cluster All nodes equal Thursday, June 14, 12 -
Riak is Distributed r1n1 r1n2 r1n3 r1n4 r1n5 Nodes are
joined into a cluster All nodes equal Self-sharding Thursday, June 14, 12 -
Riak is Distributed r1n1 r1n2 r1n3 r1n4 r1n5 Nodes are
joined into a cluster All nodes equal Self-sharding Generalizable structure for distributed software Thursday, June 14, 12 -
Riak is Distributed r1n1 r1n2 r1n3 r1n4 r1n5 Nodes are
joined into a cluster All nodes equal Self-sharding Generalizable structure for distributed software http://git.io/riak_core Thursday, June 14, 12 -
Riak is Fault-Tolerant Thursday, June 14, 12 Riak objects are
distributed on a “ring,” shown here as a “pizza,” and split into 2^n slices; strictly speaking, five doesn’t count, but it makes the diagrams easier. -
Riak is Fault-Tolerant r1n1 r1n2 r1n3 r1n4 r1n5 Thursday, June
14, 12 We could store each partition only on a single node, but that’s not fault tolerant; if you lose a node, you lose 1/N your data. -
Riak is Fault-Tolerant r1n1 r1n2 r1n3 r1n4 r1n5 Thursday, June
14, 12 Instead, we put each partition on multiple nodes, three by default. -
Riak is Fault-Tolerant r1n1 r1n2 r1n3 r1n4 r1n5 Thursday, June
14, 12 So when r1n3 blows up, spring, turquoise, and blueberry are still available on two other nodes. -
Riak is Fault-Tolerant r1n1 r1n2 r1n3 r1n4 r1n5 Thursday, June
14, 12 So when r1n3 blows up, spring, turquoise, and blueberry are still available on two other nodes. -
Riak is Fault-Tolerant r1n1 r1n2 r1n3 r1n4 r1n5 Thursday, June
14, 12 And when a new node replaces it, it can bring itself back online based on other replicas. -
Riak is Fault-Tolerant r1n1 r1n2 r1n3 r1n4 r1n5 Thursday, June
14, 12 And when a new node replaces it, it can bring itself back online based on other replicas. -
Riak is Open Source http://git.io/basho http://git.io/riak http://git.io/riak_core http://git.io/riak_kv (mostly in
Erlang) Thursday, June 14, 12 -
Key-Value Buckets containing keys Keys can be given names or
auto-named /riak/orug/YwEFtEC8NUUiEUjuY4mW8WfqtNd Thursday, June 14, 12 -
Search Full text Structure Aware Post-commit hook Painful Re-indexing Thursday,
June 14, 12 Structure aware: it knows how to index fields in JSON, XML, and Erlang terms, besides plain text That it only indexes on save means that if you have to re-index, it has to scan each value -
Secondary Indexes Requires LevelDB backend Indexes specified as metadata My
Favorite Thursday, June 14, 12 You can (and for my HIPAA-sensitive project, we do) index at-rest encrypted data this way One re-indexing caveat: you have to re-specify indexes when you update a record -
Map-Reduce Not Really Querying Aggregate and Filter Results Thursday, June
14, 12 You can (and for my HIPAA-sensitive project, we do) index at-rest encrypted data this way One re-indexing caveat: you have to re-specify indexes when you update a record -
Riak and Ruby Riak-Client http://rubygems.org/gems/riak-client Objects and methods for talking
to Riak Thursday, June 14, 12 -
Riak and Ruby Riak-Client http://rubygems.org/gems/riak-client Objects and methods for talking
to Riak Building block for… Thursday, June 14, 12 -
Riak and Ruby Ripple http://rubygems.org/gems/ripple ActiveModel for Riak Validations, Associations,
and more! Thursday, June 14, 12 -
Query Design owner_id zombie_id hideout_id eaten_at size Brains Thursday, June
14, 12 Let’s start with a zombie-themed example. There’s a bunch of brains wandering around, they get eaten by zombies. We want to easily query a few things about them. -
Query Design owner_id zombie_id zombie_id-hideout_id zombie_id-eaten_at hideout-eaten_at owner_id zombie_id hideout_id
eaten_at size Brains Thursday, June 14, 12 We want to find a given brain by owner (i.e. who got their brain eaten), the zombie that ate it, find what brains a zombie ate where, find what brains a zombie ate on a given day, and what days a given hideout gets raided on. -
owner_id zombie_id hideout_id eaten_at size Brains range query multiple records
one record Query Design owner_id zombie_id zombie_id-hideout_id zombie_id-eaten_at hideout-eaten_at Thursday, June 14, 12 -
class Brain include Ripple::Document property :owner_key, String, presence: true, index:
true property :zombie_key, String, index: true property :hideout_id, Integer property :eaten_at, Time one :owner, using: :stored_key one :zombie, using: :stored_key index :zombie_hideout, String do "#{zombie_key}-#{hideout_id}" end index :zombie_eaten, String do "#{zombie_key}-#{eaten_at}" end index :hideout_eaten, String do owner_id zombie_id hideout_id eaten_at size Brains Query Design Thursday, June 14, 12 For the sake of discussion, Hideouts are stored with ActiveRecord -
class Brain include Ripple::Document property :owner_key, String, presence: true, index:
true property :zombie_key, String, index: true property :hideout_id, Integer property :eaten_at, Time one :owner, using: :stored_key one :zombie, using: :stored_key index :zombie_hideout, String do "#{zombie_key}-#{hideout_id}" end index :zombie_eaten, String do "#{zombie_key}-#{eaten_at}" end index :hideout_eaten, String do "#{hideout_id}-#{eaten_at}" end def hideout Hideout.find hideout_id end def hideout=(hideout) owner_id zombie_id hideout_id eaten_at size Brains Query Design Thursday, June 14, 12 Cardinality is important in multi-field indexes. If we wanted to group by zombie_key, these would be fine -
index :zombie_hideout, String do "#{zombie_key}-#{hideout_id}" end index :zombie_eaten, String do
"#{zombie_key}-#{eaten_at}" end index :hideout_eaten, String do "#{hideout_id}-#{eaten_at}" end def hideout Hideout.find hideout_id end def hideout=(hideout) self.hideout_id = hideout.id end end owner_id zombie_id hideout_id eaten_at size Brains Query Design Thursday, June 14, 12 For the sake of discussion, Hideouts are stored with ActiveRecord -
owner_id zombie_id hideout_id eaten_at size Brains Query Design Thursday, June
14, 12 For the sake of discussion, Hideouts are stored with ActiveRecord -
owner_id zombie_id hideout_id eaten_at size Brains Query Design Thursday, June
14, 12 We’re going to load all our brains into “bs” -
bs = Brain.find_by_index '$bucket', '_' #=> [<Brain:Xo0FJfY8yDHcY5tqQ4icHFL8Vkp owner_key="…" zombie_key="…" hideout_id=5
eaten_at=2012-06-14 03:39:30 UTC>] bs.first.indexes_for_persistence #=> {"owner_key_bin"=>#<Set: {"WuKuWyyi6YPdDGgHcoUuQmu3xQ9"}>, "zombie_key_bin"=>#<Set: {"IzlkD8XbH3efwkXj5n8tAQzzCZp"}>, "zombie_hideout_bin"=>#<Set: {"IzlkD8XbH3efwkXj5n8tAQzzCZp-5"}>, "zombie_eaten_bin"=>#<Set: owner_id zombie_id hideout_id eaten_at size Brains Query Design Thursday, June 14, 12 We’re going to load all our brains into “bs” -
bs = Brain.find_by_index '$bucket', '_' #=> [<Brain:Xo0FJfY8yDHcY5tqQ4icHFL8Vkp owner_key="…" zombie_key="…" hideout_id=5
eaten_at=2012-06-14 03:39:30 UTC>] bs.first.indexes_for_persistence #=> {"owner_key_bin"=>#<Set: {"WuKuWyyi6YPdDGgHcoUuQmu3xQ9"}>, "zombie_key_bin"=>#<Set: {"IzlkD8XbH3efwkXj5n8tAQzzCZp"}>, "zombie_hideout_bin"=>#<Set: {"IzlkD8XbH3efwkXj5n8tAQzzCZp-5"}>, "zombie_eaten_bin"=>#<Set: owner_id zombie_id hideout_id eaten_at size Brains Query Design $bucket is a special 2i Thursday, June 14, 12 We’re going to load all our brains into “bs” -
bs = Brain.find_by_index '$bucket', '_' #=> [<Brain:Xo0FJfY8yDHcY5tqQ4icHFL8Vkp owner_key="…" zombie_key="…" hideout_id=5
eaten_at=2012-06-14 03:39:30 UTC>] bs.first.indexes_for_persistence #=> {"owner_key_bin"=>#<Set: {"WuKuWyyi6YPdDGgHcoUuQmu3xQ9"}>, "zombie_key_bin"=>#<Set: {"IzlkD8XbH3efwkXj5n8tAQzzCZp"}>, "zombie_hideout_bin"=>#<Set: {"IzlkD8XbH3efwkXj5n8tAQzzCZp-5"}>, "zombie_eaten_bin"=>#<Set: owner_id zombie_id hideout_id eaten_at size Brains Query Design Thursday, June 14, 12 We’re going to load all our brains into “bs” -
ow zom hid eat siz Query Design Thursday, June 14,
12 These are the five indexes -
#=> [<Brain:Xo0FJfY8yDHcY5tqQ4icHFL8Vkp owner_key="…" zombie_key="…" hideout_id=5 eaten_at=2012-06-14 03:39:30 UTC>] bs.first.indexes_for_persistence #=>
{"owner_key_bin"=>#<Set: {"WuKuWyyi6YPdDGgHcoUuQmu3xQ9"}>, "zombie_key_bin"=>#<Set: {"IzlkD8XbH3efwkXj5n8tAQzzCZp"}>, "zombie_hideout_bin"=>#<Set: {"IzlkD8XbH3efwkXj5n8tAQzzCZp-5"}>, "zombie_eaten_bin"=>#<Set: {"IzlkD8XbH3efwkXj5n8tAQzzCZp-0000000000133 9645170"}>, "hideout_eaten_bin"=>#<Set: {"5-00000000001339645170"}>} Brain.find_by_index 'hideout_eaten', ow zom hid eat siz Query Design Thursday, June 14, 12 These are the five indexes -
owner_id zombie_id hideout_id eaten_at size Brains Query Design Thursday, June
14, 12 We use a range query to find all the brains that got eaten at hideout 5 between 100 and 200, and we see that 170 counts -
{"IzlkD8XbH3efwkXj5n8tAQzzCZp-5"}>, "zombie_eaten_bin"=>#<Set: {"IzlkD8XbH3efwkXj5n8tAQzzCZp-0000000000133 9645170"}>, "hideout_eaten_bin"=>#<Set: {"5-00000000001339645170"}>} Brain.find_by_index 'hideout_eaten', ("5-00000000001339645100".."5-0000000000133 9645200")
#=> [<Brain:Xo0FJfY8yDHcY5tqQ4icHFL8Vkp owner_key="…" zombie_key="…" hideout_id=5 eaten_at=2012-06-14 03:39:30 UTC>] Brain.find_by_index 'hideout_eaten', ("5-00000000001339645100".."5-0000000000133 9645150") #=> [] owner_id zombie_id hideout_id eaten_at size Brains Query Design Thursday, June 14, 12 We use a range query to find all the brains that got eaten at hideout 5 between 100 and 200, and we see that 170 counts -
owner_id zombie_id hideout_id eaten_at size Brains Query Design Thursday, June
14, 12 And there wasn’t anything between 100 and 150; 170 is outside of that range. -
("5-00000000001339645100".."5-0000000000133 9645200") #=> [<Brain:Xo0FJfY8yDHcY5tqQ4icHFL8Vkp owner_key="…" zombie_key="…" hideout_id=5 eaten_at=2012-06-14 03:39:30 UTC>]
Brain.find_by_index 'hideout_eaten', ("5-00000000001339645100".."5-0000000000133 9645150") #=> [] owner_id zombie_id hideout_id eaten_at size Brains Query Design Thursday, June 14, 12 And there wasn’t anything between 100 and 150; 170 is outside of that range. -
owner_id zombie_id hideout_id eaten_at size Brains Query Design Thursday, June
14, 12 And there wasn’t anything between 100 and 150; 170 is outside of that range. -
Hooking Up Rails Add ripple to Gemfile Add config/ripple.yml Put
Ripple-backed models in app/models Thursday, June 14, 12 -
Hooking Up Rails Add ripple to Gemfile source 'https://rubygems.org' gem
'rails', '3.2.5' gem 'sqlite3' gem 'ripple', '~> 1.0.0.beta2' gem 'haml' Thursday, June 14, 12 -
Hooking Up Rails Add config/ripple.yml development: host: localhost http_port: 8091
pb_port: 8081 Thursday, June 14, 12 -
Hooking Up Rails Put Ripple-backed models in app/models class Zombie
include Ripple::Document end Thursday, June 14, 12 -
Eventually Consistent Counters http://bit.ly/prime-cheat “You can’t use prime numbers to
cheat on CRDTs” Thursday, June 14, 12 -
Photos http://flic.kr/p/5NfKi2 - Satellite Beach http://flic.kr/p/9kTSCe - USF Bull http://flic.kr/p/53K5JV
- Coconut Grove http://flic.kr/p/4r3Vjk - Steven Bristol Thursday, June 14, 12