Redis: Persistence Power

A practical, use case approach to looking into Redis.

Given at RubyMidwest 2010. http://rubymidwest.com/

Transcript

1. Redis: Persistence Power Nick Quaranto / @qrush / nick@quaran.to Tuesday, August 10, 2010
2. What is Redis? “advanced key-value store” REmote DIctionary Server data structures server Tuesday, August 10, 2010
3. YOUR APP Tuesday, August 10, 2010
4. YOUR APP REDIS Tuesday, August 10, 2010
5. the basics persist data as you think of it in memory, sync to disk in background ridiculously fast master-slave replication keys = strings, value = data structures Tuesday, August 10, 2010
6. http://try.redis-db.com Tuesday, August 10, 2010
7. use it: redis-cli % ./redis-cli SET user:1:name qrush OK % ./redis-cli GET user:1:name "qrush" Tuesday, August 10, 2010
8. use it: redis-rb % gem install redis % irb -rubygems -rredis >> $redis = Redis.new => #<Redis client v2.0.3 connected... >> $redis.set "user:1:name", "qrush" => "OK" >> $redis.get "user:1:name" => "qrush" Tuesday, August 10, 2010
9. FEATURE SWITCHES STRINGS based on http://github.com/blog/677 http://github.com/bvandenbos/redis_feature_control Tuesday, August 10, 2010
10. Rediswitch.features << :super_secret Rediswitch.features << :payment_gateway Rediswitch.features << :twitter if Rediswitch.enabled?(:twitter) # post to twitter else # failwhale ahoy! end Tuesday, August 10, 2010
11. begin # take some money rescue PaymentGateway::TotallyDown => ohno Rediswitch.disable(:payment_gateway) # notify the troops end Tuesday, August 10, 2010
12. class Rediswitch def self.enabled?(feature) $redis.exists(feature) end def self.enable(feature) $redis.incr(feature) end def self.disable(feature) $redis.del(feature) end end Tuesday, August 10, 2010
13. feature switch lessons the real win: no-deploy configuration fast enough to be transparent next step: separate users into buckets with sets http://github.com/jamesgolick/rollout Tuesday, August 10, 2010
14. RATE LIMITER STRINGS soon to be in place at http://hoptoadapp.com Tuesday, August 10, 2010
15. class Choker def restrict? track count_for > 60 end end Tuesday, August 10, 2010
16. class Choker def count_for $memcache.get(key, true).to_i end end Tuesday, August 10, 2010
17. class Choker def track if !$memcache.get(key, true) $memcache.add(key, "0", 1.minute.from_now, true) end $memcache.incr(key) end end Tuesday, August 10, 2010
18. class Choker def track if !$redis.exists(key) $redis.setex(key, 60, 0) end $redis.incr(key) end end Tuesday, August 10, 2010
19. class Choker def count_for $redis.get(key).to_i end end Tuesday, August 10, 2010
20. rate limiter lessons expire semantics are changing in redis 2.2 benchmark the crap out of it could use a sorted set instead of strings Tuesday, August 10, 2010
21. API USAGE LOGGING STRINGS SORTED SETS based off http://www.production-hacks.com/2010/07/10/redis-api-access-logger/ Tuesday, August 10, 2010
22. # one way to do it class ActionHit < ActiveRecord::Base # t.string :controller # t.string :action # t.integer :counter end class UserHit < ActiveRecord::Base # t.string :controller_action # t.integer :user_id # t.integer :counter end Tuesday, August 10, 2010
23. # for all controllers { "statuses#update" => 1410, "users#create" => 931, "home#index" => 2936 } # users hitting an action { "101" => 42, "102" => 13, "103" => 34 } Tuesday, August 10, 2010
24. class StatusesController < ApplicationController def update $redis.incr "statuses#update" $redis.incr "statuses#update:#{user.id}" end end Tuesday, August 10, 2010
25. class StatusesController < ApplicationController def update key = "statuses#update" $redis.zincrby "actions", 1, key $redis.zincrby "users:#{key}", 1, user.id end end Tuesday, August 10, 2010
26. # hits for a specific user >> $redis.zscore "users:statuses#update", 1001 => 42 # list all the controller actions, sorted >> $redis.zrevrange "actions", 0, -1, :with_scores => true => ["home#index", "2936", "statuses#update", "1410", "users#create", "931"] Tuesday, August 10, 2010
27. api usage logging lessons sorted set = high score list bad at historical usage, trends good for a simple heartbeat or pulse Tuesday, August 10, 2010
28. JOB QUEUE LISTS based on http://github.com/defunkt/resque Tuesday, August 10, 2010
29. class Staple @queue = :default def self.perform(post_id, tempfile) # complex image resizing, cropping end end Tuesday, August 10, 2010
30. class Post < ActiveRecord::Base after_save :process_with_stapler def process_with_stapler Resque.enqueue(Staple, self.id, @tempfile) end end Tuesday, August 10, 2010
31. module Resque extend self def push(queue, item) redis.rpush "q:#{queue}", encode(item) end def pop(queue) decode redis.lpop("q:#{queue}") end end Tuesday, August 10, 2010
32. class Resque::Worker def work loop do if job = Resque.pop(queue) job.perform else sleep 5 end end end end Tuesday, August 10, 2010
33. module Resque extend self def bpop(queue) decode redis.blpop("q:#{queue}") end end Tuesday, August 10, 2010
34. class Resque::Worker def work loop do job = Resque.bpop(queue) job.perform end end end Tuesday, August 10, 2010
35. job queue lessons guaranteed atomic actions, no row locking blocking commands simplify daemons many more queue commands in redis itself! Tuesday, August 10, 2010
36. GLOBAL ERRORS SETS MULTI/EXEC a new feature at http://hoptoadapp.com Tuesday, August 10, 2010
37. # text :globals, :default => '', :null => false class Project < ActiveRecord::Base def has_global?(name) @globals ||= globals.gsub(/,/,' ').split @globals.include?(name) end end Tuesday, August 10, 2010
38. # MORE TABLES!!!! class Global < ActiveRecord::Base belongs_to :project end class Project < ActiveRecord::Base has_many :globals end Tuesday, August 10, 2010
39. # Project#global_errors ["MySQL::Error", "MemCache::Error", "Net::HTTPFatalError"] Tuesday, August 10, 2010
40. class Project < ActiveRecord::Base def global_key "project-#{id}-globals" end def has_global?(name) $redis.sismember(global_key, name) end end Tuesday, August 10, 2010
41. class Project < ActiveRecord::Base after_save :save_globals def save_globals $redis.del global_key @globals.each do |g| $redis.sadd global_key, g end end end Tuesday, August 10, 2010
42. [Mysql::Error, MemCache::Error, Net::HTTPFatalError] SISMEMBER SISMEMBER Tuesday, August 10, 2010
43. [] DEL Tuesday, August 10, 2010
44. [Mysql::Error] DEL SADD Tuesday, August 10, 2010
45. [Mysql::Error, OpenURI::HTTPError] DEL SADD SADD Tuesday, August 10, 2010
46. [Mysql::Error, OpenURI::HTTPError] DEL SADD SADD SISMEMBER [] Tuesday, August 10, 2010
47. [Mysql::Error, OpenURI::HTTPError] MULTI EXEC DEL SADD SADD SISMEMBER Tuesday, August 10, 2010
48. class Project < ActiveRecord::Base after_save :save_globals def save_globals $redis.multi do $redis.del global_key @globals.each do |g| $redis.sadd global_key, g end end end end Tuesday, August 10, 2010
49. global error lessons avoid joins for simple data consider race conditions use append-only file (AOF) Tuesday, August 10, 2010
50. MULTIPLAYER NOTEPAD PUB/SUB based on http://github.com/laktek/realie Tuesday, August 10, 2010
51. # usage: ruby pub.rb room username data = {"user" => ARGV[1]} loop do msg = STDIN.gets $redis.publish ARGV[0], data.merge('msg' => msg.strip).to_json end Tuesday, August 10, 2010
52. # sub.rb $redis = Redis.new(:timeout => 0) $redis.subscribe('rubyonrails', 'rubymidwest') do |on| on.message do |room, msg| data = JSON.parse(msg) puts "##{room} - [#{data['user']}]: #{data['msg']}" end end Tuesday, August 10, 2010
53. % ruby pub.rb rubymidwest qrush i give up, i hate markdown % ruby sub.rb #rubymidwest - [qrush]: i give up, i hate markdown #rubyonrails - [railsn00b]: undefined method posts_path? wtf? #rubymidwest - [turbage]: seriously. Tuesday, August 10, 2010
54. multiplayer notepad lessons combine with other data structures can subscribe to channels via patterns concurrency in ruby is hard use eventmachine! (or node.js) Tuesday, August 10, 2010
55. more to learn know your data! (via @antirez) command reference on the wiki active IRC, mailing list Tuesday, August 10, 2010
56. AKASENTAI.com redis in the cloud Tuesday, August 10, 2010
57. Thanks! http://redis.io @qrush http://rediscookbook.com http://scr.bi/redispower Tuesday, August 10, 2010
58. BONUS ROUND! I prepared way too many examples. Jackpot! Tuesday, August 10, 2010
59. URL SHORTENER STRINGS based on http://github.com/mattmatt/relink Tuesday, August 10, 2010
60. require 'sinatra' require 'redis_url' post '/' do RedisUrl.new(params[:url]).save end Tuesday, August 10, 2010
61. class RedisUrl attr_accessor :url, :id def initialize(url) @url = url @id = seed # unique string algorithm end def save $redis.set("relink.url|#{@id}", @url) $redis.set("relink.url.rev|#{@url}", @id) end end Tuesday, August 10, 2010
62. get %r{/(.+)} do |url| u = RedisUrl.find(url) if u u.clicked redirect u.url else status 404 end end Tuesday, August 10, 2010
63. class RedisUrl def self.find(id) u = $redis.get("relink.url|#{id}") if u redis_url = RedisUrl.new(u) redis_url.id = id redis_url end end def clicked $redis.incr("relink.url.clicks|#{@id}") end end Tuesday, August 10, 2010
64. url shortener lessons common pattern: namespacing incr/decr assumes value is an integer wrap behavior into ActiveRecord-like objects next step: store URLs in a list Tuesday, August 10, 2010
65. LIVE DEBUGGING LISTS based on http://github.com/quirkey/redisk Tuesday, August 10, 2010
66. def after_save begin # make request to external service rescue Exception => ex logger.error "this shouldn't ever happen!" logger.error ex logger.error ex.backtrace end end Tuesday, August 10, 2010
67. # config/initializers/logger.rb require 'redisk' path = "#{Rails.env}.log" config.logger = Redisk::Logger.new(path) Tuesday, August 10, 2010
68. class Redisk::IO def write(string) redis.rpush "#{name}:_list", string end def self.readlines(name) redis.lrange("#{name}:_list", 0, -1) end end Tuesday, August 10, 2010
69. live debugging lessons enables real-time data about your system dump serialized/marshalled data fast run the redis instance on a different box dive deeper: hummingbird Tuesday, August 10, 2010
70. COUNTING DOWNLOADS STRINGS SORTED SETS HASHES based on http://github.com/rubygems/gemcutter Tuesday, August 10, 2010
71. # bad idea, dude class Download < ActiveRecord::Base belongs_to :rubygem end class Rubygem < ActiveRecord::Base has_many :downloads end Tuesday, August 10, 2010
72. class Download def self.incr(rubygem) $redis.incr("all") $redis.incr(rubygem) $redis.zincrby("today", 1, rubygem) end end Tuesday, August 10, 2010
73. class Download def self.rollover(version) $redis.rename "today", "yesterday" dls = Hash[*$redis.zrange("yesterday", 0, -1, :with_scores => true)] dls.each do |key, score| $redis.hincrby key, Date.today, score Rubygem.find_by_name(key).increment!(:downloads, score) end end end Tuesday, August 10, 2010
74. get "/api/v1/downloads/rails.json" do $redis.hgetall("rails").to_json end # returns... { "2010-07-09" => 1908, "2010-07-10" => 1032, "2010-07-11" => 1091, } Tuesday, August 10, 2010
75. counting downloads lessons hybrid approach does work! redis is really not for search test your migration away from SQL Tuesday, August 10, 2010