Redis: Persistence Power

Redis: Persistence Power

A practical, use case approach to looking into Redis.

Given at RubyMidwest 2010. http://rubymidwest.com/

Eb8975af8e49e19e3dd6b6b84a542e26?s=128

Nick Quaranto

January 12, 2012
Tweet

Transcript

  1. Redis: Persistence Power Nick Quaranto / @qrush / nick@quaran.to Tuesday,

    August 10, 2010
  2. What is Redis? “advanced key-value store” REmote DIctionary Server data

    structures server Tuesday, August 10, 2010
  3. YOUR APP Tuesday, August 10, 2010

  4. YOUR APP REDIS Tuesday, August 10, 2010

  5. the basics persist data as you think of it in

    memory, sync to disk in background ridiculously fast master-slave replication keys = strings, value = data structures Tuesday, August 10, 2010
  6. http://try.redis-db.com Tuesday, August 10, 2010

  7. use it: redis-cli % ./redis-cli SET user:1:name qrush OK %

    ./redis-cli GET user:1:name "qrush" Tuesday, August 10, 2010
  8. use it: redis-rb % gem install redis % irb -rubygems

    -rredis >> $redis = Redis.new => #<Redis client v2.0.3 connected... >> $redis.set "user:1:name", "qrush" => "OK" >> $redis.get "user:1:name" => "qrush" Tuesday, August 10, 2010
  9. FEATURE SWITCHES STRINGS based on http://github.com/blog/677 http://github.com/bvandenbos/redis_feature_control Tuesday, August 10,

    2010
  10. Rediswitch.features << :super_secret Rediswitch.features << :payment_gateway Rediswitch.features << :twitter if

    Rediswitch.enabled?(:twitter) # post to twitter else # failwhale ahoy! end Tuesday, August 10, 2010
  11. begin # take some money rescue PaymentGateway::TotallyDown => ohno Rediswitch.disable(:payment_gateway)

    # notify the troops end Tuesday, August 10, 2010
  12. class Rediswitch def self.enabled?(feature) $redis.exists(feature) end def self.enable(feature) $redis.incr(feature) end

    def self.disable(feature) $redis.del(feature) end end Tuesday, August 10, 2010
  13. feature switch lessons the real win: no-deploy configuration fast enough

    to be transparent next step: separate users into buckets with sets http://github.com/jamesgolick/rollout Tuesday, August 10, 2010
  14. RATE LIMITER STRINGS soon to be in place at http://hoptoadapp.com

    Tuesday, August 10, 2010
  15. class Choker def restrict? track count_for > 60 end end

    Tuesday, August 10, 2010
  16. class Choker def count_for $memcache.get(key, true).to_i end end Tuesday, August

    10, 2010
  17. class Choker def track if !$memcache.get(key, true) $memcache.add(key, "0", 1.minute.from_now,

    true) end $memcache.incr(key) end end Tuesday, August 10, 2010
  18. class Choker def track if !$redis.exists(key) $redis.setex(key, 60, 0) end

    $redis.incr(key) end end Tuesday, August 10, 2010
  19. class Choker def count_for $redis.get(key).to_i end end Tuesday, August 10,

    2010
  20. rate limiter lessons expire semantics are changing in redis 2.2

    benchmark the crap out of it could use a sorted set instead of strings Tuesday, August 10, 2010
  21. API USAGE LOGGING STRINGS SORTED SETS based off http://www.production-hacks.com/2010/07/10/redis-api-access-logger/ Tuesday,

    August 10, 2010
  22. # one way to do it class ActionHit < ActiveRecord::Base

    # t.string :controller # t.string :action # t.integer :counter end class UserHit < ActiveRecord::Base # t.string :controller_action # t.integer :user_id # t.integer :counter end Tuesday, August 10, 2010
  23. # for all controllers { "statuses#update" => 1410, "users#create" =>

    931, "home#index" => 2936 } # users hitting an action { "101" => 42, "102" => 13, "103" => 34 } Tuesday, August 10, 2010
  24. class StatusesController < ApplicationController def update $redis.incr "statuses#update" $redis.incr "statuses#update:#{user.id}"

    end end Tuesday, August 10, 2010
  25. class StatusesController < ApplicationController def update key = "statuses#update" $redis.zincrby

    "actions", 1, key $redis.zincrby "users:#{key}", 1, user.id end end Tuesday, August 10, 2010
  26. # hits for a specific user >> $redis.zscore "users:statuses#update", 1001

    => 42 # list all the controller actions, sorted >> $redis.zrevrange "actions", 0, -1, :with_scores => true => ["home#index", "2936", "statuses#update", "1410", "users#create", "931"] Tuesday, August 10, 2010
  27. api usage logging lessons sorted set = high score list

    bad at historical usage, trends good for a simple heartbeat or pulse Tuesday, August 10, 2010
  28. JOB QUEUE LISTS based on http://github.com/defunkt/resque Tuesday, August 10, 2010

  29. class Staple @queue = :default def self.perform(post_id, tempfile) # complex

    image resizing, cropping end end Tuesday, August 10, 2010
  30. class Post < ActiveRecord::Base after_save :process_with_stapler def process_with_stapler Resque.enqueue(Staple, self.id,

    @tempfile) end end Tuesday, August 10, 2010
  31. module Resque extend self def push(queue, item) redis.rpush "q:#{queue}", encode(item)

    end def pop(queue) decode redis.lpop("q:#{queue}") end end Tuesday, August 10, 2010
  32. class Resque::Worker def work loop do if job = Resque.pop(queue)

    job.perform else sleep 5 end end end end Tuesday, August 10, 2010
  33. module Resque extend self def bpop(queue) decode redis.blpop("q:#{queue}") end end

    Tuesday, August 10, 2010
  34. class Resque::Worker def work loop do job = Resque.bpop(queue) job.perform

    end end end Tuesday, August 10, 2010
  35. job queue lessons guaranteed atomic actions, no row locking blocking

    commands simplify daemons many more queue commands in redis itself! Tuesday, August 10, 2010
  36. GLOBAL ERRORS SETS MULTI/EXEC a new feature at http://hoptoadapp.com Tuesday,

    August 10, 2010
  37. # text :globals, :default => '', :null => false class

    Project < ActiveRecord::Base def has_global?(name) @globals ||= globals.gsub(/,/,' ').split @globals.include?(name) end end Tuesday, August 10, 2010
  38. # MORE TABLES!!!! class Global < ActiveRecord::Base belongs_to :project end

    class Project < ActiveRecord::Base has_many :globals end Tuesday, August 10, 2010
  39. # Project#global_errors ["MySQL::Error", "MemCache::Error", "Net::HTTPFatalError"] Tuesday, August 10, 2010

  40. class Project < ActiveRecord::Base def global_key "project-#{id}-globals" end def has_global?(name)

    $redis.sismember(global_key, name) end end Tuesday, August 10, 2010
  41. class Project < ActiveRecord::Base after_save :save_globals def save_globals $redis.del global_key

    @globals.each do |g| $redis.sadd global_key, g end end end Tuesday, August 10, 2010
  42. [Mysql::Error, MemCache::Error, Net::HTTPFatalError] SISMEMBER SISMEMBER Tuesday, August 10, 2010

  43. [] DEL Tuesday, August 10, 2010

  44. [Mysql::Error] DEL SADD Tuesday, August 10, 2010

  45. [Mysql::Error, OpenURI::HTTPError] DEL SADD SADD Tuesday, August 10, 2010

  46. [Mysql::Error, OpenURI::HTTPError] DEL SADD SADD SISMEMBER [] Tuesday, August 10,

    2010
  47. [Mysql::Error, OpenURI::HTTPError] MULTI EXEC DEL SADD SADD SISMEMBER Tuesday, August

    10, 2010
  48. class Project < ActiveRecord::Base after_save :save_globals def save_globals $redis.multi do

    $redis.del global_key @globals.each do |g| $redis.sadd global_key, g end end end end Tuesday, August 10, 2010
  49. global error lessons avoid joins for simple data consider race

    conditions use append-only file (AOF) Tuesday, August 10, 2010
  50. MULTIPLAYER NOTEPAD PUB/SUB based on http://github.com/laktek/realie Tuesday, August 10, 2010

  51. # usage: ruby pub.rb room username data = {"user" =>

    ARGV[1]} loop do msg = STDIN.gets $redis.publish ARGV[0], data.merge('msg' => msg.strip).to_json end Tuesday, August 10, 2010
  52. # sub.rb $redis = Redis.new(:timeout => 0) $redis.subscribe('rubyonrails', 'rubymidwest') do

    |on| on.message do |room, msg| data = JSON.parse(msg) puts "##{room} - [#{data['user']}]: #{data['msg']}" end end Tuesday, August 10, 2010
  53. % ruby pub.rb rubymidwest qrush i give up, i hate

    markdown % ruby sub.rb #rubymidwest - [qrush]: i give up, i hate markdown #rubyonrails - [railsn00b]: undefined method posts_path? wtf? #rubymidwest - [turbage]: seriously. Tuesday, August 10, 2010
  54. multiplayer notepad lessons combine with other data structures can subscribe

    to channels via patterns concurrency in ruby is hard use eventmachine! (or node.js) Tuesday, August 10, 2010
  55. more to learn know your data! (via @antirez) command reference

    on the wiki active IRC, mailing list Tuesday, August 10, 2010
  56. AKASENTAI.com redis in the cloud Tuesday, August 10, 2010

  57. Thanks! http://redis.io @qrush http://rediscookbook.com http://scr.bi/redispower Tuesday, August 10, 2010

  58. BONUS ROUND! I prepared way too many examples. Jackpot! Tuesday,

    August 10, 2010
  59. URL SHORTENER STRINGS based on http://github.com/mattmatt/relink Tuesday, August 10, 2010

  60. require 'sinatra' require 'redis_url' post '/' do RedisUrl.new(params[:url]).save end Tuesday,

    August 10, 2010
  61. class RedisUrl attr_accessor :url, :id def initialize(url) @url = url

    @id = seed # unique string algorithm end def save $redis.set("relink.url|#{@id}", @url) $redis.set("relink.url.rev|#{@url}", @id) end end Tuesday, August 10, 2010
  62. get %r{/(.+)} do |url| u = RedisUrl.find(url) if u u.clicked

    redirect u.url else status 404 end end Tuesday, August 10, 2010
  63. class RedisUrl def self.find(id) u = $redis.get("relink.url|#{id}") if u redis_url

    = RedisUrl.new(u) redis_url.id = id redis_url end end def clicked $redis.incr("relink.url.clicks|#{@id}") end end Tuesday, August 10, 2010
  64. url shortener lessons common pattern: namespacing incr/decr assumes value is

    an integer wrap behavior into ActiveRecord-like objects next step: store URLs in a list Tuesday, August 10, 2010
  65. LIVE DEBUGGING LISTS based on http://github.com/quirkey/redisk Tuesday, August 10, 2010

  66. def after_save begin # make request to external service rescue

    Exception => ex logger.error "this shouldn't ever happen!" logger.error ex logger.error ex.backtrace end end Tuesday, August 10, 2010
  67. # config/initializers/logger.rb require 'redisk' path = "#{Rails.env}.log" config.logger = Redisk::Logger.new(path)

    Tuesday, August 10, 2010
  68. class Redisk::IO def write(string) redis.rpush "#{name}:_list", string end def self.readlines(name)

    redis.lrange("#{name}:_list", 0, -1) end end Tuesday, August 10, 2010
  69. live debugging lessons enables real-time data about your system dump

    serialized/marshalled data fast run the redis instance on a different box dive deeper: hummingbird Tuesday, August 10, 2010
  70. COUNTING DOWNLOADS STRINGS SORTED SETS HASHES based on http://github.com/rubygems/gemcutter Tuesday,

    August 10, 2010
  71. # bad idea, dude class Download < ActiveRecord::Base belongs_to :rubygem

    end class Rubygem < ActiveRecord::Base has_many :downloads end Tuesday, August 10, 2010
  72. class Download def self.incr(rubygem) $redis.incr("all") $redis.incr(rubygem) $redis.zincrby("today", 1, rubygem) end

    end Tuesday, August 10, 2010
  73. class Download def self.rollover(version) $redis.rename "today", "yesterday" dls = Hash[*$redis.zrange("yesterday",

    0, -1, :with_scores => true)] dls.each do |key, score| $redis.hincrby key, Date.today, score Rubygem.find_by_name(key).increment!(:downloads, score) end end end Tuesday, August 10, 2010
  74. get "/api/v1/downloads/rails.json" do $redis.hgetall("rails").to_json end # returns... { "2010-07-09" =>

    1908, "2010-07-10" => 1032, "2010-07-11" => 1091, } Tuesday, August 10, 2010
  75. counting downloads lessons hybrid approach does work! redis is really

    not for search test your migration away from SQL Tuesday, August 10, 2010