$30 off During Our Annual Pro Sale. View Details »

Redis: Persistence Power

Redis: Persistence Power

A practical, use case approach to looking into Redis.

Given at RubyMidwest 2010. http://rubymidwest.com/

Nick Quaranto

January 12, 2012
Tweet

More Decks by Nick Quaranto

Other Decks in Programming

Transcript

  1. Redis: Persistence Power
    Nick Quaranto / @qrush / [email protected]
    Tuesday, August 10, 2010

    View Slide

  2. What is Redis?
    “advanced key-value store”
    REmote DIctionary Server
    data structures server
    Tuesday, August 10, 2010

    View Slide

  3. YOUR APP
    Tuesday, August 10, 2010

    View Slide

  4. YOUR APP
    REDIS
    Tuesday, August 10, 2010

    View Slide

  5. the basics
    persist data as you think of it
    in memory, sync to disk in background
    ridiculously fast
    master-slave replication
    keys = strings, value = data structures
    Tuesday, August 10, 2010

    View Slide

  6. http://try.redis-db.com
    Tuesday, August 10, 2010

    View Slide

  7. use it: redis-cli
    % ./redis-cli SET user:1:name qrush
    OK
    % ./redis-cli GET user:1:name
    "qrush"
    Tuesday, August 10, 2010

    View Slide

  8. use it: redis-rb
    % gem install redis
    % irb -rubygems -rredis
    >> $redis = Redis.new
    => #>> $redis.set "user:1:name", "qrush"
    => "OK"
    >> $redis.get "user:1:name"
    => "qrush"
    Tuesday, August 10, 2010

    View Slide

  9. FEATURE
    SWITCHES
    STRINGS
    based on
    http://github.com/blog/677
    http://github.com/bvandenbos/redis_feature_control
    Tuesday, August 10, 2010

    View Slide

  10. Rediswitch.features << :super_secret
    Rediswitch.features << :payment_gateway
    Rediswitch.features << :twitter
    if Rediswitch.enabled?(:twitter)
    # post to twitter
    else
    # failwhale ahoy!
    end
    Tuesday, August 10, 2010

    View Slide

  11. begin
    # take some money
    rescue PaymentGateway::TotallyDown => ohno
    Rediswitch.disable(:payment_gateway)
    # notify the troops
    end
    Tuesday, August 10, 2010

    View Slide

  12. class Rediswitch
    def self.enabled?(feature)
    $redis.exists(feature)
    end
    def self.enable(feature)
    $redis.incr(feature)
    end
    def self.disable(feature)
    $redis.del(feature)
    end
    end
    Tuesday, August 10, 2010

    View Slide

  13. feature switch lessons
    the real win: no-deploy configuration
    fast enough to be transparent
    next step: separate users into buckets with sets
    http://github.com/jamesgolick/rollout
    Tuesday, August 10, 2010

    View Slide

  14. RATE
    LIMITER
    STRINGS
    soon to be in place at
    http://hoptoadapp.com
    Tuesday, August 10, 2010

    View Slide

  15. class Choker
    def restrict?
    track
    count_for > 60
    end
    end
    Tuesday, August 10, 2010

    View Slide

  16. class Choker
    def count_for
    $memcache.get(key, true).to_i
    end
    end
    Tuesday, August 10, 2010

    View Slide

  17. class Choker
    def track
    if !$memcache.get(key, true)
    $memcache.add(key, "0", 1.minute.from_now, true)
    end
    $memcache.incr(key)
    end
    end
    Tuesday, August 10, 2010

    View Slide

  18. class Choker
    def track
    if !$redis.exists(key)
    $redis.setex(key, 60, 0)
    end
    $redis.incr(key)
    end
    end
    Tuesday, August 10, 2010

    View Slide

  19. class Choker
    def count_for
    $redis.get(key).to_i
    end
    end
    Tuesday, August 10, 2010

    View Slide

  20. rate limiter lessons
    expire semantics are changing in redis 2.2
    benchmark the crap out of it
    could use a sorted set instead of strings
    Tuesday, August 10, 2010

    View Slide

  21. API USAGE
    LOGGING
    STRINGS
    SORTED SETS
    based off
    http://www.production-hacks.com/2010/07/10/redis-api-access-logger/
    Tuesday, August 10, 2010

    View Slide

  22. # one way to do it
    class ActionHit < ActiveRecord::Base
    # t.string :controller
    # t.string :action
    # t.integer :counter
    end
    class UserHit < ActiveRecord::Base
    # t.string :controller_action
    # t.integer :user_id
    # t.integer :counter
    end
    Tuesday, August 10, 2010

    View Slide

  23. # for all controllers
    {
    "statuses#update" => 1410,
    "users#create" => 931,
    "home#index" => 2936
    }
    # users hitting an action
    {
    "101" => 42,
    "102" => 13,
    "103" => 34
    }
    Tuesday, August 10, 2010

    View Slide

  24. class StatusesController < ApplicationController
    def update
    $redis.incr "statuses#update"
    $redis.incr "statuses#update:#{user.id}"
    end
    end
    Tuesday, August 10, 2010

    View Slide

  25. class StatusesController < ApplicationController
    def update
    key = "statuses#update"
    $redis.zincrby "actions", 1, key
    $redis.zincrby "users:#{key}", 1, user.id
    end
    end
    Tuesday, August 10, 2010

    View Slide

  26. # hits for a specific user
    >> $redis.zscore "users:statuses#update", 1001
    => 42
    # list all the controller actions, sorted
    >> $redis.zrevrange "actions", 0, -1,
    :with_scores => true
    => ["home#index", "2936",
    "statuses#update", "1410",
    "users#create", "931"]
    Tuesday, August 10, 2010

    View Slide

  27. api usage logging
    lessons
    sorted set = high score list
    bad at historical usage, trends
    good for a simple heartbeat or pulse
    Tuesday, August 10, 2010

    View Slide

  28. JOB QUEUE
    LISTS
    based on
    http://github.com/defunkt/resque
    Tuesday, August 10, 2010

    View Slide

  29. class Staple
    @queue = :default
    def self.perform(post_id, tempfile)
    # complex image resizing, cropping
    end
    end
    Tuesday, August 10, 2010

    View Slide

  30. class Post < ActiveRecord::Base
    after_save :process_with_stapler
    def process_with_stapler
    Resque.enqueue(Staple, self.id, @tempfile)
    end
    end
    Tuesday, August 10, 2010

    View Slide

  31. module Resque
    extend self
    def push(queue, item)
    redis.rpush "q:#{queue}", encode(item)
    end
    def pop(queue)
    decode redis.lpop("q:#{queue}")
    end
    end
    Tuesday, August 10, 2010

    View Slide

  32. class Resque::Worker
    def work
    loop do
    if job = Resque.pop(queue)
    job.perform
    else
    sleep 5
    end
    end
    end
    end
    Tuesday, August 10, 2010

    View Slide

  33. module Resque
    extend self
    def bpop(queue)
    decode redis.blpop("q:#{queue}")
    end
    end
    Tuesday, August 10, 2010

    View Slide

  34. class Resque::Worker
    def work
    loop do
    job = Resque.bpop(queue)
    job.perform
    end
    end
    end
    Tuesday, August 10, 2010

    View Slide

  35. job queue lessons
    guaranteed atomic actions, no row locking
    blocking commands simplify daemons
    many more queue commands in redis itself!
    Tuesday, August 10, 2010

    View Slide

  36. GLOBAL
    ERRORS
    SETS
    MULTI/EXEC
    a new feature at
    http://hoptoadapp.com
    Tuesday, August 10, 2010

    View Slide

  37. # text :globals, :default => '', :null => false
    class Project < ActiveRecord::Base
    def has_global?(name)
    @globals ||= globals.gsub(/,/,' ').split
    @globals.include?(name)
    end
    end
    Tuesday, August 10, 2010

    View Slide

  38. # MORE TABLES!!!!
    class Global < ActiveRecord::Base
    belongs_to :project
    end
    class Project < ActiveRecord::Base
    has_many :globals
    end
    Tuesday, August 10, 2010

    View Slide

  39. # Project#global_errors
    ["MySQL::Error",
    "MemCache::Error",
    "Net::HTTPFatalError"]
    Tuesday, August 10, 2010

    View Slide

  40. class Project < ActiveRecord::Base
    def global_key
    "project-#{id}-globals"
    end
    def has_global?(name)
    $redis.sismember(global_key, name)
    end
    end
    Tuesday, August 10, 2010

    View Slide

  41. class Project < ActiveRecord::Base
    after_save :save_globals
    def save_globals
    $redis.del global_key
    @globals.each do |g|
    $redis.sadd global_key, g
    end
    end
    end
    Tuesday, August 10, 2010

    View Slide

  42. [Mysql::Error, MemCache::Error, Net::HTTPFatalError]
    SISMEMBER SISMEMBER
    Tuesday, August 10, 2010

    View Slide

  43. []
    DEL
    Tuesday, August 10, 2010

    View Slide

  44. [Mysql::Error]
    DEL
    SADD
    Tuesday, August 10, 2010

    View Slide

  45. [Mysql::Error, OpenURI::HTTPError]
    DEL
    SADD
    SADD
    Tuesday, August 10, 2010

    View Slide

  46. [Mysql::Error, OpenURI::HTTPError]
    DEL
    SADD
    SADD
    SISMEMBER
    []
    Tuesday, August 10, 2010

    View Slide

  47. [Mysql::Error, OpenURI::HTTPError]
    MULTI
    EXEC
    DEL
    SADD
    SADD
    SISMEMBER
    Tuesday, August 10, 2010

    View Slide

  48. class Project < ActiveRecord::Base
    after_save :save_globals
    def save_globals
    $redis.multi do
    $redis.del global_key
    @globals.each do |g|
    $redis.sadd global_key, g
    end
    end
    end
    end
    Tuesday, August 10, 2010

    View Slide

  49. global error lessons
    avoid joins for simple data
    consider race conditions
    use append-only file (AOF)
    Tuesday, August 10, 2010

    View Slide

  50. MULTIPLAYER
    NOTEPAD
    PUB/SUB
    based on
    http://github.com/laktek/realie
    Tuesday, August 10, 2010

    View Slide

  51. # usage: ruby pub.rb room username
    data = {"user" => ARGV[1]}
    loop do
    msg = STDIN.gets
    $redis.publish ARGV[0],
    data.merge('msg' => msg.strip).to_json
    end
    Tuesday, August 10, 2010

    View Slide

  52. # sub.rb
    $redis = Redis.new(:timeout => 0)
    $redis.subscribe('rubyonrails', 'rubymidwest') do |on|
    on.message do |room, msg|
    data = JSON.parse(msg)
    puts "##{room} - [#{data['user']}]: #{data['msg']}"
    end
    end
    Tuesday, August 10, 2010

    View Slide

  53. % ruby pub.rb rubymidwest qrush
    i give up, i hate markdown
    % ruby sub.rb
    #rubymidwest - [qrush]: i give up, i hate markdown
    #rubyonrails - [railsn00b]: undefined method posts_path? wtf?
    #rubymidwest - [turbage]: seriously.
    Tuesday, August 10, 2010

    View Slide

  54. multiplayer notepad
    lessons
    combine with other data structures
    can subscribe to channels via patterns
    concurrency in ruby is hard
    use eventmachine! (or node.js)
    Tuesday, August 10, 2010

    View Slide

  55. more to learn
    know your data! (via @antirez)
    command reference on the wiki
    active IRC, mailing list
    Tuesday, August 10, 2010

    View Slide

  56. AKASENTAI.com
    redis in the cloud
    Tuesday, August 10, 2010

    View Slide

  57. Thanks!
    http://redis.io @qrush
    http://rediscookbook.com
    http://scr.bi/redispower
    Tuesday, August 10, 2010

    View Slide

  58. BONUS ROUND!
    I prepared way too many examples. Jackpot!
    Tuesday, August 10, 2010

    View Slide

  59. URL
    SHORTENER
    STRINGS
    based on
    http://github.com/mattmatt/relink
    Tuesday, August 10, 2010

    View Slide

  60. require 'sinatra'
    require 'redis_url'
    post '/' do
    RedisUrl.new(params[:url]).save
    end
    Tuesday, August 10, 2010

    View Slide

  61. class RedisUrl
    attr_accessor :url, :id
    def initialize(url)
    @url = url
    @id = seed # unique string algorithm
    end
    def save
    $redis.set("relink.url|#{@id}", @url)
    $redis.set("relink.url.rev|#{@url}", @id)
    end
    end
    Tuesday, August 10, 2010

    View Slide

  62. get %r{/(.+)} do |url|
    u = RedisUrl.find(url)
    if u
    u.clicked
    redirect u.url
    else
    status 404
    end
    end
    Tuesday, August 10, 2010

    View Slide

  63. class RedisUrl
    def self.find(id)
    u = $redis.get("relink.url|#{id}")
    if u
    redis_url = RedisUrl.new(u)
    redis_url.id = id
    redis_url
    end
    end
    def clicked
    $redis.incr("relink.url.clicks|#{@id}")
    end
    end
    Tuesday, August 10, 2010

    View Slide

  64. url shortener lessons
    common pattern: namespacing
    incr/decr assumes value is an integer
    wrap behavior into ActiveRecord-like objects
    next step: store URLs in a list
    Tuesday, August 10, 2010

    View Slide

  65. LIVE
    DEBUGGING
    LISTS
    based on
    http://github.com/quirkey/redisk
    Tuesday, August 10, 2010

    View Slide

  66. def after_save
    begin
    # make request to external service
    rescue Exception => ex
    logger.error "this shouldn't ever happen!"
    logger.error ex
    logger.error ex.backtrace
    end
    end
    Tuesday, August 10, 2010

    View Slide

  67. # config/initializers/logger.rb
    require 'redisk'
    path = "#{Rails.env}.log"
    config.logger = Redisk::Logger.new(path)
    Tuesday, August 10, 2010

    View Slide

  68. class Redisk::IO
    def write(string)
    redis.rpush "#{name}:_list", string
    end
    def self.readlines(name)
    redis.lrange("#{name}:_list", 0, -1)
    end
    end
    Tuesday, August 10, 2010

    View Slide

  69. live debugging lessons
    enables real-time data about your system
    dump serialized/marshalled data fast
    run the redis instance on a different box
    dive deeper: hummingbird
    Tuesday, August 10, 2010

    View Slide

  70. COUNTING
    DOWNLOADS
    STRINGS
    SORTED SETS
    HASHES
    based on
    http://github.com/rubygems/gemcutter
    Tuesday, August 10, 2010

    View Slide

  71. # bad idea, dude
    class Download < ActiveRecord::Base
    belongs_to :rubygem
    end
    class Rubygem < ActiveRecord::Base
    has_many :downloads
    end
    Tuesday, August 10, 2010

    View Slide

  72. class Download
    def self.incr(rubygem)
    $redis.incr("all")
    $redis.incr(rubygem)
    $redis.zincrby("today", 1, rubygem)
    end
    end
    Tuesday, August 10, 2010

    View Slide

  73. class Download
    def self.rollover(version)
    $redis.rename "today", "yesterday"
    dls = Hash[*$redis.zrange("yesterday",
    0, -1,
    :with_scores => true)]
    dls.each do |key, score|
    $redis.hincrby key, Date.today, score
    Rubygem.find_by_name(key).increment!(:downloads, score)
    end
    end
    end
    Tuesday, August 10, 2010

    View Slide

  74. get "/api/v1/downloads/rails.json" do
    $redis.hgetall("rails").to_json
    end
    # returns...
    {
    "2010-07-09" => 1908,
    "2010-07-10" => 1032,
    "2010-07-11" => 1091,
    }
    Tuesday, August 10, 2010

    View Slide

  75. counting downloads
    lessons
    hybrid approach does work!
    redis is really not for search
    test your migration away from SQL
    Tuesday, August 10, 2010

    View Slide