Redis: Persistence Power

Slide 1

Slide 1 text

Redis: Persistence Power Nick Quaranto / @qrush / nick@quaran.to Tuesday, August 10, 2010

Slide 2

Slide 2 text

What is Redis? “advanced key-value store” REmote DIctionary Server data structures server Tuesday, August 10, 2010

Slide 3

Slide 3 text

YOUR APP Tuesday, August 10, 2010

Slide 4

Slide 4 text

YOUR APP REDIS Tuesday, August 10, 2010

Slide 5

Slide 5 text

the basics persist data as you think of it in memory, sync to disk in background ridiculously fast master-slave replication keys = strings, value = data structures Tuesday, August 10, 2010

Slide 6

Slide 6 text

http://try.redis-db.com Tuesday, August 10, 2010

Slide 7

Slide 7 text

use it: redis-cli % ./redis-cli SET user:1:name qrush OK % ./redis-cli GET user:1:name "qrush" Tuesday, August 10, 2010

Slide 8

Slide 8 text

use it: redis-rb % gem install redis % irb -rubygems -rredis >> $redis = Redis.new => #> $redis.set "user:1:name", "qrush" => "OK" >> $redis.get "user:1:name" => "qrush" Tuesday, August 10, 2010

Slide 9

Slide 9 text

FEATURE SWITCHES STRINGS based on http://github.com/blog/677 http://github.com/bvandenbos/redis_feature_control Tuesday, August 10, 2010

Slide 10

Slide 10 text

Rediswitch.features << :super_secret Rediswitch.features << :payment_gateway Rediswitch.features << :twitter if Rediswitch.enabled?(:twitter) # post to twitter else # failwhale ahoy! end Tuesday, August 10, 2010

Slide 11

Slide 11 text

begin # take some money rescue PaymentGateway::TotallyDown => ohno Rediswitch.disable(:payment_gateway) # notify the troops end Tuesday, August 10, 2010

Slide 12

Slide 12 text

class Rediswitch def self.enabled?(feature) $redis.exists(feature) end def self.enable(feature) $redis.incr(feature) end def self.disable(feature) $redis.del(feature) end end Tuesday, August 10, 2010

Slide 13

Slide 13 text

feature switch lessons the real win: no-deploy conﬁguration fast enough to be transparent next step: separate users into buckets with sets http://github.com/jamesgolick/rollout Tuesday, August 10, 2010

Slide 14

Slide 14 text

RATE LIMITER STRINGS soon to be in place at http://hoptoadapp.com Tuesday, August 10, 2010

Slide 15

Slide 15 text

class Choker def restrict? track count_for > 60 end end Tuesday, August 10, 2010

Slide 16

Slide 16 text

class Choker def count_for $memcache.get(key, true).to_i end end Tuesday, August 10, 2010

Slide 17

Slide 17 text

class Choker def track if !$memcache.get(key, true) $memcache.add(key, "0", 1.minute.from_now, true) end $memcache.incr(key) end end Tuesday, August 10, 2010

Slide 18

Slide 18 text

class Choker def track if !$redis.exists(key) $redis.setex(key, 60, 0) end $redis.incr(key) end end Tuesday, August 10, 2010

Slide 19

Slide 19 text

class Choker def count_for $redis.get(key).to_i end end Tuesday, August 10, 2010

Slide 20

Slide 20 text

rate limiter lessons expire semantics are changing in redis 2.2 benchmark the crap out of it could use a sorted set instead of strings Tuesday, August 10, 2010

Slide 21

Slide 21 text

API USAGE LOGGING STRINGS SORTED SETS based off http://www.production-hacks.com/2010/07/10/redis-api-access-logger/ Tuesday, August 10, 2010

Slide 22

Slide 22 text

# one way to do it class ActionHit < ActiveRecord::Base # t.string :controller # t.string :action # t.integer :counter end class UserHit < ActiveRecord::Base # t.string :controller_action # t.integer :user_id # t.integer :counter end Tuesday, August 10, 2010

Slide 23

Slide 23 text

# for all controllers { "statuses#update" => 1410, "users#create" => 931, "home#index" => 2936 } # users hitting an action { "101" => 42, "102" => 13, "103" => 34 } Tuesday, August 10, 2010

Slide 24

Slide 24 text

class StatusesController < ApplicationController def update $redis.incr "statuses#update" $redis.incr "statuses#update:#{user.id}" end end Tuesday, August 10, 2010

Slide 25

Slide 25 text

class StatusesController < ApplicationController def update key = "statuses#update" $redis.zincrby "actions", 1, key $redis.zincrby "users:#{key}", 1, user.id end end Tuesday, August 10, 2010

Slide 26

Slide 26 text

# hits for a specific user >> $redis.zscore "users:statuses#update", 1001 => 42 # list all the controller actions, sorted >> $redis.zrevrange "actions", 0, -1, :with_scores => true => ["home#index", "2936", "statuses#update", "1410", "users#create", "931"] Tuesday, August 10, 2010

Slide 27

Slide 27 text

api usage logging lessons sorted set = high score list bad at historical usage, trends good for a simple heartbeat or pulse Tuesday, August 10, 2010

Slide 28

Slide 28 text

JOB QUEUE LISTS based on http://github.com/defunkt/resque Tuesday, August 10, 2010

Slide 29

Slide 29 text

class Staple @queue = :default def self.perform(post_id, tempfile) # complex image resizing, cropping end end Tuesday, August 10, 2010

Slide 30

Slide 30 text

class Post < ActiveRecord::Base after_save :process_with_stapler def process_with_stapler Resque.enqueue(Staple, self.id, @tempfile) end end Tuesday, August 10, 2010

Slide 31

Slide 31 text

module Resque extend self def push(queue, item) redis.rpush "q:#{queue}", encode(item) end def pop(queue) decode redis.lpop("q:#{queue}") end end Tuesday, August 10, 2010

Slide 32

Slide 32 text

class Resque::Worker def work loop do if job = Resque.pop(queue) job.perform else sleep 5 end end end end Tuesday, August 10, 2010

Slide 33

Slide 33 text

module Resque extend self def bpop(queue) decode redis.blpop("q:#{queue}") end end Tuesday, August 10, 2010

Slide 34

Slide 34 text

class Resque::Worker def work loop do job = Resque.bpop(queue) job.perform end end end Tuesday, August 10, 2010

Slide 35

Slide 35 text

job queue lessons guaranteed atomic actions, no row locking blocking commands simplify daemons many more queue commands in redis itself! Tuesday, August 10, 2010

Slide 36

Slide 36 text

GLOBAL ERRORS SETS MULTI/EXEC a new feature at http://hoptoadapp.com Tuesday, August 10, 2010

Slide 37

Slide 37 text

# text :globals, :default => '', :null => false class Project < ActiveRecord::Base def has_global?(name) @globals ||= globals.gsub(/,/,' ').split @globals.include?(name) end end Tuesday, August 10, 2010

Slide 38

Slide 38 text

# MORE TABLES!!!! class Global < ActiveRecord::Base belongs_to :project end class Project < ActiveRecord::Base has_many :globals end Tuesday, August 10, 2010

Slide 39

Slide 39 text

# Project#global_errors ["MySQL::Error", "MemCache::Error", "Net::HTTPFatalError"] Tuesday, August 10, 2010

Slide 40

Slide 40 text

class Project < ActiveRecord::Base def global_key "project-#{id}-globals" end def has_global?(name) $redis.sismember(global_key, name) end end Tuesday, August 10, 2010

Slide 41

Slide 41 text

class Project < ActiveRecord::Base after_save :save_globals def save_globals $redis.del global_key @globals.each do |g| $redis.sadd global_key, g end end end Tuesday, August 10, 2010

Slide 42

Slide 42 text

[Mysql::Error, MemCache::Error, Net::HTTPFatalError] SISMEMBER SISMEMBER Tuesday, August 10, 2010

Slide 43

Slide 43 text

[] DEL Tuesday, August 10, 2010

Slide 44

Slide 44 text

[Mysql::Error] DEL SADD Tuesday, August 10, 2010

Slide 45

Slide 45 text

[Mysql::Error, OpenURI::HTTPError] DEL SADD SADD Tuesday, August 10, 2010

Slide 46

Slide 46 text

[Mysql::Error, OpenURI::HTTPError] DEL SADD SADD SISMEMBER [] Tuesday, August 10, 2010

Slide 47

Slide 47 text

[Mysql::Error, OpenURI::HTTPError] MULTI EXEC DEL SADD SADD SISMEMBER Tuesday, August 10, 2010

Slide 48

Slide 48 text

class Project < ActiveRecord::Base after_save :save_globals def save_globals $redis.multi do $redis.del global_key @globals.each do |g| $redis.sadd global_key, g end end end end Tuesday, August 10, 2010

Slide 49

Slide 49 text

global error lessons avoid joins for simple data consider race conditions use append-only ﬁle (AOF) Tuesday, August 10, 2010

Slide 50

Slide 50 text

MULTIPLAYER NOTEPAD PUB/SUB based on http://github.com/laktek/realie Tuesday, August 10, 2010

Slide 51

Slide 51 text

# usage: ruby pub.rb room username data = {"user" => ARGV[1]} loop do msg = STDIN.gets $redis.publish ARGV[0], data.merge('msg' => msg.strip).to_json end Tuesday, August 10, 2010

Slide 52

Slide 52 text

# sub.rb $redis = Redis.new(:timeout => 0) $redis.subscribe('rubyonrails', 'rubymidwest') do |on| on.message do |room, msg| data = JSON.parse(msg) puts "##{room} - [#{data['user']}]: #{data['msg']}" end end Tuesday, August 10, 2010

Slide 53

Slide 53 text

% ruby pub.rb rubymidwest qrush i give up, i hate markdown % ruby sub.rb #rubymidwest - [qrush]: i give up, i hate markdown #rubyonrails - [railsn00b]: undefined method posts_path? wtf? #rubymidwest - [turbage]: seriously. Tuesday, August 10, 2010

Slide 54

Slide 54 text

multiplayer notepad lessons combine with other data structures can subscribe to channels via patterns concurrency in ruby is hard use eventmachine! (or node.js) Tuesday, August 10, 2010

Slide 55

Slide 55 text

more to learn know your data! (via @antirez) command reference on the wiki active IRC, mailing list Tuesday, August 10, 2010

Slide 56

Slide 56 text

AKASENTAI.com redis in the cloud Tuesday, August 10, 2010

Slide 57

Slide 57 text

Thanks! http://redis.io @qrush http://rediscookbook.com http://scr.bi/redispower Tuesday, August 10, 2010

Slide 58

Slide 58 text

BONUS ROUND! I prepared way too many examples. Jackpot! Tuesday, August 10, 2010

Slide 59

Slide 59 text

URL SHORTENER STRINGS based on http://github.com/mattmatt/relink Tuesday, August 10, 2010

Slide 60

Slide 60 text

require 'sinatra' require 'redis_url' post '/' do RedisUrl.new(params[:url]).save end Tuesday, August 10, 2010

Slide 61

Slide 61 text

class RedisUrl attr_accessor :url, :id def initialize(url) @url = url @id = seed # unique string algorithm end def save $redis.set("relink.url|#{@id}", @url) $redis.set("relink.url.rev|#{@url}", @id) end end Tuesday, August 10, 2010

Slide 62

Slide 62 text

get %r{/(.+)} do |url| u = RedisUrl.find(url) if u u.clicked redirect u.url else status 404 end end Tuesday, August 10, 2010

Slide 63

Slide 63 text

class RedisUrl def self.find(id) u = $redis.get("relink.url|#{id}") if u redis_url = RedisUrl.new(u) redis_url.id = id redis_url end end def clicked $redis.incr("relink.url.clicks|#{@id}") end end Tuesday, August 10, 2010

Slide 64

Slide 64 text

url shortener lessons common pattern: namespacing incr/decr assumes value is an integer wrap behavior into ActiveRecord-like objects next step: store URLs in a list Tuesday, August 10, 2010

Slide 65

Slide 65 text

LIVE DEBUGGING LISTS based on http://github.com/quirkey/redisk Tuesday, August 10, 2010

Slide 66

Slide 66 text

def after_save begin # make request to external service rescue Exception => ex logger.error "this shouldn't ever happen!" logger.error ex logger.error ex.backtrace end end Tuesday, August 10, 2010

Slide 67

Slide 67 text

# config/initializers/logger.rb require 'redisk' path = "#{Rails.env}.log" config.logger = Redisk::Logger.new(path) Tuesday, August 10, 2010

Slide 68

Slide 68 text

class Redisk::IO def write(string) redis.rpush "#{name}:_list", string end def self.readlines(name) redis.lrange("#{name}:_list", 0, -1) end end Tuesday, August 10, 2010

Slide 69

Slide 69 text

live debugging lessons enables real-time data about your system dump serialized/marshalled data fast run the redis instance on a different box dive deeper: hummingbird Tuesday, August 10, 2010

Slide 70

Slide 70 text

COUNTING DOWNLOADS STRINGS SORTED SETS HASHES based on http://github.com/rubygems/gemcutter Tuesday, August 10, 2010

Slide 71

Slide 71 text

# bad idea, dude class Download < ActiveRecord::Base belongs_to :rubygem end class Rubygem < ActiveRecord::Base has_many :downloads end Tuesday, August 10, 2010

Slide 72

Slide 72 text

class Download def self.incr(rubygem) $redis.incr("all") $redis.incr(rubygem) $redis.zincrby("today", 1, rubygem) end end Tuesday, August 10, 2010

Slide 73

Slide 73 text

class Download def self.rollover(version) $redis.rename "today", "yesterday" dls = Hash[*$redis.zrange("yesterday", 0, -1, :with_scores => true)] dls.each do |key, score| $redis.hincrby key, Date.today, score Rubygem.find_by_name(key).increment!(:downloads, score) end end end Tuesday, August 10, 2010

Slide 74

Slide 74 text

get "/api/v1/downloads/rails.json" do $redis.hgetall("rails").to_json end # returns... { "2010-07-09" => 1908, "2010-07-10" => 1032, "2010-07-11" => 1091, } Tuesday, August 10, 2010

Slide 75

Slide 75 text

counting downloads lessons hybrid approach does work! redis is really not for search test your migration away from SQL Tuesday, August 10, 2010