Redis: Persistence Power

Redis: Persistence Power
Nick Quaranto / @qrush / [email protected]
Tuesday, August 10, 2010

View Slide

What is Redis?
“advanced key-value store”
REmote DIctionary Server
data structures server

View Slide

YOUR APP

View Slide

YOUR APP
REDIS

View Slide

the basics
persist data as you think of it
in memory, sync to disk in background
ridiculously fast
master-slave replication
keys = strings, value = data structures

View Slide

http://try.redis-db.com

View Slide

use it: redis-cli
% ./redis-cli SET user:1:name qrush
OK
% ./redis-cli GET user:1:name
"qrush"

View Slide

use it: redis-rb
% gem install redis
% irb -rubygems -rredis
>> $redis = Redis.new
=> #>> $redis.set "user:1:name", "qrush"
=> "OK"
>> $redis.get "user:1:name"
=> "qrush"

View Slide

FEATURE
SWITCHES
STRINGS
based on
http://github.com/blog/677
http://github.com/bvandenbos/redis_feature_control

View Slide

Rediswitch.features << :super_secret
Rediswitch.features << :payment_gateway
Rediswitch.features << :twitter
if Rediswitch.enabled?(:twitter)
# post to twitter
else
# failwhale ahoy!
end

View Slide

begin
# take some money
rescue PaymentGateway::TotallyDown => ohno
Rediswitch.disable(:payment_gateway)
# notify the troops
end

View Slide

class Rediswitch
def self.enabled?(feature)
$redis.exists(feature)
end
def self.enable(feature)
$redis.incr(feature)
end
def self.disable(feature)
$redis.del(feature)
end
end

View Slide

feature switch lessons
the real win: no-deploy conﬁguration
fast enough to be transparent
next step: separate users into buckets with sets
http://github.com/jamesgolick/rollout

View Slide

RATE
LIMITER
STRINGS
soon to be in place at
http://hoptoadapp.com

View Slide

class Choker
def restrict?
track
count_for > 60
end
end

View Slide

class Choker
def count_for
$memcache.get(key, true).to_i
end
end

View Slide

class Choker
def track
if !$memcache.get(key, true)
$memcache.add(key, "0", 1.minute.from_now, true)
end
$memcache.incr(key)
end
end

View Slide

class Choker
def track
if !$redis.exists(key)
$redis.setex(key, 60, 0)
end
$redis.incr(key)
end
end

View Slide

class Choker
def count_for
$redis.get(key).to_i
end
end

View Slide

rate limiter lessons
expire semantics are changing in redis 2.2
benchmark the crap out of it
could use a sorted set instead of strings

View Slide

API USAGE
LOGGING
STRINGS
SORTED SETS
based off
http://www.production-hacks.com/2010/07/10/redis-api-access-logger/

View Slide

# one way to do it
class ActionHit < ActiveRecord::Base
# t.string :controller
# t.string :action
# t.integer :counter
end
class UserHit < ActiveRecord::Base
# t.string :controller_action
# t.integer :user_id
# t.integer :counter
end

View Slide

# for all controllers
{
"statuses#update" => 1410,
"users#create" => 931,
"home#index" => 2936
}
# users hitting an action
{
"101" => 42,
"102" => 13,
"103" => 34
}

View Slide

class StatusesController < ApplicationController
def update
$redis.incr "statuses#update"
$redis.incr "statuses#update:#{user.id}"
end
end

View Slide

class StatusesController < ApplicationController
def update
key = "statuses#update"
$redis.zincrby "actions", 1, key
$redis.zincrby "users:#{key}", 1, user.id
end
end

View Slide

# hits for a specific user
>> $redis.zscore "users:statuses#update", 1001
=> 42
# list all the controller actions, sorted
>> $redis.zrevrange "actions", 0, -1,
:with_scores => true
=> ["home#index", "2936",
"statuses#update", "1410",
"users#create", "931"]

View Slide

api usage logging
lessons
sorted set = high score list
bad at historical usage, trends
good for a simple heartbeat or pulse

View Slide

JOB QUEUE
LISTS
based on
http://github.com/defunkt/resque

View Slide

class Staple
@queue = :default
def self.perform(post_id, tempfile)
# complex image resizing, cropping
end
end

View Slide

class Post < ActiveRecord::Base
after_save :process_with_stapler
def process_with_stapler
Resque.enqueue(Staple, self.id, @tempfile)
end
end

View Slide

module Resque
extend self
def push(queue, item)
redis.rpush "q:#{queue}", encode(item)
end
def pop(queue)
decode redis.lpop("q:#{queue}")
end
end

View Slide

class Resque::Worker
def work
loop do
if job = Resque.pop(queue)
job.perform
else
sleep 5
end
end
end
end

View Slide

module Resque
extend self
def bpop(queue)
decode redis.blpop("q:#{queue}")
end
end

View Slide

class Resque::Worker
def work
loop do
job = Resque.bpop(queue)
job.perform
end
end
end

View Slide

job queue lessons
guaranteed atomic actions, no row locking
blocking commands simplify daemons
many more queue commands in redis itself!

View Slide

GLOBAL
ERRORS
SETS
MULTI/EXEC
a new feature at
http://hoptoadapp.com

View Slide

# text :globals, :default => '', :null => false
class Project < ActiveRecord::Base
def has_global?(name)
@globals ||= globals.gsub(/,/,' ').split
@globals.include?(name)
end
end

View Slide

# MORE TABLES!!!!
class Global < ActiveRecord::Base
belongs_to :project
end
has_many :globals
end

View Slide

# Project#global_errors
["MySQL::Error",
"MemCache::Error",
"Net::HTTPFatalError"]

View Slide

def global_key
"project-#{id}-globals"
end
def has_global?(name)
$redis.sismember(global_key, name)
end
end

View Slide

after_save :save_globals
def save_globals
$redis.del global_key
@globals.each do |g|
$redis.sadd global_key, g
end
end
end

View Slide

[Mysql::Error, MemCache::Error, Net::HTTPFatalError]
SISMEMBER SISMEMBER

View Slide

[]
DEL

View Slide

[Mysql::Error]
DEL
SADD

View Slide

[Mysql::Error, OpenURI::HTTPError]
DEL
SADD
SADD

View Slide

DEL
SADD
SADD
SISMEMBER
[]

View Slide

MULTI
EXEC
DEL
SADD
SADD
SISMEMBER

View Slide

after_save :save_globals
def save_globals
$redis.multi do
$redis.del global_key
@globals.each do |g|
$redis.sadd global_key, g
end
end
end
end

View Slide

global error lessons
avoid joins for simple data
consider race conditions
use append-only ﬁle (AOF)

View Slide

MULTIPLAYER
NOTEPAD
PUB/SUB
based on
http://github.com/laktek/realie

View Slide

# usage: ruby pub.rb room username
data = {"user" => ARGV[1]}
loop do
msg = STDIN.gets
$redis.publish ARGV[0],
data.merge('msg' => msg.strip).to_json
end

View Slide

# sub.rb
$redis = Redis.new(:timeout => 0)
$redis.subscribe('rubyonrails', 'rubymidwest') do |on|
on.message do |room, msg|
data = JSON.parse(msg)
puts "##{room} - [#{data['user']}]: #{data['msg']}"
end
end

View Slide

% ruby pub.rb rubymidwest qrush
i give up, i hate markdown
% ruby sub.rb
#rubymidwest - [qrush]: i give up, i hate markdown
#rubyonrails - [railsn00b]: undefined method posts_path? wtf?
#rubymidwest - [turbage]: seriously.

View Slide

multiplayer notepad
lessons
combine with other data structures
can subscribe to channels via patterns
concurrency in ruby is hard
use eventmachine! (or node.js)

View Slide

more to learn
know your data! (via @antirez)
command reference on the wiki
active IRC, mailing list

View Slide

AKASENTAI.com
redis in the cloud

View Slide

Thanks!
http://redis.io @qrush
http://rediscookbook.com
http://scr.bi/redispower

View Slide

BONUS ROUND!
I prepared way too many examples. Jackpot!

View Slide

URL
SHORTENER
STRINGS
based on
http://github.com/mattmatt/relink

View Slide

require 'sinatra'
require 'redis_url'
post '/' do
RedisUrl.new(params[:url]).save
end

View Slide

class RedisUrl
attr_accessor :url, :id
def initialize(url)
@url = url
@id = seed # unique string algorithm
end
def save
$redis.set("relink.url|#{@id}", @url)
$redis.set("relink.url.rev|#{@url}", @id)
end
end

View Slide

get %r{/(.+)} do |url|
u = RedisUrl.find(url)
if u
u.clicked
redirect u.url
else
status 404
end
end

View Slide

class RedisUrl
def self.find(id)
u = $redis.get("relink.url|#{id}")
if u
redis_url = RedisUrl.new(u)
redis_url.id = id
redis_url
end
end
def clicked
$redis.incr("relink.url.clicks|#{@id}")
end
end

View Slide

url shortener lessons
common pattern: namespacing
incr/decr assumes value is an integer
wrap behavior into ActiveRecord-like objects
next step: store URLs in a list

View Slide

LIVE
DEBUGGING
LISTS
based on
http://github.com/quirkey/redisk

View Slide

def after_save
begin
# make request to external service
rescue Exception => ex
logger.error "this shouldn't ever happen!"
logger.error ex
logger.error ex.backtrace
end
end

View Slide

# config/initializers/logger.rb
require 'redisk'
path = "#{Rails.env}.log"
config.logger = Redisk::Logger.new(path)

View Slide

class Redisk::IO
def write(string)
redis.rpush "#{name}:_list", string
end
def self.readlines(name)
redis.lrange("#{name}:_list", 0, -1)
end
end

View Slide

live debugging lessons
enables real-time data about your system
dump serialized/marshalled data fast
run the redis instance on a different box
dive deeper: hummingbird

View Slide

COUNTING
DOWNLOADS
STRINGS
SORTED SETS
HASHES
based on
http://github.com/rubygems/gemcutter

View Slide

# bad idea, dude
class Download < ActiveRecord::Base
belongs_to :rubygem
end
class Rubygem < ActiveRecord::Base
has_many :downloads
end

View Slide

class Download
def self.incr(rubygem)
$redis.incr("all")
$redis.incr(rubygem)
$redis.zincrby("today", 1, rubygem)
end
end

View Slide

class Download
def self.rollover(version)
$redis.rename "today", "yesterday"
dls = Hash[*$redis.zrange("yesterday",
0, -1,
:with_scores => true)]
dls.each do |key, score|
$redis.hincrby key, Date.today, score
Rubygem.find_by_name(key).increment!(:downloads, score)
end
end
end

View Slide

get "/api/v1/downloads/rails.json" do
$redis.hgetall("rails").to_json
end
# returns...
{
"2010-07-09" => 1908,
"2010-07-10" => 1032,
"2010-07-11" => 1091,
}

View Slide

counting downloads
lessons
hybrid approach does work!
redis is really not for search
test your migration away from SQL

View Slide

Redis: Persistence Power

Redis: Persistence Power

Nick Quaranto

More Decks by Nick Quaranto

Other Decks in Programming

Featured

Transcript