$30 off During Our Annual Pro Sale. View Details »

Redis Steady Go (Nov 2012)

Peter Cooper
November 23, 2012

Redis Steady Go (Nov 2012)

Slides from All Your Base Conf in Oxford, England.

Peter Cooper

November 23, 2012
Tweet

More Decks by Peter Cooper

Other Decks in Programming

Transcript

  1. Peter Cooper presents..
    A rapid tour of Redis,
    the super-fast data
    structure server!
    Friday, 23 November 12

    View Slide

  2. Friday, 23 November 12

    View Slide

  3. Salvatore San!lippo
    a.k.a. antirez
    Friday, 23 November 12

    View Slide

  4. WHAT IS REDIS?
    Friday, 23 November 12

    View Slide

  5. remote dictionary server
    Friday, 23 November 12

    View Slide

  6. remote hash server
    Friday, 23 November 12

    View Slide

  7. data structure server!
    Friday, 23 November 12

    View Slide

  8. data structure server!
    isn’t that a “database”?
    Friday, 23 November 12

    View Slide

  9. in a way, but..
    no “tables”
    no SQL
    no enforced relationships
    lots of working with “primitives”
    isn’t that a “database”?
    Friday, 23 November 12

    View Slide

  10. think less..
    and more..
    name age
    user 1 jim 21
    user 2 amy 44
    user:1:name = Jim
    user:1:age = 21
    user:2:name = Amy
    user:2:age = 44
    just keys and values..
    a la SQLite, MySQL, etc..
    Friday, 23 November 12

    View Slide

  11. In Ruby:
    a = “Hello world”
    b = “This is a test”
    storing some strings
    Friday, 23 November 12

    View Slide

  12. In Redis:
    SET A “Hello world”
    SET B “This is a test”
    storing some strings
    Friday, 23 November 12

    View Slide

  13. And to retrieve:
    GET A
    GET B
    storing some strings
    Friday, 23 November 12

    View Slide

  14. 1
    0
    1
    0
    1
    0
    1
    0
    like an assembly language?
    mov rax, 0x0a68732f
    push rax
    xor rax, rax
    mov rsi, rsp
    ...
    Friday, 23 November 12

    View Slide

  15. 1
    0
    1
    0
    1
    0
    1
    0
    SURE! :-)
    but it’s hard to categorize
    Friday, 23 November 12

    View Slide

  16. DSL for abstract data types
    and that’s from the Redis manifesto!
    google “redis manifesto”
    Friday, 23 November 12

    View Slide

  17. open source (BSD)
    !rst public release in 2009
    VMware hired Salvatore in 2010
    (and Pieter Noordhuis!)
    Friday, 23 November 12

    View Slide

  18. add more commands
    add persistence
    add more data types
    and more..
    familiar with memcached?
    Friday, 23 November 12

    View Slide

  19. LIVE DEMO TIME!
    Friday, 23 November 12

    View Slide

  20. REDIS HOSTING
    Friday, 23 November 12

    View Slide

  21. redistogo.com
    garantiadata.com
    openredis.com
    redis4you.com
    redisgreen.net
    Friday, 23 November 12

    View Slide

  22. USES FOR REDIS
    Friday, 23 November 12

    View Slide

  23. The three biggies..
    Friday, 23 November 12

    View Slide

  24. The three biggies..
    Database
    Friday, 23 November 12

    View Slide

  25. The three biggies..
    Database Messaging
    Friday, 23 November 12

    View Slide

  26. The three biggies..
    Database Messaging Cache
    Friday, 23 November 12

    View Slide

  27. Real time logging
    LLOOGGs!
    Just keep pushing stu" onto a list..
    LLOOGG was/is a service run by Salvatore that Redis was born out of.
    Friday, 23 November 12

    View Slide

  28. INCR pageviews:127.0.0.1:2012-11-23
    INCR pageviews:127.0.0.1:2012-11-23-[HH]
    SADD users:logged_in:2012-11-23 [user_id]
    HINCRBY users:gina total_logins
    Friday, 23 November 12

    View Slide

  29. Rate limiting
    INCR hits:33.12.32.22
    EXPIRE hits:33.12.32.22 3600
    INCR hits:33.12.32.22
    It’s a bit naive but.. you get the idea.
    Friday, 23 November 12

    View Slide

  30. Scoreboarding
    Sony did/does this
    Sorted sets are ideal!
    Add entries to a sorted set and query for results later.
    SuperDude1 20 pts
    xx_idiot_xx 19 pts
    mrkewl 11 pts
    freddieM 2 pts
    Friday, 23 November 12

    View Slide

  31. Inter Process Communication (IPC)
    Talking between programs
    Use lists, a pre-agreed set of keys OR.. PUB/SUB!
    Friday, 23 November 12

    View Slide

  32. Job queues
    resque does this!
    Push jobs onto a queue and forget about them (sort of..!)
    Friday, 23 November 12

    View Slide

  33. Session storage
    redis-store library can help with this for your Ruby webapps
    SET session:[session_key] “{ stuff: 123, blah: 234 }”
    Friday, 23 November 12

    View Slide

  34. Cache
    Think like old school memcached usage
    Some folks use Redis and memcached, keeping memcached for cache
    only and Redis for the more complex stu"
    Friday, 23 November 12

    View Slide

  35. YouP***.com
    Yep, really. Redis is their primary datastore.
    Alexa ranking 104
    Friday, 23 November 12

    View Slide

  36. GroupOn, Stack Over#ow, Craigslist
    Tumblr, Microsoft
    There’s a lot more than this but just to make the point..
    Redis is not some crazy far out technology.
    & 200+ Redis servers at Pinterest..!
    Friday, 23 November 12

    View Slide

  37. Friday, 23 November 12

    View Slide

  38. coder.io
    Friday, 23 November 12

    View Slide

  39. CHARACTERISTICS
    Friday, 23 November 12

    View Slide

  40. Single threaded and event driven
    Think Node.js or a simple EventMachine loop
    (technically in Redis 2.4+ some threads are used for background IO tasks but not servicing requests)
    Friday, 23 November 12

    View Slide

  41. Individual operations are atomic
    Nothing can interrupt or clash with a command you issue.
    Remember: Redis is single threaded
    processed just one at a time..
    Friday, 23 November 12

    View Slide

  42. Data can be “expired”
    Want a key/value to die in 60 seconds? You got it.
    INCR hits:33.12.32.22
    EXPIRE hits:33.12.32.22 3600
    EXPIREAT hits:33.12.32.22 1928374758
    Or cache invalidation!
    Friday, 23 November 12

    View Slide

  43. Everything is stored in memory
    It has persistence but.. your entire dataset is in memory too.
    If it can’t !t, stu"* will be pruned.
    * - special technical term
    Friday, 23 November 12

    View Slide

  44. Everything is stored in memory
    THIS MAKES REDIS SUPER FAST!
    Friday, 23 November 12

    View Slide

  45. But it does have persistence!
    This makes Redis actually useful ;-)
    RDB “snapshots” and AOF logs
    Forks in order to do dumps and “copy on write” saves a lot of pain
    Friday, 23 November 12

    View Slide

  46. Did I mention SUPER FAST?
    Think 50k+ GETs and SETs per second.
    Friday, 23 November 12

    View Slide

  47. Lots of Redis clients
    C, C#, C++, Clojure, Lisp, Ruby, Python, Go, Erlang
    Scala, Objective C, Lua, Node, PHP, Smalltalk, Tcl
    Some languages even have ORMs supporting Redis.
    Like Ohm in Ruby. DataMapper has an adapter too.And Python’s redis_wrap.
    Friday, 23 November 12

    View Slide

  48. Scriptable (with Lua)
    A handy way to keep more complex operations atomic
    Server can also store scripts and recall them using SHA1 hash
    Friday, 23 November 12

    View Slide

  49. EXEC
    if (redis.call("type",KEYS[1]) ~= "string" or
    redis.call("get",KEYS[1]) ~= ARGV[1])
    then
    redis.call("set",KEYS[1],ARGV[2])
    end
    , 1, “some key”
    Friday, 23 November 12

    View Slide

  50. EXECSHA 1ad692e76fa983ee712cabb,
    1, “some key”
    Friday, 23 November 12

    View Slide

  51. Can be used from PostgreSQL
    with a foreign data wrapper
    CREATE EXTENSION redis_fdw;
    CREATE SERVER redis_server
    FOREIGN DATA WRAPPER redis_fdw
    OPTIONS (address '127.0.0.1', port '6379');
    CREATE FOREIGN TABLE redis_db0 (key text, value text)
    SERVER redis_server
    OPTIONS (database '0');
    ...
    SELECT id, email, value as visits FROM users, redis_db0 WHERE ('user_'
    || cast(id as text)) = cast(redis_db0.key as text) AND cast(value as
    int) > 40;
    Friday, 23 November 12

    View Slide

  52. DATA TYPES
    Friday, 23 November 12

    View Slide

  53. 5 Data Types
    Strings
    Lists
    Sets
    Sorted Sets
    Hashes
    Friday, 23 November 12

    View Slide

  54. STRINGS
    As simple as it gets..
    Friday, 23 November 12

    View Slide

  55. SET mystring “hello world”
    ./redis-cli
    Redis command
    line client app
    GET mystring returns
    “hello world”
    ./redis-cli
    Key Value
    Friday, 23 November 12

    View Slide

  56. GETSET
    MGET
    SETNX
    SETEX
    MSET
    MSETNX
    INCR
    INCRBY
    DECR
    DECRBY
    APPEND
    SUBSTR
    Works on
    strings that
    appear to be
    integers.
    Magic!
    Friday, 23 November 12

    View Slide

  57. GET hits:33.12.32.22
    #=> (nil)
    INCR hits:33.12.32.22
    #=> 1
    It even does the initializing..
    Friday, 23 November 12

    View Slide

  58. GENERIC OPERATIONS
    Yes, a short break with those data types..
    Friday, 23 November 12

    View Slide

  59. TYPE
    EXISTS
    RENAME
    DEL
    EXPIRE
    EXPIREAT
    TTL
    PERSIST
    Friday, 23 November 12

    View Slide

  60. LISTS
    Lurvely linked lists with super-fast PUSH and POP
    Friday, 23 November 12

    View Slide

  61. f
    e
    d
    c
    b
    a
    RPUSH
    LPOP
    LPUSH
    RPOP
    RPUSH my_q f
    e.g.
    Friday, 23 November 12

    View Slide

  62. f
    e
    d
    c
    b
    a
    }
    LRANGE 2 3
    LLEN == 6
    LINDEX 5
    X
    LREM 1 b
    Friday, 23 November 12

    View Slide

  63. f
    e
    d
    c
    b
    a
    RPUSH
    LPOP RPUSH my_q a
    RPUSH my_q b
    LPOP my_q == “a”
    LPOP my_q == “b”
    LPOP my_q == (nil)
    Or BLPOP to block (wait)
    until something can be
    popped
    Queues
    Not a native data type. Just a list!
    Friday, 23 November 12

    View Slide

  64. def enqueue(name, data)
    SADD queues [name]
    RPUSH queue:[name] [data.to_json]
    def dequeue(name)
    LPOP queue:[name]
    def length(name)
    LLEN queue:[name]
    def clear(name)
    DEL queue:[name]
    Friday, 23 November 12

    View Slide

  65. def enqueue(name, data)
    SADD queues [name]
    RPUSH queue:[name] [data.to_json]
    def dequeue(name)
    BLPOP queue:[name] 60
    def length(name)
    LLEN queue:[name]
    def clear(name)
    DEL queue:[name]
    Friday, 23 November 12

    View Slide

  66. def enqueue(name, data)
    SADD queues [name]
    RPUSH queue:[name] [data.to_json]
    def dequeue(names)
    BLPOP *names (e.g. BLPOP queue:h queue:m queue:l)
    def length(name)
    LLEN queue:[name]
    def clear(name)
    DEL queue:[name]
    Friday, 23 November 12

    View Slide

  67. LTRIM myqueue 0 99
    Want to keep a list just 100 items long?
    Friday, 23 November 12

    View Slide

  68. SETS
    Think a list but with only unique elements.
    Friday, 23 November 12

    View Slide

  69. SADD uniques 127.0.0.1
    SADD uniques 127.0.0.2
    SADD uniques 127.0.0.3
    SCARD uniques # => 3
    Friday, 23 November 12

    View Slide

  70. SADD contains:aba abacus
    SADD contains:aba cabal
    SADD contains:aba baba
    # .. etc ..
    SISMEMBER contains:aba abacus
    #=> 1 (or true)
    Friday, 23 November 12

    View Slide

  71. contains:aba
    contains:ase
    abacus cabal baba hello teabag
    base cabaret database
    vase vaseline baseline uncase
    unbased phase database tease
    SADD contains:ase suitcase
    SREM contains:aba hello
    SMOVE contains:aba contains:ase base
    Friday, 23 November 12

    View Slide

  72. contains:aba
    contains:ase
    abacus cabal baba teabag
    cabaret database
    vase vaseline baseline unbased
    phase database suitcase
    SCARD contains:aba == 6
    SISMEMBER contains:aba chips == 0 (meaning false)
    SRANDMEMBER contains:aba == “teabag”
    SMEMBERS contains:ase == vase, vaseline,
    baseline, unbased,
    phase, database,
    suitcase
    Friday, 23 November 12

    View Slide

  73. contains:aba
    contains:ase
    abacus cabal baba teabag
    cabaret
    vase vaseline baseline
    unbased phase suitcase
    SINTER contains:aba contains:ase == database
    database
    This is only a simple example. SINTER can take any number of arguments!
    SUNION is another command that will join sets together.
    SDIFF will give you the difference.
    Friday, 23 November 12

    View Slide

  74. contains:aba
    contains:ase
    abacus cabal baba teabag
    cabaret
    vase vaseline baseline
    unbased phase suitcase
    SINTERSTORE resultset contains:aba contains:ase
    database
    SUNIONSTORE does the same for set unions.
    database
    resultset
    Friday, 23 November 12

    View Slide

  75. $id = INCR next_user_id
    SET user:$id peterc
    SADD starts:p $id
    SADD starts:pe $id
    SADD starts:pet $id
    ... etc.
    Imagine an autocomplete system..
    Friday, 23 November 12

    View Slide

  76. $text = “pet”
    $ids = SMEMBERS starts:$text
    MGET *user:$ids
    And to query..
    Not actual syntax, implying.. MGET user:1 user:2 user:3, etc.
    Friday, 23 November 12

    View Slide

  77. Things like inverted indexes,
    graphs, spell checkers, and similar data
    oriented tasks are also simple using
    Redis’ primitives.
    Friday, 23 November 12

    View Slide

  78. SORTED SETS
    a.k.a. “scored” sets
    Like the sets we saw before, but
    with a “score” for each member too.
    Friday, 23 November 12

    View Slide

  79. ZADD scoreboard 10 “Johnny”
    ZADD scoreboard 8 “Fred”
    ZADD scoreboard 20 “Amy”
    ZREVRANGE scoreboard 0 1
    A very short example..
    .. gets us the top 2 scorers!
    Friday, 23 November 12

    View Slide

  80. HASHES
    Like Ruby’s hashes but with string-only values.
    Friday, 23 November 12

    View Slide

  81. Yep, you can only use strings in a hash.
    No super duper weird recursive data structures for you!
    Friday, 23 November 12

    View Slide

  82. HSET user:1 name “John”
    HGET user:1 name
    Yet another very short example..
    Other commands include HGETALL, HKEYS, HVALUES, HINCRBY, etc..
    name John
    age 20
    gender male
    user:1
    Friday, 23 November 12

    View Slide

  83. OTHER STUFF!
    Friday, 23 November 12

    View Slide

  84. SERVER COMMANDS
    Friday, 23 November 12

    View Slide

  85. SAVE
    BGSAVE
    INFO
    MONITOR
    SHUTDOWN
    FLUSHDB
    QUIT
    and more..
    Friday, 23 November 12

    View Slide

  86. TRANSACTIONS
    Friday, 23 November 12

    View Slide

  87. MULTI
    INCR a
    INCR b
    EXEC This is when “stu!” actually
    happens.
    Friday, 23 November 12

    View Slide

  88. WATCH somekey
    GET somekey
    MULTI
    SET someotherkey $x
    EXEC
    You can also ensure a transaction fails
    if a dependency is broken.
    Put result into $x
    This fails if somekey changed.
    (check and set)
    Friday, 23 November 12

    View Slide

  89. REPLICATION
    Friday, 23 November 12

    View Slide

  90. Very simple master/slave replication.
    Server 1 Server 2
    12.34.56.78:6378 12.34.56.78:6379
    Friday, 23 November 12

    View Slide

  91. Simple master/slave replication.
    Server 1 Server 2
    12.34.56.78:6378 12.34.56.78:6379
    SLAVEOF 12.34.56.78 6378
    Friday, 23 November 12

    View Slide

  92. Clients can hammer server 1 while
    server 2 does persistence, perhaps?
    Server 1 Server 2
    SAVE
    disk
    Friday, 23 November 12

    View Slide

  93. Want to do a live replacement of Redis
    without a!ecting clients?
    REPLICATE AND SWITCH!
    Friday, 23 November 12

    View Slide

  94. FUTURE
    Friday, 23 November 12

    View Slide

  95. Redis Sentinel
    Distributed monitoring
    Noti"cations
    Auto failover
    Client con"guration server
    Friday, 23 November 12

    View Slide

  96. Redis Cluster
    Distributed and fault tolerant Redis
    Essentially.. auto-sharding (to 4096 slots)
    Only a subset of Redis (no complex multi-key ops)
    Friday, 23 November 12

    View Slide

  97. Redis on Windows
    Slightly touchy point
    Both the community & Microsoft have
    done some work
    MS has 64 bit version coming soon
    Friday, 23 November 12

    View Slide

  98. IN CLOSING
    Friday, 23 November 12

    View Slide

  99. search for..
    “try redis”
    “redis documentation”
    “the little redis book”
    “redis [your language]”
    Friday, 23 November 12

    View Slide

  100. or visit..
    redis.io
    #redis on irc.freenode.net
    @redisfeed
    Friday, 23 November 12

    View Slide

  101. redis.io
    Friday, 23 November 12

    View Slide

  102. coming mid 2013..
    out in 2011
    very short & simple
    beta out, print in 2013
    Friday, 23 November 12

    View Slide

  103. bye!
    [email protected]
    @peterc
    Friday, 23 November 12

    View Slide