Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Redis Steady Go (Nov 2012)

Redis Steady Go (Nov 2012)

Slides from All Your Base Conf in Oxford, England.

509e15b158eaac5ab3a5332a3f89d4a7?s=128

Peter Cooper

November 23, 2012
Tweet

Transcript

  1. Peter Cooper presents.. A rapid tour of Redis, the super-fast

    data structure server! Friday, 23 November 12
  2. Friday, 23 November 12

  3. Salvatore San!lippo a.k.a. antirez Friday, 23 November 12

  4. WHAT IS REDIS? Friday, 23 November 12

  5. remote dictionary server Friday, 23 November 12

  6. remote hash server Friday, 23 November 12

  7. data structure server! Friday, 23 November 12

  8. data structure server! isn’t that a “database”? Friday, 23 November

    12
  9. in a way, but.. no “tables” no SQL no enforced

    relationships lots of working with “primitives” isn’t that a “database”? Friday, 23 November 12
  10. think less.. and more.. name age user 1 jim 21

    user 2 amy 44 user:1:name = Jim user:1:age = 21 user:2:name = Amy user:2:age = 44 just keys and values.. a la SQLite, MySQL, etc.. Friday, 23 November 12
  11. In Ruby: a = “Hello world” b = “This is

    a test” storing some strings Friday, 23 November 12
  12. In Redis: SET A “Hello world” SET B “This is

    a test” storing some strings Friday, 23 November 12
  13. And to retrieve: GET A GET B storing some strings

    Friday, 23 November 12
  14. 1 0 1 0 1 0 1 0 like an

    assembly language? mov rax, 0x0a68732f push rax xor rax, rax mov rsi, rsp ... Friday, 23 November 12
  15. 1 0 1 0 1 0 1 0 SURE! :-)

    but it’s hard to categorize Friday, 23 November 12
  16. DSL for abstract data types and that’s from the Redis

    manifesto! google “redis manifesto” Friday, 23 November 12
  17. open source (BSD) !rst public release in 2009 VMware hired

    Salvatore in 2010 (and Pieter Noordhuis!) Friday, 23 November 12
  18. add more commands add persistence add more data types and

    more.. familiar with memcached? Friday, 23 November 12
  19. LIVE DEMO TIME! Friday, 23 November 12

  20. REDIS HOSTING Friday, 23 November 12

  21. redistogo.com garantiadata.com openredis.com redis4you.com redisgreen.net Friday, 23 November 12

  22. USES FOR REDIS Friday, 23 November 12

  23. The three biggies.. Friday, 23 November 12

  24. The three biggies.. Database Friday, 23 November 12

  25. The three biggies.. Database Messaging Friday, 23 November 12

  26. The three biggies.. Database Messaging Cache Friday, 23 November 12

  27. Real time logging LLOOGGs! Just keep pushing stu" onto a

    list.. LLOOGG was/is a service run by Salvatore that Redis was born out of. Friday, 23 November 12
  28. INCR pageviews:127.0.0.1:2012-11-23 INCR pageviews:127.0.0.1:2012-11-23-[HH] SADD users:logged_in:2012-11-23 [user_id] HINCRBY users:gina total_logins

    Friday, 23 November 12
  29. Rate limiting INCR hits:33.12.32.22 EXPIRE hits:33.12.32.22 3600 INCR hits:33.12.32.22 It’s

    a bit naive but.. you get the idea. Friday, 23 November 12
  30. Scoreboarding Sony did/does this Sorted sets are ideal! Add entries

    to a sorted set and query for results later. SuperDude1 20 pts xx_idiot_xx 19 pts mrkewl 11 pts freddieM 2 pts Friday, 23 November 12
  31. Inter Process Communication (IPC) Talking between programs Use lists, a

    pre-agreed set of keys OR.. PUB/SUB! Friday, 23 November 12
  32. Job queues resque does this! Push jobs onto a queue

    and forget about them (sort of..!) Friday, 23 November 12
  33. Session storage redis-store library can help with this for your

    Ruby webapps SET session:[session_key] “{ stuff: 123, blah: 234 }” Friday, 23 November 12
  34. Cache Think like old school memcached usage Some folks use

    Redis and memcached, keeping memcached for cache only and Redis for the more complex stu" Friday, 23 November 12
  35. YouP***.com Yep, really. Redis is their primary datastore. Alexa ranking

    104 Friday, 23 November 12
  36. GroupOn, Stack Over#ow, Craigslist Tumblr, Microsoft There’s a lot more

    than this but just to make the point.. Redis is not some crazy far out technology. & 200+ Redis servers at Pinterest..! Friday, 23 November 12
  37. Friday, 23 November 12

  38. coder.io Friday, 23 November 12

  39. CHARACTERISTICS Friday, 23 November 12

  40. Single threaded and event driven Think Node.js or a simple

    EventMachine loop (technically in Redis 2.4+ some threads are used for background IO tasks but not servicing requests) Friday, 23 November 12
  41. Individual operations are atomic Nothing can interrupt or clash with

    a command you issue. Remember: Redis is single threaded processed just one at a time.. Friday, 23 November 12
  42. Data can be “expired” Want a key/value to die in

    60 seconds? You got it. INCR hits:33.12.32.22 EXPIRE hits:33.12.32.22 3600 EXPIREAT hits:33.12.32.22 1928374758 Or cache invalidation! Friday, 23 November 12
  43. Everything is stored in memory It has persistence but.. your

    entire dataset is in memory too. If it can’t !t, stu"* will be pruned. * - special technical term Friday, 23 November 12
  44. Everything is stored in memory THIS MAKES REDIS SUPER FAST!

    Friday, 23 November 12
  45. But it does have persistence! This makes Redis actually useful

    ;-) RDB “snapshots” and AOF logs Forks in order to do dumps and “copy on write” saves a lot of pain Friday, 23 November 12
  46. Did I mention SUPER FAST? Think 50k+ GETs and SETs

    per second. Friday, 23 November 12
  47. Lots of Redis clients C, C#, C++, Clojure, Lisp, Ruby,

    Python, Go, Erlang Scala, Objective C, Lua, Node, PHP, Smalltalk, Tcl Some languages even have ORMs supporting Redis. Like Ohm in Ruby. DataMapper has an adapter too.And Python’s redis_wrap. Friday, 23 November 12
  48. Scriptable (with Lua) A handy way to keep more complex

    operations atomic Server can also store scripts and recall them using SHA1 hash Friday, 23 November 12
  49. EXEC if (redis.call("type",KEYS[1]) ~= "string" or redis.call("get",KEYS[1]) ~= ARGV[1]) then

    redis.call("set",KEYS[1],ARGV[2]) end , 1, “some key” Friday, 23 November 12
  50. EXECSHA 1ad692e76fa983ee712cabb, 1, “some key” Friday, 23 November 12

  51. Can be used from PostgreSQL with a foreign data wrapper

    CREATE EXTENSION redis_fdw; CREATE SERVER redis_server FOREIGN DATA WRAPPER redis_fdw OPTIONS (address '127.0.0.1', port '6379'); CREATE FOREIGN TABLE redis_db0 (key text, value text) SERVER redis_server OPTIONS (database '0'); ... SELECT id, email, value as visits FROM users, redis_db0 WHERE ('user_' || cast(id as text)) = cast(redis_db0.key as text) AND cast(value as int) > 40; Friday, 23 November 12
  52. DATA TYPES Friday, 23 November 12

  53. 5 Data Types Strings Lists Sets Sorted Sets Hashes Friday,

    23 November 12
  54. STRINGS As simple as it gets.. Friday, 23 November 12

  55. SET mystring “hello world” ./redis-cli Redis command line client app

    GET mystring returns “hello world” ./redis-cli Key Value Friday, 23 November 12
  56. GETSET MGET SETNX SETEX MSET MSETNX INCR INCRBY DECR DECRBY

    APPEND SUBSTR Works on strings that appear to be integers. Magic! Friday, 23 November 12
  57. GET hits:33.12.32.22 #=> (nil) INCR hits:33.12.32.22 #=> 1 It even

    does the initializing.. Friday, 23 November 12
  58. GENERIC OPERATIONS Yes, a short break with those data types..

    Friday, 23 November 12
  59. TYPE EXISTS RENAME DEL EXPIRE EXPIREAT TTL PERSIST Friday, 23

    November 12
  60. LISTS Lurvely linked lists with super-fast PUSH and POP Friday,

    23 November 12
  61. f e d c b a RPUSH LPOP LPUSH RPOP

    RPUSH my_q f e.g. Friday, 23 November 12
  62. f e d c b a } LRANGE 2 3

    LLEN == 6 LINDEX 5 X LREM 1 b Friday, 23 November 12
  63. f e d c b a RPUSH LPOP RPUSH my_q

    a RPUSH my_q b LPOP my_q == “a” LPOP my_q == “b” LPOP my_q == (nil) Or BLPOP to block (wait) until something can be popped Queues Not a native data type. Just a list! Friday, 23 November 12
  64. def enqueue(name, data) SADD queues [name] RPUSH queue:[name] [data.to_json] def

    dequeue(name) LPOP queue:[name] def length(name) LLEN queue:[name] def clear(name) DEL queue:[name] Friday, 23 November 12
  65. def enqueue(name, data) SADD queues [name] RPUSH queue:[name] [data.to_json] def

    dequeue(name) BLPOP queue:[name] 60 def length(name) LLEN queue:[name] def clear(name) DEL queue:[name] Friday, 23 November 12
  66. def enqueue(name, data) SADD queues [name] RPUSH queue:[name] [data.to_json] def

    dequeue(names) BLPOP *names (e.g. BLPOP queue:h queue:m queue:l) def length(name) LLEN queue:[name] def clear(name) DEL queue:[name] Friday, 23 November 12
  67. LTRIM myqueue 0 99 Want to keep a list just

    100 items long? Friday, 23 November 12
  68. SETS Think a list but with only unique elements. Friday,

    23 November 12
  69. SADD uniques 127.0.0.1 SADD uniques 127.0.0.2 SADD uniques 127.0.0.3 SCARD

    uniques # => 3 Friday, 23 November 12
  70. SADD contains:aba abacus SADD contains:aba cabal SADD contains:aba baba #

    .. etc .. SISMEMBER contains:aba abacus #=> 1 (or true) Friday, 23 November 12
  71. contains:aba contains:ase abacus cabal baba hello teabag base cabaret database

    vase vaseline baseline uncase unbased phase database tease SADD contains:ase suitcase SREM contains:aba hello SMOVE contains:aba contains:ase base Friday, 23 November 12
  72. contains:aba contains:ase abacus cabal baba teabag cabaret database vase vaseline

    baseline unbased phase database suitcase SCARD contains:aba == 6 SISMEMBER contains:aba chips == 0 (meaning false) SRANDMEMBER contains:aba == “teabag” SMEMBERS contains:ase == vase, vaseline, baseline, unbased, phase, database, suitcase Friday, 23 November 12
  73. contains:aba contains:ase abacus cabal baba teabag cabaret vase vaseline baseline

    unbased phase suitcase SINTER contains:aba contains:ase == database database This is only a simple example. SINTER can take any number of arguments! SUNION is another command that will join sets together. SDIFF will give you the difference. Friday, 23 November 12
  74. contains:aba contains:ase abacus cabal baba teabag cabaret vase vaseline baseline

    unbased phase suitcase SINTERSTORE resultset contains:aba contains:ase database SUNIONSTORE does the same for set unions. database resultset Friday, 23 November 12
  75. $id = INCR next_user_id SET user:$id peterc SADD starts:p $id

    SADD starts:pe $id SADD starts:pet $id ... etc. Imagine an autocomplete system.. Friday, 23 November 12
  76. $text = “pet” $ids = SMEMBERS starts:$text MGET *user:$ids And

    to query.. Not actual syntax, implying.. MGET user:1 user:2 user:3, etc. Friday, 23 November 12
  77. Things like inverted indexes, graphs, spell checkers, and similar data

    oriented tasks are also simple using Redis’ primitives. Friday, 23 November 12
  78. SORTED SETS a.k.a. “scored” sets Like the sets we saw

    before, but with a “score” for each member too. Friday, 23 November 12
  79. ZADD scoreboard 10 “Johnny” ZADD scoreboard 8 “Fred” ZADD scoreboard

    20 “Amy” ZREVRANGE scoreboard 0 1 A very short example.. .. gets us the top 2 scorers! Friday, 23 November 12
  80. HASHES Like Ruby’s hashes but with string-only values. Friday, 23

    November 12
  81. Yep, you can only use strings in a hash. No

    super duper weird recursive data structures for you! Friday, 23 November 12
  82. HSET user:1 name “John” HGET user:1 name Yet another very

    short example.. Other commands include HGETALL, HKEYS, HVALUES, HINCRBY, etc.. name John age 20 gender male user:1 Friday, 23 November 12
  83. OTHER STUFF! Friday, 23 November 12

  84. SERVER COMMANDS Friday, 23 November 12

  85. SAVE BGSAVE INFO MONITOR SHUTDOWN FLUSHDB QUIT and more.. Friday,

    23 November 12
  86. TRANSACTIONS Friday, 23 November 12

  87. MULTI INCR a INCR b EXEC This is when “stu!”

    actually happens. Friday, 23 November 12
  88. WATCH somekey GET somekey MULTI SET someotherkey $x EXEC You

    can also ensure a transaction fails if a dependency is broken. Put result into $x This fails if somekey changed. (check and set) Friday, 23 November 12
  89. REPLICATION Friday, 23 November 12

  90. Very simple master/slave replication. Server 1 Server 2 12.34.56.78:6378 12.34.56.78:6379

    Friday, 23 November 12
  91. Simple master/slave replication. Server 1 Server 2 12.34.56.78:6378 12.34.56.78:6379 SLAVEOF

    12.34.56.78 6378 Friday, 23 November 12
  92. Clients can hammer server 1 while server 2 does persistence,

    perhaps? Server 1 Server 2 SAVE disk Friday, 23 November 12
  93. Want to do a live replacement of Redis without a!ecting

    clients? REPLICATE AND SWITCH! Friday, 23 November 12
  94. FUTURE Friday, 23 November 12

  95. Redis Sentinel Distributed monitoring Noti"cations Auto failover Client con"guration server

    Friday, 23 November 12
  96. Redis Cluster Distributed and fault tolerant Redis Essentially.. auto-sharding (to

    4096 slots) Only a subset of Redis (no complex multi-key ops) Friday, 23 November 12
  97. Redis on Windows Slightly touchy point Both the community &

    Microsoft have done some work MS has 64 bit version coming soon Friday, 23 November 12
  98. IN CLOSING Friday, 23 November 12

  99. search for.. “try redis” “redis documentation” “the little redis book”

    “redis [your language]” Friday, 23 November 12
  100. or visit.. redis.io #redis on irc.freenode.net @redisfeed Friday, 23 November

    12
  101. redis.io Friday, 23 November 12

  102. coming mid 2013.. out in 2011 very short & simple

    beta out, print in 2013 Friday, 23 November 12
  103. bye! peter@peterc.org @peterc Friday, 23 November 12