Redis Steady Go (Nov 2012)

Slides from All Your Base Conf in Oxford, England.

Transcript

1. Peter Cooper presents.. A rapid tour of Redis, the super-fast data structure server! Friday, 23 November 12
2. Friday, 23 November 12
3. Salvatore San!lippo a.k.a. antirez Friday, 23 November 12
4. WHAT IS REDIS? Friday, 23 November 12
5. remote dictionary server Friday, 23 November 12
6. remote hash server Friday, 23 November 12
7. data structure server! Friday, 23 November 12
8. data structure server! isn’t that a “database”? Friday, 23 November 12
9. in a way, but.. no “tables” no SQL no enforced relationships lots of working with “primitives” isn’t that a “database”? Friday, 23 November 12
10. think less.. and more.. name age user 1 jim 21 user 2 amy 44 user:1:name = Jim user:1:age = 21 user:2:name = Amy user:2:age = 44 just keys and values.. a la SQLite, MySQL, etc.. Friday, 23 November 12
11. In Ruby: a = “Hello world” b = “This is a test” storing some strings Friday, 23 November 12
12. In Redis: SET A “Hello world” SET B “This is a test” storing some strings Friday, 23 November 12
13. And to retrieve: GET A GET B storing some strings Friday, 23 November 12
14. 1 0 1 0 1 0 1 0 like an assembly language? mov rax, 0x0a68732f push rax xor rax, rax mov rsi, rsp ... Friday, 23 November 12
15. 1 0 1 0 1 0 1 0 SURE! :-) but it’s hard to categorize Friday, 23 November 12
16. DSL for abstract data types and that’s from the Redis manifesto! google “redis manifesto” Friday, 23 November 12
17. open source (BSD) !rst public release in 2009 VMware hired Salvatore in 2010 (and Pieter Noordhuis!) Friday, 23 November 12
18. add more commands add persistence add more data types and more.. familiar with memcached? Friday, 23 November 12
19. LIVE DEMO TIME! Friday, 23 November 12
20. REDIS HOSTING Friday, 23 November 12
21. redistogo.com garantiadata.com openredis.com redis4you.com redisgreen.net Friday, 23 November 12
22. USES FOR REDIS Friday, 23 November 12
23. The three biggies.. Friday, 23 November 12
24. The three biggies.. Database Friday, 23 November 12
25. The three biggies.. Database Messaging Friday, 23 November 12
26. The three biggies.. Database Messaging Cache Friday, 23 November 12
27. Real time logging LLOOGGs! Just keep pushing stu" onto a list.. LLOOGG was/is a service run by Salvatore that Redis was born out of. Friday, 23 November 12
28. INCR pageviews:127.0.0.1:2012-11-23 INCR pageviews:127.0.0.1:2012-11-23-[HH] SADD users:logged_in:2012-11-23 [user_id] HINCRBY users:gina total_logins Friday, 23 November 12
29. Rate limiting INCR hits:33.12.32.22 EXPIRE hits:33.12.32.22 3600 INCR hits:33.12.32.22 It’s a bit naive but.. you get the idea. Friday, 23 November 12
30. Scoreboarding Sony did/does this Sorted sets are ideal! Add entries to a sorted set and query for results later. SuperDude1 20 pts xx_idiot_xx 19 pts mrkewl 11 pts freddieM 2 pts Friday, 23 November 12
31. Inter Process Communication (IPC) Talking between programs Use lists, a pre-agreed set of keys OR.. PUB/SUB! Friday, 23 November 12
32. Job queues resque does this! Push jobs onto a queue and forget about them (sort of..!) Friday, 23 November 12
33. Session storage redis-store library can help with this for your Ruby webapps SET session:[session_key] “{ stuff: 123, blah: 234 }” Friday, 23 November 12
34. Cache Think like old school memcached usage Some folks use Redis and memcached, keeping memcached for cache only and Redis for the more complex stu" Friday, 23 November 12
35. YouP***.com Yep, really. Redis is their primary datastore. Alexa ranking 104 Friday, 23 November 12
36. GroupOn, Stack Over#ow, Craigslist Tumblr, Microsoft There’s a lot more than this but just to make the point.. Redis is not some crazy far out technology. & 200+ Redis servers at Pinterest..! Friday, 23 November 12
37. Friday, 23 November 12
38. coder.io Friday, 23 November 12
39. CHARACTERISTICS Friday, 23 November 12
40. Single threaded and event driven Think Node.js or a simple EventMachine loop (technically in Redis 2.4+ some threads are used for background IO tasks but not servicing requests) Friday, 23 November 12
41. Individual operations are atomic Nothing can interrupt or clash with a command you issue. Remember: Redis is single threaded processed just one at a time.. Friday, 23 November 12
42. Data can be “expired” Want a key/value to die in 60 seconds? You got it. INCR hits:33.12.32.22 EXPIRE hits:33.12.32.22 3600 EXPIREAT hits:33.12.32.22 1928374758 Or cache invalidation! Friday, 23 November 12
43. Everything is stored in memory It has persistence but.. your entire dataset is in memory too. If it can’t !t, stu"* will be pruned. * - special technical term Friday, 23 November 12
44. Everything is stored in memory THIS MAKES REDIS SUPER FAST! Friday, 23 November 12
45. But it does have persistence! This makes Redis actually useful ;-) RDB “snapshots” and AOF logs Forks in order to do dumps and “copy on write” saves a lot of pain Friday, 23 November 12
46. Did I mention SUPER FAST? Think 50k+ GETs and SETs per second. Friday, 23 November 12
47. Lots of Redis clients C, C#, C++, Clojure, Lisp, Ruby, Python, Go, Erlang Scala, Objective C, Lua, Node, PHP, Smalltalk, Tcl Some languages even have ORMs supporting Redis. Like Ohm in Ruby. DataMapper has an adapter too.And Python’s redis_wrap. Friday, 23 November 12
48. Scriptable (with Lua) A handy way to keep more complex operations atomic Server can also store scripts and recall them using SHA1 hash Friday, 23 November 12
49. EXEC if (redis.call("type",KEYS[1]) ~= "string" or redis.call("get",KEYS[1]) ~= ARGV[1]) then redis.call("set",KEYS[1],ARGV[2]) end , 1, “some key” Friday, 23 November 12
50. EXECSHA 1ad692e76fa983ee712cabb, 1, “some key” Friday, 23 November 12
51. Can be used from PostgreSQL with a foreign data wrapper CREATE EXTENSION redis_fdw; CREATE SERVER redis_server FOREIGN DATA WRAPPER redis_fdw OPTIONS (address '127.0.0.1', port '6379'); CREATE FOREIGN TABLE redis_db0 (key text, value text) SERVER redis_server OPTIONS (database '0'); ... SELECT id, email, value as visits FROM users, redis_db0 WHERE ('user_' || cast(id as text)) = cast(redis_db0.key as text) AND cast(value as int) > 40; Friday, 23 November 12
52. DATA TYPES Friday, 23 November 12
53. 5 Data Types Strings Lists Sets Sorted Sets Hashes Friday, 23 November 12
54. STRINGS As simple as it gets.. Friday, 23 November 12
55. SET mystring “hello world” ./redis-cli Redis command line client app GET mystring returns “hello world” ./redis-cli Key Value Friday, 23 November 12
56. GETSET MGET SETNX SETEX MSET MSETNX INCR INCRBY DECR DECRBY APPEND SUBSTR Works on strings that appear to be integers. Magic! Friday, 23 November 12
57. GET hits:33.12.32.22 #=> (nil) INCR hits:33.12.32.22 #=> 1 It even does the initializing.. Friday, 23 November 12
58. GENERIC OPERATIONS Yes, a short break with those data types.. Friday, 23 November 12
59. TYPE EXISTS RENAME DEL EXPIRE EXPIREAT TTL PERSIST Friday, 23 November 12
60. LISTS Lurvely linked lists with super-fast PUSH and POP Friday, 23 November 12
61. f e d c b a RPUSH LPOP LPUSH RPOP RPUSH my_q f e.g. Friday, 23 November 12
62. f e d c b a } LRANGE 2 3 LLEN == 6 LINDEX 5 X LREM 1 b Friday, 23 November 12
63. f e d c b a RPUSH LPOP RPUSH my_q a RPUSH my_q b LPOP my_q == “a” LPOP my_q == “b” LPOP my_q == (nil) Or BLPOP to block (wait) until something can be popped Queues Not a native data type. Just a list! Friday, 23 November 12
64. def enqueue(name, data) SADD queues [name] RPUSH queue:[name] [data.to_json] def dequeue(name) LPOP queue:[name] def length(name) LLEN queue:[name] def clear(name) DEL queue:[name] Friday, 23 November 12
65. def enqueue(name, data) SADD queues [name] RPUSH queue:[name] [data.to_json] def dequeue(name) BLPOP queue:[name] 60 def length(name) LLEN queue:[name] def clear(name) DEL queue:[name] Friday, 23 November 12
66. def enqueue(name, data) SADD queues [name] RPUSH queue:[name] [data.to_json] def dequeue(names) BLPOP *names (e.g. BLPOP queue:h queue:m queue:l) def length(name) LLEN queue:[name] def clear(name) DEL queue:[name] Friday, 23 November 12
67. LTRIM myqueue 0 99 Want to keep a list just 100 items long? Friday, 23 November 12
68. SETS Think a list but with only unique elements. Friday, 23 November 12
69. SADD uniques 127.0.0.1 SADD uniques 127.0.0.2 SADD uniques 127.0.0.3 SCARD uniques # => 3 Friday, 23 November 12
70. SADD contains:aba abacus SADD contains:aba cabal SADD contains:aba baba # .. etc .. SISMEMBER contains:aba abacus #=> 1 (or true) Friday, 23 November 12
71. contains:aba contains:ase abacus cabal baba hello teabag base cabaret database vase vaseline baseline uncase unbased phase database tease SADD contains:ase suitcase SREM contains:aba hello SMOVE contains:aba contains:ase base Friday, 23 November 12
72. contains:aba contains:ase abacus cabal baba teabag cabaret database vase vaseline baseline unbased phase database suitcase SCARD contains:aba == 6 SISMEMBER contains:aba chips == 0 (meaning false) SRANDMEMBER contains:aba == “teabag” SMEMBERS contains:ase == vase, vaseline, baseline, unbased, phase, database, suitcase Friday, 23 November 12
73. contains:aba contains:ase abacus cabal baba teabag cabaret vase vaseline baseline unbased phase suitcase SINTER contains:aba contains:ase == database database This is only a simple example. SINTER can take any number of arguments! SUNION is another command that will join sets together. SDIFF will give you the difference. Friday, 23 November 12
74. contains:aba contains:ase abacus cabal baba teabag cabaret vase vaseline baseline unbased phase suitcase SINTERSTORE resultset contains:aba contains:ase database SUNIONSTORE does the same for set unions. database resultset Friday, 23 November 12
75. $id = INCR next_user_id SET user:$id peterc SADD starts:p $id SADD starts:pe $id SADD starts:pet $id ... etc. Imagine an autocomplete system.. Friday, 23 November 12
76. $text = “pet” $ids = SMEMBERS starts:$text MGET *user:$ids And to query.. Not actual syntax, implying.. MGET user:1 user:2 user:3, etc. Friday, 23 November 12
77. Things like inverted indexes, graphs, spell checkers, and similar data oriented tasks are also simple using Redis’ primitives. Friday, 23 November 12
78. SORTED SETS a.k.a. “scored” sets Like the sets we saw before, but with a “score” for each member too. Friday, 23 November 12
79. ZADD scoreboard 10 “Johnny” ZADD scoreboard 8 “Fred” ZADD scoreboard 20 “Amy” ZREVRANGE scoreboard 0 1 A very short example.. .. gets us the top 2 scorers! Friday, 23 November 12
80. HASHES Like Ruby’s hashes but with string-only values. Friday, 23 November 12
81. Yep, you can only use strings in a hash. No super duper weird recursive data structures for you! Friday, 23 November 12
82. HSET user:1 name “John” HGET user:1 name Yet another very short example.. Other commands include HGETALL, HKEYS, HVALUES, HINCRBY, etc.. name John age 20 gender male user:1 Friday, 23 November 12
83. OTHER STUFF! Friday, 23 November 12
84. SERVER COMMANDS Friday, 23 November 12
85. SAVE BGSAVE INFO MONITOR SHUTDOWN FLUSHDB QUIT and more.. Friday, 23 November 12
86. TRANSACTIONS Friday, 23 November 12
87. MULTI INCR a INCR b EXEC This is when “stu!” actually happens. Friday, 23 November 12
88. WATCH somekey GET somekey MULTI SET someotherkey $x EXEC You can also ensure a transaction fails if a dependency is broken. Put result into $x This fails if somekey changed. (check and set) Friday, 23 November 12
89. REPLICATION Friday, 23 November 12
90. Very simple master/slave replication. Server 1 Server 2 12.34.56.78:6378 12.34.56.78:6379 Friday, 23 November 12
91. Simple master/slave replication. Server 1 Server 2 12.34.56.78:6378 12.34.56.78:6379 SLAVEOF 12.34.56.78 6378 Friday, 23 November 12
92. Clients can hammer server 1 while server 2 does persistence, perhaps? Server 1 Server 2 SAVE disk Friday, 23 November 12
93. Want to do a live replacement of Redis without a!ecting clients? REPLICATE AND SWITCH! Friday, 23 November 12
94. FUTURE Friday, 23 November 12
95. Redis Sentinel Distributed monitoring Noti"cations Auto failover Client con"guration server Friday, 23 November 12
96. Redis Cluster Distributed and fault tolerant Redis Essentially.. auto-sharding (to 4096 slots) Only a subset of Redis (no complex multi-key ops) Friday, 23 November 12
97. Redis on Windows Slightly touchy point Both the community & Microsoft have done some work MS has 64 bit version coming soon Friday, 23 November 12
98. IN CLOSING Friday, 23 November 12
99. search for.. “try redis” “redis documentation” “the little redis book” “redis [your language]” Friday, 23 November 12
100. or visit.. redis.io #redis on irc.freenode.net @redisfeed Friday, 23 November 12
101. redis.io Friday, 23 November 12
102. coming mid 2013.. out in 2011 very short & simple beta out, print in 2013 Friday, 23 November 12
103. bye! peter@peterc.org @peterc Friday, 23 November 12