Redis First steps

A basic introduction to Redis

Transcript

1. First steps
2. HGETALL that:tomatoeyes:guy github.com/slok @slok69 slok69@gmail.com http://xlarrakoetxea.org Xabier Larrakoetxea
3. GET course:examples:repo https://github.com/slok/redis-first-steps.git
4. KEYS * • History • What is redis • Who uses it • Characteristics • Data types • Installing • Swiss knife • Use cases • Play!
5. ZRANGEBYSCORE redis:history 2009 2013 WITHSCORES • 2009 Salvatore (Antirez) Sanfilippo • 2009 Realtime analitycs • 2009 Antirez startup, redis in production • 2010 VMWare hires Antirez • 2010 Full time on Redis
6. GET redis:description Redis is an open-source, networked, in- memory, key-value data store with optional durability. It is written in ANSI C. The development of Redis is sponsored by VMware. Redis is the most popular key-value store Repeat after me: "Thank you wikipedia" :)
7. SMEMBERS redis:users • Twitter • Github • Yahoo • Instagram • Blizzard • Stackoverflow • Flicker • youporn And many others!
8. LRANGE redis:characteristics 0 -1 • "Key-value" store database • Open source • High performance • Always In memory • Possible Persistance • Swiss knife of DBs • Easy • Flexible
9. SMEMBERS redis:key.standars • No restriction with keys • Defacto standard • Key domains are splitted with ":" • Words are splitted with "." Key Example Correct? Linux SET Linux "Open source OS" ✔ Linux:{ID} HGETALL linux:5 ✔ Linux-{ID} ✘ Linux:debian.like:distros LLEN Linux:debian.like:distros ✔ Linux:RH like:distros ✘ Linux:{ID}:characteristics HGETALL Linux:5:characteristics ✔ Linux:characteristics:{ID} HGETALL Linux:characteristics:5 ✔
10. SMEMBERS redis:data.types • String • Hash • List • Set • Sorted set (zset)
11. GET redis:data.types:string Strings are the most basic kind of Redis value. A value is mapped to a key We could use SET and GET commands to set and get keys Redis Strings are binary safe, this means that a Redis string can contain any kind of data, for instance a JPEG image or a serialized Python object. Thanks to this, we could use it for different types of use cases: • Integer/float: As a counter (INCR, DECR, INCRBY, INCRBYFLOAT...) • Bit: as bitmaps/binary stuff (SETBIT, GETBIT, BITCOUNT, BITOP...) • String: regular DB (APPEND, GETRANGE...) redis 127.0.0.1:6379> set "key" "value" OK redis 127.0.0.1:6379> get "key" "value" - A String value can be at max 512 Megabytes in length! - Integer commands (INCR, DECR...) in redis are Atomic!
12. GET redis:data.types:string redis 127.0.0.1:6379> set "this:is:my:key" "Slok is awesome :O" OK redis 127.0.0.1:6379> keys * 1) "this:is:my:key" redis 127.0.0.1:6379> get "this:is:my:key" "Slok is awesome :O" redis 127.0.0.1:6379> incr "this:is:my:key:counter" (integer) 1 redis 127.0.0.1:6379> get "this:is:my:key:counter" "1" redis 127.0.0.1:6379> incr "this:is:my:key:counter" 5 (error) ERR wrong number of arguments for 'incr' command redis 127.0.0.1:6379> incrby "this:is:my:key:counter" 5 (integer) 6 redis 127.0.0.1:6379> decr "this:is:my:key:counter" (integer) 5
13. GET redis:data.types:hash Redis Hashes are maps between string fields and string values, so they are the perfect data type to represent objects Why use this instead of various String redis data structures? This has better performance, in memory access and data compression Every hash can store up to 232 - 1 field-value pairs (more than 4 billion) redis 127.0.0.1:6379> hset "myapp:person:1" "name" "Xabier" (integer) 1 redis 127.0.0.1:6379> hget "myapp:person:1" "name" "Xabier" redis 127.0.0.1:6379> hset "myapp:person:1" "last_name" "Larrakoetxea" OK redis 127.0.0.1:6379> hgetall "myapp:person:1" 1) "name" 2) "Xabier" 3) "last_name" 4) "Larrakoetxea" - Use Hash instead of strings if possible (If that make sense!). - Hash fields also could work as (atomic) counters!
14. GET redis:data.types:hash redis 127.0.0.1:6379> hmset "myapp:user:1" "id" 1 "username" "slok" "password" "WTFInPlainText!" OK redis 127.0.0.1:6379> hgetall "myapp:user:1" 1) "id" 2) "1" 3) "username" 4) "slok" 5) "password" 6) "WTFInPlainText!" redis 127.0.0.1:6379> hincrby "myapp:user:1" "id" 5 (integer) 6 redis 127.0.0.1:6379> hdel "myapp:user:1" "password" (integer) 1 redis 127.0.0.1:6379> hgetall "myapp:user:1" 1) "id" 2) "6" 3) "username" 4) "slok" redis 127.0.0.1:6379> hset "myapp:user:1" "username" "slok69" (integer) 0 redis 127.0.0.1:6379> hgetall "myapp:user:1" 1) "id" 2) "6" 3) "username" 4) "slok69"
15. GET redis:data.types:list Redis Lists are simply lists of strings, sorted by insertion order. It is possible to add elements to a Redis List pushing new elements on the head (on the left) or on the tail (on the right) of the list. redis 127.0.0.1:6379> lpush "myapp:admin:users.id" 4 (integer) 1 redis 127.0.0.1:6379> lpush "myapp:admin:users.id" 6 (integer) 2 redis 127.0.0.1:6379> rpush "myapp:admin:users.id" 1 (integer) 3 redis 127.0.0.1:6379> lrange "myapp:admin:users.id" 0 -1 1) "6" 2) "4" 3) "1" - When we add an item it returns the length of the list. - Constant time insertion and deletion O(1). - Access by index is O(N) Very fast! (sides access: O(1)). - The max length of a list is 232 - 1 elements (> 4 billion).
16. GET redis:data.types:list redis 127.0.0.1:6379> rpush "myapp:admin:users.id" 1 3 5 7 9 11 13 15 (integer) 8 redis 127.0.0.1:6379> lindex "myapp:admin:users.id" 0 "1" redis 127.0.0.1:6379> lrange "myapp:admin:users.id" 2 5 1) "5" 2) "7" 3) "9" 4) "11" redis 127.0.0.1:6379> rpop "myapp:admin:users.id" "15" redis 127.0.0.1:6379> llen "myapp:admin:users.id" (integer) 7 redis 127.0.0.1:6379> keys * 1) "myapp:admin:users.id" redis 127.0.0.1:6379> rpoplpush"myapp:admin:users.id" "myapp:editors:users.id" "13" redis 127.0.0.1:6379> keys * 1) "myapp:admin:users.id" 2) "myapp:editors:users.id" redis 127.0.0.1:6379> lrange "myapp:editors:users.id" 0 -1 1) "13"
17. GET redis:data.types:set Redis Sets are an unordered collection of Strings. It is possible to add, remove, and test for existence of members. Redis Sets have the desirable property of not allowing repeated members. Adding the same element multiple times will result in a set having a single copy of this element. redis 127.0.0.1:6379> sadd "myapp:admin:users.id" 1 2 3 4 5 6 7 8 (integer) 8 redis 127.0.0.1:6379> sadd "myapp:editors:users.id" 1 3 7 8 (integer) 4 redis 127.0.0.1:6379> sdiff "myapp:admin:users.id" "myapp:editors:users.id" 1) "2" 2) "4" 3) "5" 4) "6" - Supports unions, intersections, differences of sets very fast - The max length of a set is 232 - 1 elements (> 4 billion).
18. GET redis:data.types:set redis 127.0.0.1:6379> sadd "users:admin" 1 4 567 24 23 545 5 1 1234 57 89 (integer) 10 redis 127.0.0.1:6379> sadd "users:mod" 3 100 88 (integer) 3 redis 127.0.0.1:6379> sadd "users:editor" 77 66 12 (integer) 3 redis 127.0.0.1:6379> sunion "users:mod" "users:editor" 1) "3" 2) "12" 3) "66" 4) "77" 5) "88" 6) "100" redis 127.0.0.1:6379> smove "users:editor" "users:editor" 66 (integer) 1 redis 127.0.0.1:6379> sunionstore "users:mod.editor" "users:mod" "users:editor" (integer) 6 redis 127.0.0.1:6379> smembers "users:mod.editor" 1) "3" 2) "12" 3) "66" 4) "77" 5) "88" 6) "100"
19. GET redis:data.types:zset Redis Sorted Sets are, similarly to Redis Sets, non repeating collections of Strings. The difference is that every member of a Sorted Set is associated with score, that is used in order to take the sorted set ordered, from the smallest to the greatest score. While members are unique, scores may be repeated. redis 127.0.0.1:6379> zadd "prog:langs:90" 1991 "Python" 1993 "Lua" 1995 "Ruby" 1995 "Java" (integer) 4 redis 127.0.0.1:6379> zrange "prog:langs:90" 0 -1 1) "Python" 2) "Lua" 3) "Java" 4) "Ruby" - You can get ranges by score or by rank (position) - The max length of a zset is 232 - 1 elements (> 4 billion).
20. GET redis:data.types:zset redis 127.0.0.1:6379> zadd "gnu.linux:history" 1983 GNU 1991 Linux 1993 Slackware 1993 Debian 1994 S.u.S.E (integer) 5 redis 127.0.0.1:6379> zadd "windows:history" 1985 "1.0" 1987 "2.0" 1990 "3.0" 1993 "3.1" 1995 "95" (integer) 5 redis 127.0.0.1:6379> zrank "gnu.linux:history" Slackware (integer) 3 redis 127.0.0.1:6379> zscore "gnu.linux:history" Slackware "1993" redis 127.0.0.1:6379> zrangebyscore "gnu.linux:history" -inf +inf 1) "GNU" 2) "Linux" 3) "Debian" 4) "Slackware" 5) "S.u.S.E" redis 127.0.0.1:6379> zrevrangebyscore "gnu.linux:history" +inf -inf 1) "S.u.S.E" 2) "Slackware" 3) "Debian" 4) "Linux" 5) "GNU"
21. LRANGE redis:installation 0 -1 Compiling from source (for Unix like OS) $ tar xvf ./redis-2.6.12.tar.gz $ cd ./redis-2.6.12 $ make $ make test # make install Installing in Ubuntu # apt-get install redis-server Installing in Windows Redis doens't have official support for Windows. But still there are some alternatives made by de community (Open source FTW!): https://github.com/MSOpenTech/redis
22. LRANGE redis:installation 0 -1 Run with redis-server command $ redis-server [15252] 12 Apr 12:53:14.835 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf [15252] 12 Apr 12:53:14.837 # Unable to set the max number of files limit to 10032 (Operation not permitted), setting the max clients configuration to 3984. _._ _.-``__ ''-._ _.-`` `. `_. ''-._ Redis 2.6.12 (00000000/0) 64 bit .-`` .-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) Running in stand alone mode |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379 | `-._ `._ / _.-' | PID: 15252 `-._ `-._ `-./ _.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | http://redis.io `-._ `-._`-.__.-'_.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | `-._ `-._`-.__.-'_.-' _.-' `-._ `-.__.-' _.-' `-._ _.-' `-.__.-' [15252] 12 Apr 12:53:14.839 # Server started, Redis version 2.6.12 [15252] 12 Apr 12:53:14.840 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. [15252] 12 Apr 12:53:16.109 * DB loaded from disk: 1.269 seconds [15252] 12 Apr 12:53:16.109 * The server is now ready to accept connections on port 6379
23. LRANGE redis:swiss.knife 0 -1 Redis is not only for storing data and as a cache system. Redis has a lot of fancy stuff that is worth to know. • Pipelining • Transactions • Expire • Lua Scripting • Pub/sub
24. GET redis:swiss.knife:pipeline Pipeline is the possible to send multiple commands to the server without waiting for the replies at all, and finally read the replies in a single step. This is very powerful and reduces the time of request/response a lot Without pipelining: 125.34 seconds With pipelining: 24.8 seconds - The server will be forced to queue the replies, using memory. Split if mult. commands - No atomic execution. Between the list of commands the server could execute others For example this is the result for a loop that Increments 1 million new keys (not the same key, all the keys will finish with 1 value)
25. import time import redis TIMES = 1000000 def with_pipelining(): r = redis.Redis() p = r.pipeline() map(p.incr, range(TIMES)) p.execute() def without_pipelining(): r = redis.Redis() map(r.incr, range(TIMES)) start = time.clock() without_pipelining() print("Without pipelining: {0} seconds".format(time.clock() - start)) start = time.clock() with_pipelining() print("With pipelining: {0} seconds".format(time.clock() - start)) GET redis:swiss.knife:pipeline
26. GET redis:swiss.knife:transactions They allow the execution of a group of commands in a single step, with two important guarantees: • All the commands in a transaction are serialized and executed sequentially. It can never happen that a request issued by another client is served in the middle of the execution of a Redis transaction • Either all of the commands or none are processed, so a Redis transaction is also atomic. redis 127.0.0.1:6379> multi OK redis 127.0.0.1:6379> set "this" "will" "fail" QUEUED redis 127.0.0.1:6379> rpush "myList" 1 2 3 4 QUEUED redis 127.0.0.1:6379> exec 1) (error) ERR syntax error 2) (integer) 4 - Redis transactions doesn't support rollback - Transactions use more memory and are slower than pipelines
27. $ python ./redistransaction.py transaction Executing transaction result should be: 100000 result: 100000 $ python ./redistransaction.py pipeline Executing pipeline result shouldn't be: 100000 result: 455 GET redis:transactions:vs:pipelines Pipelines and transactions are similar but not the same. Transactions are slower and requiere more memory, On the other side pipelines guarantee that the operations are atomic Transaction doesn't support rollback, like the SQL databases. Why? • Redis commands can fail only if called with a wrong syntax (and the problem is not detectable during the command queueing) • Redis is internally simplified and faster because it does not need the ability to roll back
28. GET redis:swiss.knife:expire - Keys expiring is stored in Unix timestamps - Time is flowing even when the Redis instance is not active. - If you move an RDB file from two computers with a big desync in their clocks, funny things may happen Set a timeout on key. After the timeout has expired, the key will automatically be deleted. A key with an associated timeout is often said to be volatile in Redis terminology. redis 127.0.0.1:6379> set "myKey" "this will expire" OK redis 127.0.0.1:6379> expire "myKey" 5 (integer) 1 redis 127.0.0.1:6379> get "myKey" "this will expire" redis 127.0.0.1:6379> get "myKey" (nil)
29. import time import datetime import redis r = redis.Redis() key = "this:is:a:key" secs = 5 def expiration(secs): r.set(key, "some value") r.expire(key, secs) print("Expiration seconds: {0}".format(secs)) expiration(secs) print("result at {0}: {1}".format(datetime.datetime.now(), r.get(key))) for i in range(secs * 3): print("TTL: {0}".format(r.ttl(key))) time.sleep(0.4) print("result at {0}: {1}".format(datetime.datetime.now(), r.get(key))) GET redis:swiss.knife:expire
30. GET redis:swiss.knife:lua.scripting - No other script or Redis command will be executed while a script is being executed - while the script is running no other client can execute commands since the server is busy EVAL and EVALSHA are used to evaluate scripts using the Lua interpreter built into Redis starting from version 2.6.0. The first argument of EVAL is a Lua 5.1 script. The script does not need to define a Lua function (and should not). It is just a Lua program that will run in the context of the Redis server. redis 127.0.0.1:6379> eval "return redis.call('set', KEYS[1], math. random())" 1 "random:number" OK redis 127.0.0.1:6379> get "random:number" "0.17082803611217"
31. import redis fiboLuaScript = """ local i = tonumber(ARGV[1]) local first = 0 local second = 1 local res local function fibo(x, y, max) if max ~= 0 then res = redis.call('rpush',KEYS[1],x) return fibo(y, x+y, max -1) else return res end end return fibo(first, second, i) """ r = redis.Redis() key = "fibonacci:example" fibo_digits = 100 r.flushdb() fibonacci = r.register_script(fiboLuaScript) result = fibonacci(keys=[key], args=[fibo_digits]) print("result of calling fibonacci with lua in Redis: {0}".format(result)) print("fibonacci result:\n{0}".format(r.lrange(key, 0, -1))) GET redis:swiss.knife:lua.scripting
32. GET redis:swiss.knife:pubsub - We could use patters like ( users:* ) with PSUBSCRIBE & PUNSUBSCRIBE Redis implementation of publish-subscribe pattern: senders (publishers) are not programmed to send their messages to specific receivers (subscribers). Rather, published messages are characterized into channels, without knowledge of what (if any) subscribers there may be. Subscribers express interest in one or more channels, and only receive messages that are of interest, without knowledge of what (if any) publishers there are. Redis Channel Subscriber Subscriber Subscriber Subscriber Publisher Publisher
33. GET redis:swiss.knife:pubsub import sys import time import datetime import redis channel = "Redis:pubsub:example" r = redis.Redis() def publish(): while(True): r.publish(channel, datetime.datetime.now()) time.sleep(1) def subscribe(): p = r.pubsub(channel) p.subscribe(channel) for data in p.listen(): print data['data'] if sys.argv[1] == "pub": publish() elif sys.argv[1] == "sub": subscribe() else: print("use: command [pub|sub]")
34. LRANGE redis:user.cases 0 -1 • Cache system • Realtime metrics • "Regular database" • Notifications
35. GET redis:user.cases:cache We could use Redis as a cache system instead of memcached. How? We would use the EXPIRE and TTL and configure redis to not persist the data. Take into account Redis for now (is in experimental state) doesn't support clustering. So we need to use our own algorithm of hashing for redirecting the data to the different instances of redis. Redis VS Memcached http://stackoverflow.com/questions/10558465/memcache-vs-redis
36. GET redis:user.cases:metrics We could use Redis for realtime metrics How? We would use bitmaps. A bit in the bitmap is a flag, so for example each bitmap is an user id, bitmap 1 is user 1, 2 is 2 and so on. we could use SETBIT and GETBIT So now apart from having a cheap/small and fast O(1) mechanism of statistics we could also do operations with BITCOUNT & BITOP For example keys like user:logins:YYYY-MM-DD Maximun bit lenght is 232 -1 == 512 MB == 4294967296 bits (more than 4 billion users!!) More information: http://blog.getspool.com/2011/11/29/fast-easy-realtime-metrics-using- redis-bitmaps/
37. GET redis:user.cases:db If we want to use redis as a normal databases we could do it. but we have to take into account some things: • Redis operates in memory. All the data needs to be in memory all the time • SQL DBs stores immediately, Redis no, so we have to be prepared if something bad happens (next point ->) • Set master/slave instances for each redis instance (read only and read/write) • Partition your data into instances (until redis cluster is released) based on a custom system. More information: http://moot.it/blog/technology/redis-as-primary-datastore-wtf.html
38. GET redis:user.cases:notifications We could use Redis as a queue with the pub/sub system. How? Using the commands SUBSCRIBE & PUBLISH and creating different channels for the different queues. for example user:admin:notifications, user:mod:notifications... So Redis would do the hard work of notifying things and our job would be to process of the notifications. More information: https://github.com/slok/redis-node-push-notifications-example
39. STAHP !
40. LRANGE redis:examples 0 -1
41. GET redis:examples:django http://dwall.heroku.com/
42. GET redis:examples:play http://jwall.heroku.com/
43. • Redis logo: http://redis.io/images/redis-300dpi.png • Arrow logo: http://www.peterhajas.com/media/img/arrow.png • Mail logo: http://www.peterhajas.com/media/img/email_icon.png • Twitter logo: http://www.peterhajas.com/media/img/twitter_icon.png • Github logo: http://www.peterhajas.com/media/img/github_icon.png • Approval seal: http://allthingsordinary.se/images/original/569__seal-of-approval.jpg • Amazed lolcat: http://1.bp.blogspot.com/-0K6JOfXchxc/TdD- VwPr1mI/AAAAAAAADPA/YrYm0nsbaEw/s1600/funny-pictures-cat-is-amazed.jpg • Data types: http://mirrors.creativecommons.org/presskit/icons/remix.large.png • Victorinox logo: http://vectorise.net/vectorworks/logos/Company/download/Logo% 20Victorinox.png • Lolcat: http://updatesfromthefield.files.wordpress.com/2010/09/lolcat_what.jpg • Grumpycat: http://cdn.grumpycats.com/wp-content/uploads/2013/03/03.25.2013-v2- 625x416.jpg • Django logo: http://www.aewstudios.com/media/aew/img/logo_django.png • Play logo: http://blog.sudobits.com/wp-content/uploads/2012/03/play-logo.png • Creative commons: http://www.indt.edu.uy/imgs/by-nc-sa_big.png • Heroku logo: http://logos.heroku.com/images/heroku-logo-light-300x100.png LRANGE course:images 0 -1
44. • Redis webpage: http://redis.io • Redis python: https://github.com/andymccurdy/redis-py • Redis Java: https://github.com/xetorthio/jedis • Heroku PaaS: https://www.heroku.com/ • Django framework: https://www.djangoproject.com/ • Play framework: http://www.playframework.com • Github: https://github.com/ • Code highlighter: http://hilite.me/ LRANGE course:resources 0 -1
45. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License. Except the images that have their own license by the original Author Xabier Larrakoetxea (2013)