Lessons learned using Redis

Lessons learned using Redis, lighting talk at Madrid Devops

1. -FTTPOT MFBSOFE VTJOH Fernando Blat @ferblape - fernando.blat.es jueves 24 de mayo de 2012
2. %JTDMBJNFS jueves 24 de mayo de 2012
3. jueves 24 de mayo de 2012
4. Developer jueves 24 de mayo de 2012
5. Redis is: an open source, advanced key-value store a data structure server since keys can contain strings, hashes, lists, sets and sorted sets. http://redis.io jueves 24 de mayo de 2012
6. Redis is: an open source, advanced key-value store a data structure server since keys can contain strings, hashes, lists, sets and sorted sets. http://redis.io jueves 24 de mayo de 2012
7. jueves 24 de mayo de 2012
8. jueves 24 de mayo de 2012
9. 1FSTJTUFODF RDB Snapshot of the whole database AOF (Append Only File) logs Operation logs http://antirez.com/post/redis-persistence-demystified.html http://redis.io/topics/persistence jueves 24 de mayo de 2012
10. ################################ SNAPSHOTTING ################################# # # Save the DB on disk: # # save <seconds> <changes> # # Will save the DB if both the given number of seconds and the given # number of write operations against the DB occurred. # # In the example below the behaviour will be to save: # after 900 sec (15 min) if at least 1 key changed # after 300 sec (5 min) if at least 10 keys changed # after 60 sec if at least 10000 keys changed # # Note: you can disable saving at all commenting all the "save" lines. save 1800 1 save 1200 10 save 300 10000 # Compress string objects using LZF when dump .rdb databases? # For default that's set to 'yes' as it's almost always a win. # If you want to save some CPU in the saving child set it to 'no' but # the dataset will likely be bigger if you have compressible values or keys. rdbcompression yes # The filename where to dump the DB dbfilename dump.rdb # The working directory. # # The DB will be written inside this directory, with the filename specified # above using the 'dbfilename' configuration directive. # # Also the Append Only File will be created inside this directory. # # Note that you must specify a directory here, not a file name. dir /var/lib/redis jueves 24 de mayo de 2012
11. #BDLVQT $ crontab -l ... cp /var/lib/redis/dump.rdb \ /mnt/backups/dump-`date +%Y%m%d`.rdb jueves 24 de mayo de 2012
12. 'JSTU JTTVF jueves 24 de mayo de 2012
13. Redis compiled in 32 bits jueves 24 de mayo de 2012
14. 4FDPOE JTTVF jueves 24 de mayo de 2012
15. Redis will use all the memory available if needed jueves 24 de mayo de 2012
16. $ top top - 19:48:36 up 90 days, 17:53, 1 user, load average: 0.17, 0.39, 0.46 Tasks: 119 total, 1 running, 114 sleeping, 0 stopped, 4 zombie Cpu(s): 6.3%us, 0.5%sy, 0.0%ni, 93.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 8160804k total, 8064100k used, 96704k free, 68580k buffers Swap: 262140k total, 249968k used, 12172k free, 2083720k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5630 redis 20 0 4158m 3.9g 572 S 0 49.7 116:32.46 /usr/local/bin/redis-server jueves 24 de mayo de 2012
17. $ top top - 19:48:36 up 90 days, 17:53, 1 user, load average: 0.17, 0.39, 0.46 Tasks: 119 total, 1 running, 114 sleeping, 0 stopped, 4 zombie Cpu(s): 6.3%us, 0.5%sy, 0.0%ni, 93.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 8160804k total, 8064100k used, 96704k free, 68580k buffers Swap: 262140k total, 249968k used, 12172k free, 2083720k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5630 redis 20 0 4158m 3.9g 572 S 0 49.7 116:32.46 /usr/local/bin/redis-server jueves 24 de mayo de 2012
18. $ top top - 19:48:36 up 90 days, 17:53, 1 user, load average: 0.17, 0.39, 0.46 Tasks: 119 total, 1 running, 114 sleeping, 0 stopped, 4 zombie Cpu(s): 6.3%us, 0.5%sy, 0.0%ni, 93.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 8160804k total, 8064100k used, 96704k free, 68580k buffers Swap: 262140k total, 249968k used, 12172k free, 2083720k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5630 redis 20 0 4158m 3.9g 572 S 0 49.7 116:32.46 /usr/local/bin/redis-server jueves 24 de mayo de 2012
19. $ top top - 19:48:36 up 90 days, 17:53, 1 user, load average: 0.17, 0.39, 0.46 Tasks: 119 total, 1 running, 114 sleeping, 0 stopped, 4 zombie Cpu(s): 6.3%us, 0.5%sy, 0.0%ni, 93.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 8160804k total, 8064100k used, 96704k free, 68580k buffers Swap: 262140k total, 249968k used, 12172k free, 2083720k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5630 redis 20 0 4158m 3.9g 572 S 0 49.7 116:32.46 /usr/local/bin/redis-server jueves 24 de mayo de 2012
20. $ top top - 19:48:36 up 90 days, 17:53, 1 user, load average: 0.17, 0.39, 0.46 Tasks: 119 total, 1 running, 114 sleeping, 0 stopped, 4 zombie Cpu(s): 6.3%us, 0.5%sy, 0.0%ni, 93.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 8160804k total, 8064100k used, 96704k free, 68580k buffers Swap: 262140k total, 249968k used, 12172k free, 2083720k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5630 redis 20 0 4158m 3.9g 572 S 0 49.7 116:32.46 /usr/local/bin/redis-server jueves 24 de mayo de 2012
21. $ top top - 19:48:36 up 90 days, 17:53, 1 user, load average: 0.17, 0.39, 0.46 Tasks: 119 total, 1 running, 114 sleeping, 0 stopped, 4 zombie Cpu(s): 6.3%us, 0.5%sy, 0.0%ni, 93.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 8160804k total, 8064100k used, 96704k free, 68580k buffers Swap: 262140k total, 249968k used, 12172k free, 2083720k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5630 redis 20 0 4158m 3.9g 572 S 0 49.7 116:32.46 /usr/local/bin/redis-server jueves 24 de mayo de 2012
22. $ top top - 19:48:36 up 90 days, 17:53, 1 user, load average: 0.17, 0.39, 0.46 Tasks: 119 total, 1 running, 114 sleeping, 0 stopped, 4 zombie Cpu(s): 6.3%us, 0.5%sy, 0.0%ni, 93.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 8160804k total, 8064100k used, 96704k free, 68580k buffers Swap: 262140k total, 249968k used, 12172k free, 2083720k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5630 redis 20 0 4158m 3.9g 572 S 0 49.7 116:32.46 /usr/local/bin/redis-server jueves 24 de mayo de 2012
23. # This is the path of the Redis swap file. As you can guess, swap files # can't be shared by different Redis instances, so make sure to use a swap # file for every redis process you are running. Redis will complain if the # swap file is already in use. # # The best kind of storage for the Redis swap file (that's accessed at random) # is a Solid State Disk (SSD). # # *** WARNING *** if you are using a shared hosting the default of putting # the swap file under /tmp is not secure. Create a dir with access granted # only to Redis user and configure Redis to create the swap file there. vm-swap-file /tmp/redis.swap # vm-max-memory configures the VM to use at max the specified amount of # RAM. Everything that deos not fit will be swapped on disk *if* possible, that # is, if there is still enough contiguous space in the swap file. # # With vm-max-memory 0 the system will swap everything it can. Not a good # default, just specify the max amount of RAM you can in bytes, but it's # better to leave some margin. For instance specify an amount of RAM # that's more or less between 60 and 80% of your free RAM. vm-max-memory 5368709120 7JSUVBM .FNPSZ jueves 24 de mayo de 2012
24. 7JSUVBM .FNPSZ # Virtual Memory allows Redis to work with datasets bigger than the actual # amount of RAM needed to hold the whole dataset in memory. # In order to do so very used keys are taken in memory while the other keys # are swapped into a swap file, similarly to what operating systems do # with memory pages. # # To enable VM just set 'vm-enabled' to yes, and set the following three # VM parameters accordingly to your needs. vm-enabled yes jueves 24 de mayo de 2012
25. 5IJSE JTTVF jueves 24 de mayo de 2012
26. dump.rdb does not contain all the data, at least, in version 2.2 8IFO 3FEJT TUBSUT UP “TXBQ” jueves 24 de mayo de 2012
27. #(4"7& http://redis.io/commands/bgsave Save the DB in background. Redis forks, the parent continues to server the clients, the child saves the DB on disk then exit. A client my be able to check if the operation succeeded using the LASTSAVE command. jueves 24 de mayo de 2012
28. #BDLVQT #!/bin/bash SLEEP=30 function get_last_save { rm /tmp/previous_last_save echo "lastsave" | redis-cli > /tmp/previous_last_save lastsave=`cat /tmp/previous_last_save` } get_last_save previous_last_save=$lastsave echo "bgsave" | redis-cli while [ $lastsave = $previous_last_save ]; do sleep $SLEEP get_last_save done cp /var/lib/redis/dump.rdb /var/lib/redis/backups/dump-`date '+%Y%m%d%H%M'`.rdb # Remove backups older than 60 days find /var/lib/redis/backups -mtime +60 | xargs rm jueves 24 de mayo de 2012
29. 4VNNBSJ[JOH jueves 24 de mayo de 2012
30. 4VNNBSJ[JOH know your tools invest time check your backups jueves 24 de mayo de 2012
31. 5IBOL ZPV jueves 24 de mayo de 2012