Redis Anti-Patterns

This talk is about Redis anti-patterns.
We talk about the various anti-patterns and gotchas which both startups and big organisations face during their journeys.

Transcript

1. Redis Anti-Patterns Aram Bhusal @phoenixwizard
2. $whoami • Kingdom of Himalayas • Long term consultant • Own startup • Principal Engineer @ Zoomcar • @phoenixwizard in most places Image source: https://en.wikipedia.org/wiki/File:Mount_Everest_as_seen_from_Drukair2_PLW_edit.jpg
3. What to expect • A primer • The “I have a hammer” anti-pattern • The “duct-taped life boat” anti-pattern • The “santa stuck in chimney” anti-patten
4. What to expect • The “make Redis a turtle” anti-pattern • The “annoying neighbour” anti-pattern • Flying blind into the hurricane • Kayaking in the pacific Ocean
5. The Mythical Silver Bullet Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker.
6. The “I have a hammer” anti-pattern
7. The Redis World • Binary-safe strings • Lists (basically linked list) • Sets (Unique unsorted string elements) • Sorted sets (Sets with every string having a score) • Hashes (similar to ruby or python hashes)
8. Redis World (contd) • Bit arrays (or simply bitmaps, handle strings like an array of bits) • HyperLogLogs (estimate cardinality of set [Don’t be scared !! ]) • Streams (append only collection of map-like entries)
9. The “Duct-Taped Life Boat” anti-pattern
10. • Product READY • Mail blast DONE • Press Release DONE • 100x Traffic DONE • Server Crash DONE Image source: https://www.goodfreephotos.com/vector-images/wild-west-cowboy-cartoon-vector-clipart.png.php
11. – Random article on the Web “Is your server not able to handle load? Use REDIS”
12. • You now take data from DB • Transform the data and put it on Redis • Ssshh: The parameters used to transform the data are in your MIND
13. Redis instance goes DOWN • What do you think has happened? • And just like that your whole data evaporates !! • NO RECOVERY !!
14. What could you have done?
15. Redis does do Persistence
16. Redis Database File • Popularly called RDB • Snapshot style persistence Image source: https://pixabay.com/illustrations/office-files-aktenordner-file-2761159/
17. GOTCHA? • You have set the interval for 5 minutes • What if server goes down 3 minutes after the snapshot? • Needs to fork, which is time consuming for big datasets
18. Append Only File • Popularly known as AOF • Changelog style persistent format • appendfsync can be “no/everysec/always”
19. GOTCHAS? • AOF Files bigger than RDB • AOF can be slower depending on sync policy • There are cases where AOF might not reproduce exactly same dataset on reloading
20. WHAT IS THE BEST WAY? • Have your cake and eat it !! • Do both RDB and AOF(set sync to everysec)
21. – somewhere in Redis documentation “Don’t just turn on AOF and restart your server”
22. What NOT TO DO • You already have persistence set to RDB • Change it to AOF • And Restart the Server
23. What you should DO? • config set appendonly yes
24. The “Santa stuck in chimney” anti-patten Image source: https://www.maxpixel.net/Present-Chimney-Christmas-Santa-Stuck-Holiday-2946681
25. • You are an amazing developer • Set up persistence perfectly • Now you have millions of keys in your Redis instance • BUT YOUR REDIS IS RANDOMLY GOING SLOW
26. So WHAT HAPPENED? • There is a cron (the golden bullet) which reads & writes a MILLION records in Redis (the silver bullet)
27. – Redis getting bombarded by your cron “I have a single thread, nobody loves me”
28. But I read Redis is not Single Threaded • More threads in background perform backend cleaning works • Since v4.0, deleting objects in background & blocking commands implemented by Redis modules
29. The “Make Redis a turtle” anti-pattern Image source: https://en.wikipedia.org/wiki/File:Kturtle_side_view.svg
30. • You need to process all the keys, what are you going to do? Use the KEYS command? • But did you know this is a BLOCKING CALL?
31. But What can I do now? • Use NON-BLOCKING calls wherever possible !! • Instead of KEYS use SCAN? • SCAN is a cursor based call
32. What if I want a pattern? • Use MATCH Along with your SCAN • BEWARE: MATCH works on subset returned by SCAN
33. The “annoying neighbour” anti-pattern Image source: https://pixabay.com/illustrations/loudspeaker-man-boy-holding-1459128/
34. REDIS CLUSTERS • They are not very well known to be as mature as other DB clusters • Not all commands & features available on single Redis work on partitions • Scale up and down might become tricky
35. • For high availability, there is the master-slave model • Data is partitioned among various Redis instances and remains available even when a small subset of nodes is unavailable
36. Flying blind into the hurricane Image source: https://www.needpix.com/photo/87076/plane-cartoon-line-art-colouring-book-children-book-drawing-sketch
37. You should know there is a problem to solve the problem
38. Required processes running? • Is the process running? • What is the uptime?
39. System resources utilisation • Load • CPU usage • Memory usage • Swap usage • Network Bandwidth • Disk usage
40. Redis Availability & Execution • connected_clients • keyspace • instantaneous_ops_per_sec • rdb_last_save_time • connected slaves • master_last_io_seconds_ago
41. Typical Failure Points • latency • used memory • memory fragmentation ratio • evicted keys • blocked clients
42. WHO WILL BE THY SAVIOUR?
43. • Redis-cli info command • New Relic • Nagios/Icinga • Redislabs hosted service
44. Kayaking in the pacific Ocean Image source: https://www.needpix.com/photo/87076/plane-cartoon-line-art-colouring-book-children-book-drawing-sketch
45. • How fast is your fork()? • How is it going to affect your Redis & Other Linux Processes
46. • Have you considered latency due to swapping? • This again is not limited to Redis!!
47. • You do realise that your persistence strategy can increase latency. Right?
48. • Network Latency • CPUs (Large ones with large caches) [Intel yaaay] • RAM Speed (>10 kb objects) • Bare Metal vs VM
49. Redis Documentation is AWESOME
50. I Mean SUPER AWESOME !! Image source: https://www.needpix.com/photo/87076/plane-cartoon-line-art-colouring-book-children-book-drawing-sketch
51. Quick Conclusion? • Redis is not a silver Bullet • Redis defaults are not the best for everyone • Spend time and figure what works best for you • Redis does shine in a lot of places • Make your own mistakes !!
52. @phoenixwizard me@aram.io Thank You !! Image source: https://www.needpix.com/photo/87076/plane-cartoon-line-art-colouring-book-children-book-drawing-sketch