Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Redis Anti-Patterns

548d51c49e6b3605e987b396018b85d1?s=47 Aram
September 28, 2019

Redis Anti-Patterns

This talk is about Redis anti-patterns.
We talk about the various anti-patterns and gotchas which both startups and big organisations face during their journeys.

548d51c49e6b3605e987b396018b85d1?s=128

Aram

September 28, 2019
Tweet

Transcript

  1. Redis Anti-Patterns Aram Bhusal @phoenixwizard

  2. $whoami • Kingdom of Himalayas • Long term consultant •

    Own startup • Principal Engineer @ Zoomcar • @phoenixwizard in most places Image source: https://en.wikipedia.org/wiki/File:Mount_Everest_as_seen_from_Drukair2_PLW_edit.jpg
  3. What to expect • A primer • The “I have

    a hammer” anti-pattern • The “duct-taped life boat” anti-pattern • The “santa stuck in chimney” anti-patten
  4. What to expect • The “make Redis a turtle” anti-pattern

    • The “annoying neighbour” anti-pattern • Flying blind into the hurricane • Kayaking in the pacific Ocean
  5. The Mythical Silver Bullet Redis is an open source (BSD

    licensed), in-memory data structure store, used as a database, cache and message broker.
  6. The “I have a hammer” anti-pattern

  7. The Redis World • Binary-safe strings • Lists (basically linked

    list) • Sets (Unique unsorted string elements) • Sorted sets (Sets with every string having a score) • Hashes (similar to ruby or python hashes)
  8. Redis World (contd) • Bit arrays (or simply bitmaps, handle

    strings like an array of bits) • HyperLogLogs (estimate cardinality of set [Don’t be scared !! ]) • Streams (append only collection of map-like entries)
  9. The “Duct-Taped Life Boat” anti-pattern

  10. • Product READY • Mail blast DONE • Press Release

    DONE • 100x Traffic DONE • Server Crash DONE Image source: https://www.goodfreephotos.com/vector-images/wild-west-cowboy-cartoon-vector-clipart.png.php
  11. – Random article on the Web “Is your server not

    able to handle load? Use REDIS”
  12. • You now take data from DB • Transform the

    data and put it on Redis • Ssshh: The parameters used to transform the data are in your MIND
  13. Redis instance goes DOWN • What do you think has

    happened? • And just like that your whole data evaporates !! • NO RECOVERY !!
  14. What could you have done?

  15. Redis does do Persistence

  16. Redis Database File • Popularly called RDB • Snapshot style

    persistence Image source: https://pixabay.com/illustrations/office-files-aktenordner-file-2761159/
  17. GOTCHA? • You have set the interval for 5 minutes

    • What if server goes down 3 minutes after the snapshot? • Needs to fork, which is time consuming for big datasets
  18. Append Only File • Popularly known as AOF • Changelog

    style persistent format • appendfsync can be “no/everysec/always”
  19. GOTCHAS? • AOF Files bigger than RDB • AOF can

    be slower depending on sync policy • There are cases where AOF might not reproduce exactly same dataset on reloading
  20. WHAT IS THE BEST WAY? • Have your cake and

    eat it !! • Do both RDB and AOF(set sync to everysec)
  21. – somewhere in Redis documentation “Don’t just turn on AOF

    and restart your server”
  22. What NOT TO DO • You already have persistence set

    to RDB • Change it to AOF • And Restart the Server
  23. What you should DO? • config set appendonly yes

  24. The “Santa stuck in chimney” anti-patten Image source: https://www.maxpixel.net/Present-Chimney-Christmas-Santa-Stuck-Holiday-2946681

  25. • You are an amazing developer • Set up persistence

    perfectly • Now you have millions of keys in your Redis instance • BUT YOUR REDIS IS RANDOMLY GOING SLOW
  26. So WHAT HAPPENED? • There is a cron (the golden

    bullet) which reads & writes a MILLION records in Redis (the silver bullet)
  27. – Redis getting bombarded by your cron “I have a

    single thread, nobody loves me”
  28. But I read Redis is not Single Threaded • More

    threads in background perform backend cleaning works • Since v4.0, deleting objects in background & blocking commands implemented by Redis modules
  29. The “Make Redis a turtle” anti-pattern Image source: https://en.wikipedia.org/wiki/File:Kturtle_side_view.svg

  30. • You need to process all the keys, what are

    you going to do? Use the KEYS command? • But did you know this is a BLOCKING CALL?
  31. But What can I do now? • Use NON-BLOCKING calls

    wherever possible !! • Instead of KEYS use SCAN? • SCAN is a cursor based call
  32. What if I want a pattern? • Use MATCH Along

    with your SCAN • BEWARE: MATCH works on subset returned by SCAN
  33. The “annoying neighbour” anti-pattern Image source: https://pixabay.com/illustrations/loudspeaker-man-boy-holding-1459128/

  34. REDIS CLUSTERS • They are not very well known to

    be as mature as other DB clusters • Not all commands & features available on single Redis work on partitions • Scale up and down might become tricky
  35. • For high availability, there is the master-slave model •

    Data is partitioned among various Redis instances and remains available even when a small subset of nodes is unavailable
  36. Flying blind into the hurricane Image source: https://www.needpix.com/photo/87076/plane-cartoon-line-art-colouring-book-children-book-drawing-sketch

  37. You should know there is a problem to solve the

    problem
  38. Required processes running? • Is the process running? • What

    is the uptime?
  39. System resources utilisation • Load • CPU usage • Memory

    usage • Swap usage • Network Bandwidth • Disk usage
  40. Redis Availability & Execution • connected_clients • keyspace • instantaneous_ops_per_sec

    • rdb_last_save_time • connected slaves • master_last_io_seconds_ago
  41. Typical Failure Points • latency • used memory • memory

    fragmentation ratio • evicted keys • blocked clients
  42. WHO WILL BE THY SAVIOUR?

  43. • Redis-cli info command • New Relic • Nagios/Icinga •

    Redislabs hosted service
  44. Kayaking in the pacific Ocean Image source: https://www.needpix.com/photo/87076/plane-cartoon-line-art-colouring-book-children-book-drawing-sketch

  45. • How fast is your fork()? • How is it

    going to affect your Redis & Other Linux Processes
  46. • Have you considered latency due to swapping? • This

    again is not limited to Redis!!
  47. • You do realise that your persistence strategy can increase

    latency. Right?
  48. • Network Latency • CPUs (Large ones with large caches)

    [Intel yaaay] • RAM Speed (>10 kb objects) • Bare Metal vs VM
  49. Redis Documentation is AWESOME

  50. I Mean SUPER AWESOME !! Image source: https://www.needpix.com/photo/87076/plane-cartoon-line-art-colouring-book-children-book-drawing-sketch

  51. Quick Conclusion? • Redis is not a silver Bullet •

    Redis defaults are not the best for everyone • Spend time and figure what works best for you • Redis does shine in a lot of places • Make your own mistakes !!
  52. @phoenixwizard me@aram.io Thank You !! Image source: https://www.needpix.com/photo/87076/plane-cartoon-line-art-colouring-book-children-book-drawing-sketch