AWS UG Abu Dhabi 2024 - ElastiCache Antipatterns

Transcript

1. ElastiCache Anti-Patterns Aram Bhusal Head of Engineering - Consolidate Health @phoenixwizard me@aram.io
2. $whoami •Kingdom of Himalayas •Built on AWS since 2010 •Scale technology and teams •Head Engineering @ Consolidate Health •Healthcare, Mobility, Ed-Tech, E-Commerce, Fintech etc. •@phoenixwizard in most places
3. What to Expect? •A primer •I have a hammer anti-pattern •Missing my car’s battery anti-pattern •Duct-taped life boat anti-pattern •Santa stuck in chimney anti-patten
4. •Make Redis a turtle anti-pattern •The annoying neighbour anti-pattern •Flying blind into the hurricane •Kayaking in the pacific Ocean
5. ElastiCache: The Mythical Silver Bullet • Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. (At least till version 7.3. New license from 7.4; Hi Valkey!!) • Memcached is Free & open source, high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.
6. Memcached Redis Sub-millisecond latency Yes Yes Developer ease of use Yes Yes Data partitioning Yes Yes Support for a broad set of programming languages Yes Yes Advanced data structures - Yes Multithreaded architecture Yes - Snapshots - Yes Replication - Yes Transactions - Yes Pub/Sub - Yes Lua scripting - Yes Geospatial support - Yes
7. SSHHHHHH… All fun new developments are in the last slide !!
8. I have a Hammer anti-pattern
9. Expand the world view • Binary-safe strings • 512 MB • Lists (basically linked list sorted insertion order) • Sorted insertion order • Implement stacks / Queues • Queue Management for Background Tasks (Sidekiq/Celery Anyone)? • Sets (Unique unsorted string elements) • Java Hashsets/Python sets → add, remove, and test for existence in O(1). • Track unique IPs / Set of users for role / union / Intersection
10. Look Beyond !! • Sorted sets (Sets with every string having a score) • Leaderboards, Rate limiters • Hashes (similar to ruby or python hashes) • Bitmaps (or bit arrays, RBAC Anyone?, Who has subscribed?) • HyperLogLogs (estimate cardinality of set [Probablistic!!] • Less than 1% error rate) • Streams (append only collection of map-like entries)
11. •Geospatial indexes (whereami. Longitude comes before latitude) Look Beyond !!
12. Duct-Taped Life Boat anti-pattern
13. •Product READY •Mail blast DONE •Press Release DONE •100x Traffic DONE •Server Crash DONE
14. “Is your server not able to handle load? Use ElastiCache” – Random article on the Web
15. •You now take data from DB •Transform the data and put it on ElastiCache Redis •Ssshh: The parameters used to transform the data are in your MIND
16. “When I wrote the code, only God and I knew why I wrote the code. But now …” – Famous meme from Stackoverflow
17. The instance goes DOWN. Node FAILURE •What do you think has happened? •And just like that your whole data evaporates !! •NO RECOVERY !!
18. What could you have done?
19. ElastiCache does do Persistence / Backups But, did you let ElastiCache help you?
20. RDB / BGSAVE to S3 / Awesome AWS Defaults
21. Gotchas? •You have set the interval for 5 minutes •What if server goes down 3 minutes after the snapshot? •Needs to fork, which is time consuming for big datasets
22. Append Only File Multi AZ •If you are running your own Redis instance, AOF • Changelog style persistent format • appendfsync can be “no/everysec/always”
23. Gotchas? •Not default with ElastiCache (Only available on versions 2.x) •AOF can be bigger and slower depending on sync policy •There are cases where AOF might not reproduce exactly same dataset on reloading
24. What we do? •DO BOTH •Have your cake and eat it !! •Do both RDB and AOF(set sync to everysec) / Set Multi AZ
25. “Don’t just turn on AOF and restart your server” – – somewhere in Redis documentation
26. How do you do it in ElastiCache? •Create a parameter group with the appendonly parameter set to yes. •You then assign that parameter group to your cluster
27. One More Thing ??? •Depending on you use case evaluate • Multi AZ with replication groups turning AOF off • Cluster mode enabled or disabled • Need at least 2 replicas in different AZ • Be careful about the retention periods according to your use case
28. Missing my car’s battery anti-pattern
29. “Oh God!! We have no idea where the ElastiCache server gets it’s data from” – – Scary stuff DevOps Engineers hear from Backend Engineers
30. Hidden bash script is laughing at you sitting in a hidden location triggered by a cronjob !!
31. Santa stuck in chimney anti-pattern
32. •You are an amazing Engineer •Set up persistence perfectly •Now you have a millions of keys in your Redis instance •BUT YOUR REDIS IS RANDOMLY GOING SLOW
33. So WHAT HAPPENED? •There is a cron (the golden bullet) which reads & writes a MILLION records in Redis (the silver bullet)
34. “I have a single thread, nobody loves me” – – Redis getting bombarded by your cron
35. But I read Redis is not Single Threaded •More threads in background perform backend cleaning works •Since v4.0, deleting objects in background & blocking commands implemented by Redis modules
36. Make Redis a turtle anti-pattern
37. •You need to process all the keys, what are you going to do? Use the KEYS command? •But did you know this is a BLOCKING CALL?
38. But What can I do now? •Use NON-BLOCKING calls wherever possible !! •Instead of KEYS use SCAN? •SCAN is a cursor based call
39. What if I want a pattern? •Use MATCH Along with your SCAN •BEWARE: MATCH works on subset returned by SCAN
40. The annoying neighbor anti pattern
41. “Imagine all of us start talking to each other” – – All of us
42. REDIS CLUSTERS •They are not very well known to be as mature as other DB clusters •Not all commands & features available on single Redis work on partitions. Eg: MSET •Scale up and down might become tricky
43. •For high availability, there is the master-slave model •Data is partitioned among various Redis instances and remains available even when a small subset of nodes is unavailable
44. Flying blind into the hurricane
45. You should know there is a problem to solve the problem
46. Observability? •Load •CPU usage •Memory usage •Swap usage •Network Bandwidth •Disk usage
47. Redis Availability & Execution •connected_clients •keyspace •instantaneous_ops_per_sec •rdb_last_save_time •connected slaves •master_last_io_seconds_ago
48. Typical Failure Points •latency •used memory •memory fragmentation ratio •evicted keys •blocked clients
49. Kayaking in the pacific Ocean
50. •Does your App need Cluster mode on or not? •Which class of machines should you use for which features? •AMD vs Intel vs Gravitron (spoiler AMD will not be happy) •Have you considered latency due to swapping? •You do realize that your persistence strategy can increase latency. Right?
51. Redis Documentation is AWESOME !! So is the AWS Docs on ElastiCache
53. SShhhhhh • License Change !! • BSD -> Dual license • Valkey • https://aws.amazon.com/blogs/opensource/why-aws-supports-valkey/ • Redislabs <-> Garantia Data and how it took over the OSS
54. Quick Recap? •ElastiCache is not a silver Bullet •Redis defaults are not the best for everyone but ElastiCache has made things easier •Spend time and figure what works best for you •Make your own mistakes !!
55. @phoenixwizard me@aram.io Thank You !!