ElastiCache Antipatterns - AWS Community Day UAE

Transcript

1. Elastic Cache Anti-Patterns Aram Bhusal Head of Engineering - Consolidate Health @phoenixwizard me@aram.io
2. $whoami • Kingdom of Himalayas • Built on AWS since 2010 • Scale technology and teams • Head Engineering @ Consolidate Health • Mobility, Ed-Tech, E-Commerce etc. • @phoenixwizard in most places
3. What to Expect? • A primer • I have a hammer anti-pattern • Missing my car’s battery anti-pattern • Duct-taped life boat anti-pattern • Santa stuck in chimney anti-patten
4. • Make Redis a turtle anti-pattern • The annoying neighbour anti-pattern • Flying blind into the hurricane • Kayaking in the pacific Ocean
5. ElasticCache: The Mythical Silver Bullet • Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. • Memcached is Free & open source, high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.
6. Memcached Redis Sub-millisecond latency Yes Yes Developer ease of use Yes Yes Data partitioning Yes Yes Support for a broad set of programming languages Yes Yes Advanced data structures - Yes Multithreaded architecture Yes - Snapshots - Yes Replication - Yes Transactions - Yes Pub/Sub - Yes Lua scripting - Yes Geospatial support - Yes
7. I have a Hammer anti-pattern
8. Expand the world view • Binary-safe strings • Lists (basically linked list) • Sets (Unique unsorted string elements) • Sorted sets (Sets with every string having a score) • Hashes (similar to ruby or python hashes)
9. Look Beyond !! • Bitmaps (or bit arrays, handle string like array of bits) • HyperLogLogs (estimate cardinality of set [Probablistic!!]) • Streams (append only collection of map-like entries) • Geospatial indexes (whereami. Longitude comes before latitude)
11. Duct-Taped Life Boat anti-pattern
12. • Product READY • Mail blast DONE • Press Release DONE • 100x Traffic DONE • Server Crash DONE
13. “Is your server not able to handle load? Use ElasticCache” – Random article on the Web
14. • You now take data from DB • Transform the data and put it on ElasticCache Redis • Ssshh: The parameters used to transform the data are in your MIND
15. “When I wrote the code, only God and I knew why I wrote the code. But now …” – Famous meme from Stackoverflow
16. The instance goes DOWN. Node FAILURE • What do you think has happened? • And just like that your whole data evaporates !! • NO RECOVERY !!
17. What could you have done?
18. ElasticCache does do Persistence / Backups But, did you let ElasticCache help you?
19. RDB / BGSAVE to S3 / Awesome AWS Defaults
20. Gotchas? • You have set the interval for 5 minutes • What if server goes down 3 minutes after the snapshot? • Needs to fork, which is time consuming for big datasets
21. Append Only File • Popularly known as AOF • Changelog style persistent format • appendfsync can be “no/everysec/always”
22. Gotchas? • Not default with ElasticCache • AOF can be bigger and slower depending on sync policy • There are cases where AOF might not reproduce exactly same dataset on reloading
23. What we do? • DO BOTH • Have your cake and eat it !! • Do both RDB and AOF(set sync to everysec)
24. “Don’t just turn on AOF and restart your server” – – somewhere in Redis documentation
25. How do you do it in ElasticCache? • Create a parameter group with the appendonly parameter set to yes. • You then assign that parameter group to your cluster
26. One More Thing ??? • Depending on you use case evaluate • Multi AZ with replication groups turning AOF off • Cluster mode enabled or disabled • Need at least 2 replicas in different AZ
27. Missing my car’s battery anti- pattern
28. “Oh God!! We have no idea where the ElasticCache server gets it’s data from” – – Scary stuff DevOps Engineers hear from Backend Engineers
29. Hidden bash script is laughing at you sitting in a hidden location triggered by a cronjob !!
30. Santa stuck in chimney anti-pattern
31. • You are an amazing Engineer • Set up persistence perfectly • Now you have a millions of keys in your Redis instance • BUT YOUR REDIS IS RANDOMLY GOING SLOW
32. So WHAT HAPPENED? • There is a cron (the golden bullet) which reads & writes a MILLION records in Redis (the silver bullet)
33. “I have a single thread, nobody loves me” – – Redis getting bombarded by your cron
34. But I read Redis is not Single Threaded • More threads in background perform backend cleaning works • Since v4.0, deleting objects in background & blocking commands implemented by Redis modules
35. Make Redis a turtle anti-pattern
36. • You need to process all the keys, what are you going to do? Use the KEYS command? • But did you know this is a BLOCKING CALL?
37. But What can I do now? • Use NON-BLOCKING calls wherever possible !! • Instead of KEYS use SCAN? • SCAN is a cursor based call
38. What if I want a pattern? • Use MATCH Along with your SCAN • BEWARE: MATCH works on subset returned by SCAN
39. The annoying neighbor anti pattern
40. “Imagine all of us start talking to each other” – – All of us
41. REDIS CLUSTERS • They are not very well known to be as mature as other DB clusters • Not all commands & features available on single Redis work on partitions. Eg: MSET • Scale up and down might become tricky
42. • For high availability, there is the master-slave model • Data is partitioned among various Redis instances and remains available even when a small subset of nodes is unavailable
43. Flying blind into the hurricane
44. You should know there is a problem to solve the problem
45. Observability? • Load • CPU usage • Memory usage • Swap usage • Network Bandwidth • Disk usage
46. Redis Availability & Execution • connected_clients • keyspace • instantaneous_ops_per_sec • rdb_last_save_time • connected slaves • master_last_io_seconds_ago
47. Typical Failure Points • latency • used memory • memory fragmentation ratio • evicted keys • blocked clients
48. Kayaking in the pacific Ocean
49. • Does your App need Cluster mode on or not? • Which class of machines should you use for which features? • AMD vs Intel vs Gravitron (spoiler AMD will not be happy) • Have you considered latency due to swapping? • You do realize that your persistence strategy can increase latency. Right?
50. Redis Documentation is AWESOME !! So is the AWS Docs on ElasticCache
51. Quick Recap? • ElasticCache is not a silver Bullet • Redis defaults are not the best for everyone but ElasticCache has made things easier • Spend time and figure what works best for you • Make your own mistakes !!
52. @phoenixwizard me@aram.io Thank You !!