PicCollage migration of its ‘likes’ feature

Transcript

1. PicCollage migration of its ‘likes’ feature fumin@cardinalblue.com
2. PicCollage the product
3. Problem statement The Likes feature Redis SortedSets 2.3M likers 14M collages Some ZCARD > 10K Total 100M+ rows
4. Problem statement 18GB memory exceeded Added another one which was filled up again only 1.5 months later… Ziplist optimizations? Sharding? Key or Range?
5. Problem statement Talked to guys in Redis discussion group, no good solution, twemproxy only for cache, Redis cluster not production ready, neither do service providers such as AWS ElasticCache provide distributed solutions. Cost: USD $1 / 10MB per month
6. Solution requirements Support existing functionality (fortunately, “likes” are simple) Distributed (tired of sharding, migration, config updates) Preferably fully managed (minimize human intervention and errors) Cost competitive
7. Solution AWS DynamoDB for likes AWS Data Pipeline and Elastic MapReduce for migration user_id collage_id created_at 11778407 71641331 1412205880.492 ... ... ...
8. Solution
9. Solution Redis -> S3 backups -> DynamoDB tables and global indices Several Elastic MapReduce jobs and Hive queries running at the same time, whenever dependencies among them allow. Jobs running of clusters of different sizes depending on data size to save costs.
10. Solution Multiple sweeps of reads and writes on 100M+ rows in three different kinds of storage. Sustained 80,000 writes/sec for 1.5 hours. Only limited by AWS account max quota.
11. Solution Heterogeneous cluster composition. Spot instances delivered 88% ($0.28 -> $0.0322) price reduction
12. Solution Heterogeneous language choices
13. Conclusion We moved collage likes from Redis to DynamoDB because Redis is not the most space and cost efficient storage for this case. We used AWS Data Pipeline and AWS EMR to complete a heavy data migration with non-trivial step dependencies, in a fast (parallelized), automated, and cost effective manner.