RiakAtPosterous2notes.pdf

Transcript

1. Riak at Posterous Julio Capote San Francisco Riak Meetup 1/18/2012 Thursday, January 19, 12
2. A/S/L? • Julio Capote • Backend Developer at Posterous • @capotej Thursday, January 19, 12
3. • Allows anyone to create multiple private or public spaces (blogs) • Around since 2008 • Millions of posts and users • Tons of long tail traffic Thursday, January 19, 12 Some of the first posts are still being accessed today due to search engines
4. How we store posts • Original post body goes into MySQL • Multiple variants are generated (nojs, mobile, etc) • Expensive to generate (sanitizers, expanders) Thursday, January 19, 12
5. Enter Variant Cache • A generic read/write-through cache library • Started with Memcache • Moved to Redis Thursday, January 19, 12 At the time disk store looked promising, so we moved from memcache to redis
6. Redis is awesome, but • Requires both the key and value go into memory • Terrible disk store performance • Even with 3 machines with 64gb ram, couldn’t fit entire working set • Forced to set a TTL Thursday, January 19, 12 redis wasn’t really designed to ever hit the disk
7. Thursday, January 19, 12
8. The Dream Thursday, January 19, 12
9. Thursday, January 19, 12
10. What we wanted • Key/Value store • Disk backed • Built in distribution • Use less boxes to serve more users • Consistent performance over raw performance Thursday, January 19, 12
11. Percona MySQL / HandlerSocket Thursday, January 19, 12
12. MySQL / HandlerSocket • Great performance • Can handle a huge number of rows • Mature / Safe (at least the mysql part) The Good Thursday, January 19, 12
13. MySQL / HandlerSocket • Sharding definitely not built in • HandlerSocket is pretty much abandoned The Bad Thursday, January 19, 12 No support going forward
14. Thursday, January 19, 12
15. MongoDB • Crazy fast • Built in sharding support • ...did I mention it was fast? The Good Thursday, January 19, 12
16. Thursday, January 19, 12
17. MongoDB • 30% standard deviation on fetch times (!) • Would falsely acknowledge a write The Bad Thursday, January 19, 12 This is probably tunable, but still
18. Thursday, January 19, 12
19. Riak + Bitcask • Distributed by default • Consistent and predictable performance • Highly concurrent, no perf degradation • Ops guy loves it! The Good Thursday, January 19, 12
20. Riak + Bitcask • Not crazy fast • Stuck it behind memcache • Still way faster than generating • No multi get support The Bad Thursday, January 19, 12 write and read through memcache
21. Riak in production • Started using our 3 node cluster for the global production cache • Accidentally turned off a node • Keys rebalanced, site didn’t skip a beat • No one even noticed till hours later Thursday, January 19, 12
22. Thursday, January 19, 12
23. Stats • 3 nodes • 2600+ requests/second • 300+ GB • ~200 million keys • 10 GB memcache/host Thursday, January 19, 12
24. #Protips • All nodes can serve all requests, so... • Use a vip, or... • Pass all cluster nodes to client driver (thanks @aphyr!) • Use curb instead of net/http • Use Keep Alive Thursday, January 19, 12
25. Any Questions? Thursday, January 19, 12
26. Thanks for listening! Special thanks to @twoism @vincentchu @kangchen @argv0 @pharkmillups @seancribbs @aphyr @jrecursive Thursday, January 19, 12