Redis on AWS

Presentation held in the seminar on "NotOnlySQL databases." Builds upon slides in the former presentation "Redis — memcached on steroids." Features challenges when deploying Redis on Amazon Web Services within the context of the Yahoo! Cloud Serving Benchmark.

Transcript

1. “Redis is more than just a key-value store, it’s a lifestyle. –– Mathias Meyer
2. a data structure server hash tables sets sorted sets Fundamental data structures for a fundamental API lists
3. user1234567890 field0: abc… field1: def… field2: ghi… field3: jkl… field4: mno… field5: pqr… field6: stu… field7: vwx… field8: yzä… field9: öüß…
4. hash tables user1234567890 field0: abc… field1: def… field2: ghi… field3: jkl… field4: mno… field5: pqr… field6: stu… field7: vwx… field8: yzä… field9: öüß… insert read update delete HMSET HMGET HMSET DEL RedisClient.java
5. a data structure server hash tables sets sorted sets Fundamental data structures for a fundamental API lists
6. factory settings hand-tuned hash-max-zipmap-value 2048 2.54MB 1.62MB 0.5M initial size
8. The Harry Potter Approach WWVD What Would Voldemort Do 101 @Override 102 public int scan(…) { 104 logger.warn( ⁞ "Voldemort does not support Scan semantics"); 105 return OK; 106 } db/voldemort/src/com/yahoo/ycsb/db/VoldemortClient.java
9. sorted sets
10. user1 user2 user3 user5 user8 1 2 3 5 8…
11. user1 user2 user3 user5 user8 not that easy
12. user1 user2 user3 user5 user8 not that easy record name can be anything thus: position = hash(name)
13. sorted sets 50% performance hit
15. workload A throughput target
16. workload E throughput target
17. workload F throughput target
18. “I think the… problem is… believing that there can be “one true datastore”. Different technologies excel at different things. –– Weixi Yen
19. “I think the… problem is… believing that there can be “one true datastore”. Different technologies excel at different things. –– Weixi Yen
20. Memory storage is #1 Redis key space http://spotfireblog.tibco.com/wp-content/uploads/in-memory-analytics.jpg
21. bandwidth contention network memory
22. 200 ops/sec single-threaded
23. user1234567890 field0: abc… field1: def… field2: ghi… field3: jkl… field4: mno… field5: pqr… field6: stu… field7: vwx… field8: yzä… field9: öüß… = 1 KB
24. user1234567890 field0: abc… field1: def… field2: ghi… field3: jkl… field4: mno… field5: pqr… field6: stu… field7: vwx… field8: yzä… field9: öüß… = 1 KB factory page size := 32 bytes
25. user1234567890 field0: abc… field1: def… field2: ghi… field3: jkl… field4: mno… field5: pqr… field6: stu… field7: vwx… field8: yzä… field9: öüß… = 1 KB factory page size := 32 bytes values must be swapped in their entirety, even for complex types
26. user1234567890 field0: abc… field1: def… field2: ghi… field3: jkl… field4: mno… field5: pqr… field6: stu… field7: vwx… field8: yzä… field9: öüß… = 1 KB factory page size := 32 bytes values must be swapped in their entirety, even for complex types = 32 pages 
27. user1234567890 field0: abc… field1: def… field2: ghi… field3: jkl… field4: mno… field5: pqr… field6: stu… field7: vwx… field8: yzä… field9: öüß… = 1 KB factory page size := 32 bytes values must be swapped in their entirety, even for complex types = 32 pages  vm-page-size et al
28. user1234567890 field0: abc… field1: def… field2: ghi… field3: jkl… field4: mno… field5: pqr… field6: stu… field7: vwx… field8: yzä… field9: öüß… = 1 KB = 32 pages  50% performance hit
29. 32-bit 64-bit 4 GB 8 TB large pointers
30. redis-cli INFO
31. Memory storage is #1 meh, persistence http://www.flickr.com/photos/generationbass/4827013488/ memory overhead
32. Memory storage is #1 meh, persistence http://www.flickr.com/photos/generationbass/4827013488/ memory overhead load phase fails randomly
33. there are no silver bullets