Patterns for Key-Value stores

Transcript

1. PATTERNS FOR KEY-VALUE STORES Jfokus 2013 Ole-Martin Mørk 06/02/13
2. ABOUT ME Ole-Martin Mørk Scientist Bekk Consulting AS Oslo, Norway twitter: olemartin
3. INTRODUCTION DIFFERENT KV STORES GENERAL PATTERNS RIAK PATTERNS REDIS PATTERNS RIAK+REDIS PATTERNS SUMMARY AGENDA
4. KEY VALUE session_1 Functional Programming You Already Know session_2 Fra enkel J2SE til Grid computing med GigaSpaces XAP session_3 Reinventing software quality session_4 Taming Java Agents session_5 Memory Leaks in Java session_6 Linking data without common identifiers session_7 Avinstaller Java nå! MAP
5. PUT
6. GET
7. DEL
8. KEY VALUE STORE
9. select value from lookup where key=?
10. HVORFOR BENYTTE KEY-VALUE STORES? 1 0 Speed
11. Distribution
12. HVORFOR BENYTTE KEY-VALUE STORES? 1 2 When all you have is a hammer..
13. PERSON OG ADRESSER Person Address Phone Email Twitter Facebook
14. name: “Ole-Martin Mørk”, emails: email: ”me@ole - martin.net” type: “work”, email: “ole-martin.mork@bekk.no” phone: type: “mobile”, phone: “+4790252338” addresses: type: “home” adresse: street: “Lundliveien 30b”, town: “Oslo” type: “work” address: town: “Oslo”, zipcode: “0150”
15. {name: “Ole-Martin Mørk”, emails: [ { email: ”me@ole - martin.net” }, { type: “work”, email: “ole-martin.mork@bekk.no” }], phone: [{ type: “mobile”, phone: “+4790252338” }], addresses: [{ type: “home”, address: { street: “Lundliveien 30b”, town: “Oslo” } }, { type: “work”, address: { town: “Oslo”, zipcode: “0150” } }]}
16. {name: “Ole-Martin Mørk”, emails: [ { email: ”me@ole - martin.net” }, { type: “work”, email: “ole-martin.mork@bekk.no” }], phone: [{ type: “mobile”, phone: “+4790252338” }], addresses: [{ type: “home”, address: { street: “Lundliveien 30b”, town: “Oslo” } }, { type: “work”, address: { town: “Oslo”, zipcode: “0150” } }] }
18. HTML5 LOCAL STORAGE Demo Klikk!
20. CASSANDRA 2 0
21. DETTE ER FINE DATABASER 2 1 Why?
22. PATTERNS
23. Keys
24. WELL-FORMED KEYS app:space:what:id
25. LOOKUPS ON OTHER DATA userId 1234 username olemartin email olemartin@bekk.no twitter olemartin user:id:1234 user:twitter:olemartin user:id:1234
26. RELATIONS userId 1234 username olemartin email olemartin@bekk.no twitter olemartin user:id:1234 user:id:4321 userId 4321 username grolisbeth brother_rel user:id:1234
27. Hash
28. HASHING <hash> modulo <num servers> = serverId
29. 1 2 3 4 3 4 3 2 HASHING 1 1 2
30. CONSISTENT HASHING A B C
31. CONSISTENT HASHING A B C
32. CONSISTENT HASHING A B C D A B C A B C D
33. Use cases
34. EXAMPLES ON APPLICATIONS SUITED FOR K/V RDBMS + KEY/VALUE
38. RIAK Bucket Bucket Bucket Key Key Key Key
39. vailability artition C A P onsistency
40. N W R
41. N > W
42. N > R
44. PATTERNS
45. RIAK & MAP REDUCE
46. MAP REDUCE shopping:13092012:userId
47. SECONDARY INDEXES 4 7 Secondary indexes
48. SECONDARY INDEXES 4 8 userId 1234 username olemartin email olemartin@bekk.no twitter olemartin user:id:1234 user:twitter:olemartin user:id:1234 twitter:id_bin olemartin email:id_bin olemartin@bekk.no user:sex_bin Male user:age_int 35
49. Pre Commit Post Commit
51. Hash Lists Sets Sorted Sets Strings
52. PATTERNS
53. SELECT * FROM foo WHERE ... ORDER BY time DESC LIMIT 10
54. PATTERNS FOR MILJØER MED RDBMS RDBMS REDIS
56. STATISTIKK Number of pages read by user <pages:userId, count> incr pages:userId Most read today: <most_read:300-2012, <url, count>,<url, count>> zrangebyscore most_read:300-2012 -inf +inf Unique visitors: <page:url:300-2012, <userId>, <userId>, <userId>> scard page:url:300-2012
57. SERIALIZATION 0 ms 10000 ms 20000 ms 30000 ms 40000 ms 50000 ms 60000 ms 70000 ms 80000 ms 90000 ms 100000 ms
58. KRYO IN REDIS <bean id="redisTemplate" class="org.springframework.data.redis.core.RedisTemplate"> <property name="connectionFactory" ref="connectionFactory"/> <property name="defaultSerializer" ref="serializer"/> </bean> <bean id="serializer" class="no.bekk.redis.DiggerSerializer"/>
59. KRYO public class DiggerSerializer<T> implements RedisSerializer<T> { private Kryo kryo = new Kryo(); public DiggerSerializer() { kryo.register(Story.class); } public byte[] serialize(T t) { return new ObjectBuffer(kryo).writeClassAndObject(t); } public T deserialize(byte[] bytes) { return (T) new ObjectBuffer(kryo).readClassAndObject(bytes); } }
60. REDDIT KLONE 1.  story:id – holder styr på id-en til en story 2.  story:id, value – holder selve story-objektet 3.  story:stories_zset – holder styr på antall stemmer 4.  story:tag:storyId – holder styr på tags knyttet til en story 5.  tag:tagId – holder styr på stories knyttet til en tag
61. REDIS OG RIAK + =
63. Produktkatalog Product catalog
64. RIAK Products product:2 product:1
65. POST COMMIT Products Redis RSS
66. REDIS catalog:product:id binary data catalog:product:today:sold_zset <productId, counter> catalog:product:week:sold_zset <productId, counter> catalog:product:year:sold_zset <productId, counter>
67. RESULT
68. Summary
69. SUMMARY 1.  Lots of different KV-stores are available 2.  Good key names 3.  Make sure you model your data correctly 4.  Consistent hashin 5.  Use KV-stores with RDBMS 6.  Use KV-stores with data that match 7.  Use the datastore where it excels 8.  Combine different KV-stores 9.  Be creative!
70. 7 0 Questions?
71. THANK YOU! Ole-Martin Mørk ole-martin.mork@bekk.no @olemartin