Scaling Your Data Safely with Riak

CodeMash 2014 Presentation.

Transcript

1. with Riak @alexmoore Scaling
 Data Safely

2. Internet 1 (2) (3)
3. Internet 503 Timeout 1 (2) (3)
4. Internet 1 (2) 3
5. Internet 1 (2) 3
6. Internet (2) 3 2 3 1
7. Internet 4 2 3 (2) 1 3 3
8. Internet CDN 2 6 7 2 4 3 1 3 3
9. Internet CDN 5 6 2 4 3 2 1 3 3 7 3
10. Internet CDN 6 7 2 3 2 1 3 3
11. Internet CDN P S S 3 6 7 2 5 3 2 1 4 4 3 3 7
12. Internet CDN 6 8 2 3 2 1 3 3
13. Internet CDN 6 8 2 3 2 1 3 3 3? 4? q User Session Widgets Lolcats Stuff 8? magic
14. Internet CDN 6 8 2 3 2 1 3 3 3? 4? q User Session Widgets Lolcats Stuff 8? magic
15. Internet CDN 6 8 2 3 2 1 3 3 3? 4? q User Session Widgets Lolcats Stuff 8? magic
16. Internet CDN 6 8 2 3 2 1 3 3 3? 4? q User Session Widgets Lolcats Stuff 8? magic Who’s on first?
17. Internet CDN 6 8 2 3 2 1 3 3 3? 4? q User Session Widgets Lolcats Stuff 8? magic What the hell have you built? Who’s on first?
18. Enter Riak
19. Riak is a distributed, fault-tolerant, highly available database
21. Distributed
22. Distributed a cluster of nodes each node has an identical role
23. Fault- tolerant
24. Fault- Tolerant symmetrical no Single Point of Failure
25. Highly Available
26. Highly Available replicas & redundancy
27. Scalable
28. Scalable add or remove nodes linear scalability
29. Scalable add or remove nodes linear scalability
30. Multiple Platforms
31. Multiple Platforms
32. Multiple Platforms
33. Predictable Performance
35. Internet CDN 5 6 2 3 2 1 3 3
36. Internet CDN 5 6 2 3 2 1 3 3 3 3 4 4
37. 1   2   3   4   5   6   7   8   9   10   11   12 Building A Cluster #  All  Nodes   $  riak  start   ! #  Nodes  2..n.each     $  riak-­‐admin  join  riak@node1.internal.company.com   ! #  Any  Node   $  riak-­‐admin  cluster  plan   $  riak-­‐admin  cluster  commit   !
40. Architecture
41. Riak is the ops-friendly database
42. Riak is distributed
43. Consistent Hashing hash(“bucket/key”)
44. hash ring
45. tokenize it
46. node 0 node 1 node 2
47. node 0 node 1 node 2 hash(key)
48. node 0 node 1 node 2 Replicas are stored to the N - 1 contiguous partitions
49. node 0 node 1 node 2 hash(“companies/GE”) Replicas are stored to the N - 1 contiguous partitions
50. node 0 node 1 node 2 hash(“companies/GE”) Replicas are stored to the N - 1 contiguous partitions
51. node 0 node 1 node 2
52. node 0 node 1 node 2 node 3 + Scaling out
53. Quorum requests N R W PR/PW DW
54. get(“users/clay-davis”)
55. get(“users/clay-davis”) client Riak
56. get(“users/clay-davis”) Get Handler (FSM) client Riak
57. get(“users/clay-davis”) Get Handler (FSM) client Riak hash(“users/clay-davis”) == 10, 11, 12
58. get(“users/clay-davis”) Get Handler (FSM) client Riak hash(“users/clay-davis”) == 10, 11, 12 Coordinating node Cluster 6 7 8 9 10 11 12 13 14 15 16 The Ring
59. get(“users/clay-davis”) Get Handler (FSM) client Riak get(“users/clay-davis”) Coordinating node Cluster 6 7 8 9 10 11 12 13 14 15 16 The Ring
60. get(“users/clay-davis”) Get Handler (FSM) client Riak Coordinating node Cluster 6 7 8 9 10 11 12 13 14 15 16 The Ring R=2
61. get(“users/clay-davis”) Get Handler (FSM) client Riak Coordinating node Cluster 6 7 8 9 10 11 12 13 14 15 16 The Ring R=2 obj
62. get(“users/clay-davis”) Get Handler (FSM) client Riak R=2 obj obj
63. get(“users/clay-davis”) Get Handler (FSM) client Riak R=2 obj obj
64. get(“users/clay-davis”) obj
65. Read Repair (Anti-Entropy)
66. replica replica replica
67. replica replica replica X
68. replica replica replica replica replica replica
69. Active Anti-Entropy (self healing clusters)
70. real-time updates persistent non-blocking disk-based
71. merkle tree to track changes coordinated at the vnode level runs as a background process exchange with neighbor vnodes for inconsistencies resolution semantics: trigger read-repair
72. = hashes marked “dirty”
77. = keys to read-repair
78. Using Riak
79. Riak is the ops-friendly database
80. Riak is friendly to Developers, too!
81. {“key”: “value”}
82. key value key value key value key value key value key value key value key value key value Keys are namespaced into Buckets
83. Buckets aren’t “real”
84. ✘ ✘ ✘ Buckets aren’t “real”
85. Metadata Bucket Properties
86. Aitch Tee Tee Pee GET /buckets/bucket/keys/key PUT /buckets/bucket/keys/key DEL /buckets/bucket/keys/key
87. 1   2   3   4   5   6   7   8   9   10   11   12 Working With Ruby - Starting $  gem  install  riak-­‐client   ! book  =  {          isbn:  '1111979723',          title:  'Moby  Dick',          author:  'Herman  Melville',          body:  'Call  me  Ishmael.  Some  years  ago...',          copies_owned:  3   }  
88. 1   2   3   4   5   6   7   8   9   10   11   12 Working With Ruby - Create #  Starting  Client   client  =  Riak::Client.new(protocol:   'pbc',                           pb_port:     8087,                         host:       '10.0.0.1')   ! books_bucket  =  client.bucket('books')     ! robj_moby  =  books_bucket.new(book[:isbn])   robj_moby.data  =  book   robj_moby.store   ! !
89. 1   2   3   4   5   6   7   8   9   10   11   12 Working With Ruby - Read fetched_book  =  books_bucket.get('1111979723')   ! puts  fetched_book.raw_data   !  =>  {"isbn":"1111979723","title":"Moby  Dick”,   "author":"Herman  Melville",   "body":"Call  me  Ishmael.  Some  years   ago...","copies_owned":3}   ! fetched_book.data['isbn']   !  =>  "1111979723"
90. 1   2   3   4   5   6   7   8   9   10   11   12 Working With Ruby - Delete fetched_book.delete   ! #  or   ! books_bucket.delete('1111979723')  
91. 1   2   3   4   5   6   7   8   9   10   11   12 Working With Ruby - 2I ! fetched_book.indexes['author_bin']  =  []   fetched_book.indexes['author_bin']  <<               fetched_book.data['author']   ! fetched_book.store   ! books_bucket.get_index('author_bin','Herman   Melville')   ! =>  ["1111979723"]
92. etc...
93. JUST ONE MORE THING
94. Internet CDN 5 6 2 3 2 1 3 3 3 3 4 4
95. Internet CDN 5 6 2 3 2 1 3 3 3 4
96. Internet CDN 5 6 2 3 2 1 3 3 3 4
97. in riak.conf:
98. search = off in riak.conf:
99. search = off search = on in riak.conf:
100. 1   2   3   4   5   6   7   8   9   10   11   12 client.create_search_index(“tweets_idx”)   ! bucket  =  Riak::Bucket.new(client,  "tweets")   bucket.props  =  {"search_index"  =>  "tweets_idx"}   ! #  load  some  tweets  into  riak…   ! results  =  client.search("tweets",  "text:who")   ! puts  results[‘docs'].first  
101. 1   2   3   4   5   6   7   8   9   10   11   12 pp  results[‘docs'].first   ! {"score"=>"2.50804900000000019489e+00",    "_yz_rb"=>"tweets",    "_yz_rk"=>"421672897250226176",    ...}
102. More resources: docs.basho.com “Taste of Riak”
103. Multi-Datacenter Replication (E.G. - Nuke proofing)
104. Full Sync
105. Real-Time
107. put(“users/jimmy”)
108. put(“users/jimmy”)
109. put(“users/jimmy”)
110. put(“users/jimmy”)
111. put(“users/jimmy”)
112. Internet CDN 5 6 2 3 2 1 3 3 3 4
113. Internet CDN 5 6 2 3 2 1 3 3 3 4 ?
114. Internet CDN 5 6 2 3 2 1 3 3 3 4 ?
115. Q & A Alex Moore - Client Services Engineer, Basho
 @alexmoore
 amoore@basho.com
 http://alexmoore.io