Riak Overview

Overview Mårten Gustafson Stockholm Riak Meetup #1 2011-03-31 1 ->
Don’t forget to start the demo instance 2 -> Reset Chrome and move to proper space

What is Riak and what’s the agenda? Decentralized key-value store
A database ideally suited for web applications A ﬂexible map/reduce engine

http://www.ﬂickr.com/photos/linneberg/4481309196/ Buckets Buckets contain things Things are keys, that have
values A bucket is like a namespace (and it’s cheap - could even be free)

Operations Riak Key/Value Store Bucket Bucket Bucket Item Item Item
Item Item Item Item Item HTTP Client PUT /bucket/key GET /bucket/key DELETE /bucket/key

An entry • lives in a bucket • has a
key • has a value

key • has a value bucket arbitrary name

key • has a value bucket key arbitrary name arbitrary name

key • has a value bucket key arbitrary name arbitrary name forms the path to the value

key • has a value bucket key value arbitrary name arbitrary name a binary blob and mime type forms the path to the value

key • has a value bucket key value arbitrary name arbitrary name forms the path to the value a binary blob and mime type = Store anything, yay!

Store anything bucket meetup

Store anything <html><body> <h1> <a href=”bar.html”>foo</a> </h1> </body></html> key foo.html
mime text/html bucket meetup

Store anything <html><body> <h1> <a href=”bar.html”>foo</a> </h1> </body></html> <html><body> <h1>bar!</h1>
</body></html> key foo.html mime text/html key bar.html mime text/html bucket meetup

Store anything http://127.0.0.1:8088/riak/meetup/foo.html <html><body> <h1> <a href=”bar.html”>foo</a> </h1> </body></html> <html><body>
<h1>bar!</h1> </body></html> key foo.html mime text/html key bar.html mime text/html bucket meetup

A database ideally suited for web applications A ﬂexible map/reduce engine Database -> Persistence -> Durability -> Storage backends

Storage Riak Key/Value Store Pluggable back-ends, per bucket Bitcask keeps
the key-set in memory (ie, in some (rare) cases it might not ﬁt, then look to InnoDB) ETS - Built in erlang storage, DETS is ETS on disk

Storage Riak Key/Value Store Bitcask InnoDB DETS ETS Balanced trees
File system LRU Pluggable back-ends, per bucket Bitcask keeps the key-set in memory (ie, in some (rare) cases it might not ﬁt, then look to InnoDB) ETS - Built in erlang storage, DETS is ETS on disk

File system LRU ram based, not durable disk based, durable Pluggable back-ends, per bucket Bitcask keeps the key-set in memory (ie, in some (rare) cases it might not ﬁt, then look to InnoDB) ETS - Built in erlang storage, DETS is ETS on disk

File system LRU ram based, not durable disk based, durable default common Pluggable back-ends, per bucket Bitcask keeps the key-set in memory (ie, in some (rare) cases it might not ﬁt, then look to InnoDB) ETS - Built in erlang storage, DETS is ETS on disk

A database ideally suited for web applications A ﬂexible map/reduce engine Pluggable back-ends, per bucket

A database ideally suited for web applications A ﬂexible map/reduce engine Decentralized -> Cluster - No master node - No single point of failure

The Ring The “storage space” is partitioned into a ring
Keys and their values has a primary partition

The Ring A ring size of 1024 should accommodate most
needs Once you’ve set your ring size, it’s ﬁxed Only way to change is to backup/restore your entire cluster

The Ring ring size = 12 1 2 3 4
5 6 7 8 9 10 11 12 A ring size of 1024 should accommodate most needs Once you’ve set your ring size, it’s ﬁxed Only way to change is to backup/restore your entire cluster

Consistent Hashing 1 2 3 4 5 6 7 8
9 10 11 12 Always maps key “x” to partition “y”

Read “I want “ requires us to know: where is
on the ring?

Cluster One Ring size to rule them all, One Ring
size to ﬁnd them, One Ring size to bring them all and in the cluster bind them...

Cluster node A node B node C ring size =
12 instances = 3 ring size / nodes = ~slices per instances

Cluster - Read node A node B node C

Cluster - Read node A node B node C I
can haz ? Hm, hashes to a slice of the ring owned by node C.

Cluster - Read node A node B node C Okidoki,
now where’s he...a yeah in my fourth slice I can haz ? Hey C! I need

Cluster - Read node A node B node C I
can haz ?

So what about...

...network partitions? node A node B node C

...network partitions? node A node B node C X

...failed nodes? node A node B node C

...failed nodes? node A node B

...concurrent writes? node A node B node C client 1
client 2

...concurrent writes? node A node B node C client 1
client 2 hey, A! save for me ? hey, C! save for me

Livin’ on the web • concurrent users • unreliable networks
• node failures

The N to the R to the W to the
DW and the RW Buckets have defaults for R, W, DW and RW

Node A Node B Node C N_VAL Bucket Client Number
of replicas, set per bucket

Node A Node B Node C N_VAL Bucket 1 Client
Number of replicas, set per bucket

Node A Node B Node C R Bucket GET /bucket/key?R=1
Client R = Read Will pose a request to all nodes, answer to client when R clients have agreed

Node A Node B Node C R Bucket Client GET
/bucket/key?R=2 R = Read Will pose a request to all nodes, answer to client when R clients have agreed

Node A Node B Node C R Bucket Agree? Client
GET /bucket/key?R=2 R = Read Will pose a request to all nodes, answer to client when R clients have agreed

Node A Node B Node C R Bucket Client GET
/bucket/key?R=2 R = Read Will pose a request to all nodes, answer to client when R clients have agreed

DW and the RW Buckets have defaults for R, W, DW and RW

DW and the RW Number of copies ie. distribute to N nodes Buckets have defaults for R, W, DW and RW

DW and the RW Number of copies ie. distribute to N nodes Read ie. have R nodes agree Buckets have defaults for R, W, DW and RW

DW and the RW Number of copies ie. distribute to N nodes Read ie. have R nodes agree Write ie. ack’d by W nodes Buckets have defaults for R, W, DW and RW

DW and the RW Durable write ie. persistently written by DW nodes Number of copies ie. distribute to N nodes Read ie. have R nodes agree Write ie. ack’d by W nodes Buckets have defaults for R, W, DW and RW

DW and the RW Read-write ie. persistently deleted by RW nodes Durable write ie. persistently written by DW nodes Number of copies ie. distribute to N nodes Read ie. have R nodes agree Write ie. ack’d by W nodes Buckets have defaults for R, W, DW and RW

The Quorum ([node count] / 2) + 1 quorum =
majority quorum = valid value for r, w, dw, rw

Bucket properties http://127.0.0.1:8088/riak/meetup/ Buckets have default values Clients can override
per request

Conﬂicts

When worlds collide Conﬂicts

I don’t care! allow_mult = false last_write_wins = true Bucket
properties allow_mult = false -> siblings never returned last_write_wins = true

I do care! allow_mult = true

I do care! • Resolve conﬂicts in application logic •
Conﬂicts exposed as siblings beneath a key • Response is HTTP 300 Multiple Choice • Served as mime/multipart

Example HTTP/1.1 300 Multiple Choices X-Riak-Vclock: a85hYGBgzWDKBVIsrLnh3BlMiYx5rAymfeeO8EGFWRLl30G== Content-Type: multipart/mixed; boundary=ZZ3eyjUllBi7GXRRMJsUublFxjn
Content-Length: 368 --ZZ3eyjUllBi7GXRRMJsUublFxjn Content-Type: text/plain Tuesday --ZZ3eyjUllBi7GXRRMJsUublFxjn Content-Type: text/plain Thursday --ZZ3eyjUllBi7GXRRMJsUublFxjn--

Map / Reduce • Javascript or Erlang • Exposed in
the HTTP API

Map / Reduce count words function(v) { var words =
v.values[0].data.toLowerCase().match('\w*','g'); var counts = []; for(var word in words) if (words[word] != '') { var count = {}; count[words[word]] = 1; counts.push(count); } return counts; }

Map / Reduce count words function(values) { var result =
{}; for (var value in values) { for(var word in values[value]) { if (word in result) result[word] += values[value][word]; else result[word] = values[value][word]; } } return [result]; }

Map & Reduce count words {"inputs":"bucket", "query":[{"map":{"language":"javascript", "source":"function(v) { var
words = v.values[0].data.toLowerCase().match(/ \w*/g); var counts = []; for(var word in words) if (words[word] != '') { var count = {}; count[words[word]] = 1; counts.push(count); } return counts; }"}},{"reduce":{"language":"javascript", "source":"function(values) { var result = {}; for (var value in values) { for(var word in values [value]) { if (word in result) result[word] += values[value][word]; else result[word] = values[value][word]; } } return [result]; }"}}]} Put this in your POST request and let Riak smoke it

words = v.values[0].data.toLowerCase().match(/ \w*/g); var counts = []; for(var word in words) if (words[word] != '') { var count = {}; count[words[word]] = 1; counts.push(count); } return counts; }"}},{"reduce":{"language":"javascript", "source":"function(values) { var result = {}; for (var value in values) { for(var word in values [value]) { if (word in result) result[word] += values[value][word]; else result[word] = values[value][word]; } } return [result]; }"}}]} function(v) { var words = v.values[0].data.toLowerCase().match('\w*','g'); var counts = []; for(var word in words) if (words[word] != '') { var count = {}; count[words[word]] = 1; counts.push(count); } return counts; }

words = v.values[0].data.toLowerCase().match(/ \w*/g); var counts = []; for(var word in words) if (words[word] != '') { var count = {}; count[words[word]] = 1; counts.push(count); } return counts; }"}},{"reduce":{"language":"javascript", "source":"function(values) { var result = {}; for (var value in values) { for(var word in values [value]) { if (word in result) result[word] += values[value][word]; else result[word] = values[value][word]; } } return [result]; }"}}]} function(values) { var result = {}; for (var value in values) { for(var word in values[value]) { if (word in result) result[word] += values[value][word]; else result[word] = values[value][word]; } } return [result]; }

Map / Reduce Demo

Links • Non-enforced • Traversable Link walking uses M/R behind
the scenes

Links Demo Link walking uses M/R behind the scenes

The whole enchilada Erlang / OTP Riak Key/Value Store Riak
Core Riak Search HTTP API Luwak Partitioning (consistent hashing, hinted handoff) Membership management leave/join Work distribution Cluster state gossip protocol Bitcask InnoDB DETS ETS Balanced trees File system LRU

Try it downloads.basho.com brew install riak Web admin @ github.com/gmaurice/Riaktive

Resources Riak Fast Track @ wiki.basho.com #riak @ freenode github.com/basho/

Thanks for listening Mårten Gustafson @martengustafson http://marten.gustafson.pp.se/ [email protected]

Riak Overview

Riak Overview

More Decks by Mårten Gustafson

Other Decks in Technology

Featured

Transcript