Riak

Riak presentation at codebridge

Transcript

1. Jeremy Thurgood July 4, 2012
2. What is Riak? Distributed, highly-available, key/value store. Open source, but commercial ”Enterprise” version available. Strong community, but commercial support available. basho
3. Under the hood • Keyspace is partioned (consistent hashing) • Physical nodes run multiple vnodes • Each vnode claims a partition • Replicas are stored in multiple partitions
4. CAP, replication, all that good stuff • N replicas are stored • R replicas are required for a successful read • W replicas are required for a successful write • R, W and DW can specified per-request • Version conflicts are detected with vclocks and may be resolved by the application • But! By default, this is done for you using “latest wins”
5. Pluggable backends Bitcask • High throughput, low latency, predictable performance • Keys must fit in memory, no secondary indexes LevelDB • Data compression, secondary indexes • Slower than Bitcask Memory • No persistence, good for testing Multi • Configure backend per bucket
6. APIs • HTTP (RESTful), Protocol Buffers, Erlang • GET, PUT, DELETE objects • Do various things with buckets • Map/reduce, secondary indexes • Search
7. PUT object • Bucket name • Key name (optional) • Content-Type • Vclock (if the object already exists) • Links, indexes, metadata (if any) • Quorum size: W , DW • Content Content can be arbitrary data, but something like JSON is best for search, etc.
8. GET object • Bucket name, key name • Quorum size: R Response • Content-Type • ETag, Last-Modified • Vclock • Links, indexes, metadata (if any) • Content (of course)
9. Buckets • Not really real, just namespaces • Properties • n val, allow mult, postcommit, etc. • Warning! Don’t have too many • Can’t delete (but I’m working on that) • “List keys” supported, but very expensive
10. Map/reduce • Multiple steps: map, reduce, link • Inputs: bucket/key pairs, index query, key filter • Javascript or Erlang functions • See docs for all the details
11. Secondary indexes • Requires eleveldb backend • Data types: bin, int • Orthogonal to object value • Query on exact value or range • Query on one index only • Returns keys, not objects • Can feed map/reduce operations
12. Search • Lucene (with some bits ported to Erlang) • Index distributed by term (high performance, possible latency hit) • Solr API • Search key/value data • Set precommit hook for indexer • JSON and XML are indexed by field name • Can feed map/reduce operations
13. Client libraries • Lots, for all sorts of exciting languages • Basho officially supports C/C++, Erlang, Java, PHP, Python, Ruby • We maintain Riakasaurus (Python/Twisted) • We wrote a hybrid sync/async ORM-sort-of-thing which needs a cool name
14. Operational stuff • Recommend minimum of 5 nodes, but 3 will do • All nodes are equal (no master) • Nodes can go away with no downtime (as long as there’s still a quorum) • Adding nodes is completely transparent • For best performance, use dedicated boxes with good I/O
15. Making the cluster dance • riak start • riak stop • riak-admin join riak@192.168.1.10 • riak-admin status (lots of stuff) • riak-admin diag (needs riaknostic installation)
16. Web console
17. Web console
18. So, why are we using it? We need a db that. . . • has high performance reads and writes • doesn’t require a devops army • can store relations (in a limited way) • works well with non-relational data • allows full-text searching
19. Gotchas • Bucket properties are global metadata, which impacts performance • Eventual consistency can be a pain at times
20. And that’s it Thanks! Also, come to PyCon ZA. http://za.pycon.org