Riak

Transcript

1. Jeremy Thurgood July 4, 2012

2. What is Riak? Distributed, highly-available, key/value store. Open source, but

   commercial ”Enterprise” version available. Strong community, but commercial support available. basho

3. Under the hood • Keyspace is partioned (consistent hashing) •

   Physical nodes run multiple vnodes • Each vnode claims a partition • Replicas are stored in multiple partitions

4. CAP, replication, all that good stuff • N replicas are

   stored • R replicas are required for a successful read • W replicas are required for a successful write • R, W and DW can specified per-request • Version conflicts are detected with vclocks and may be resolved by the application • But! By default, this is done for you using “latest wins”

5. Pluggable backends Bitcask • High throughput, low latency, predictable performance

   • Keys must fit in memory, no secondary indexes LevelDB • Data compression, secondary indexes • Slower than Bitcask Memory • No persistence, good for testing Multi • Configure backend per bucket

6. APIs • HTTP (RESTful), Protocol Buffers, Erlang • GET, PUT,

   DELETE objects • Do various things with buckets • Map/reduce, secondary indexes • Search

7. PUT object • Bucket name • Key name (optional) •

   Content-Type • Vclock (if the object already exists) • Links, indexes, metadata (if any) • Quorum size: W , DW • Content Content can be arbitrary data, but something like JSON is best for search, etc.

8. GET object • Bucket name, key name • Quorum size:

   R Response • Content-Type • ETag, Last-Modified • Vclock • Links, indexes, metadata (if any) • Content (of course)

9. Buckets • Not really real, just namespaces • Properties •

   n val, allow mult, postcommit, etc. • Warning! Don’t have too many • Can’t delete (but I’m working on that) • “List keys” supported, but very expensive

10. Map/reduce • Multiple steps: map, reduce, link • Inputs: bucket/key

    pairs, index query, key filter • Javascript or Erlang functions • See docs for all the details

11. Secondary indexes • Requires eleveldb backend • Data types: bin,

    int • Orthogonal to object value • Query on exact value or range • Query on one index only • Returns keys, not objects • Can feed map/reduce operations

12. Search • Lucene (with some bits ported to Erlang) •

    Index distributed by term (high performance, possible latency hit) • Solr API • Search key/value data • Set precommit hook for indexer • JSON and XML are indexed by field name • Can feed map/reduce operations

13. Client libraries • Lots, for all sorts of exciting languages

    • Basho officially supports C/C++, Erlang, Java, PHP, Python, Ruby • We maintain Riakasaurus (Python/Twisted) • We wrote a hybrid sync/async ORM-sort-of-thing which needs a cool name

14. Operational stuff • Recommend minimum of 5 nodes, but 3

    will do • All nodes are equal (no master) • Nodes can go away with no downtime (as long as there’s still a quorum) • Adding nodes is completely transparent • For best performance, use dedicated boxes with good I/O

15. Making the cluster dance • riak start • riak stop

    • riak-admin join [email protected] • riak-admin status (lots of stuff) • riak-admin diag (needs riaknostic installation)

16. Web console

17. Web console

18. So, why are we using it? We need a db

    that. . . • has high performance reads and writes • doesn’t require a devops army • can store relations (in a limited way) • works well with non-relational data • allows full-text searching

19. Gotchas • Bucket properties are global metadata, which impacts performance

    • Eventual consistency can be a pain at times

20. And that’s it Thanks! Also, come to PyCon ZA. http://za.pycon.org