Using Riak as a Storage Service for Datomic

This deck explores the architecture of Datomic and discusses why Riak is an attractive storage service for it. It also focuses on specifics around enabling Riak as a storage service for Datomic.

A Vagrant project to spin up an instance of Datomic backed by Riak can be found here: https://github.com/hectcastro/vagrant-datomic-riak

Transcript

1. Riak ♥ Datomic Using Riak as a Storage Service for Datomic
2. Hector Castro
   @hectcastro
6. Transaction coordination Write data to log (unordered) Indexing (background) {
7. Think of writes against Datomic the way you think about writes into a RDBMS.
9. Live index Cache (actual data segments) Query engine (flavor of Datalog) {
10. Queries don’t go through transactor.
12. Data log Indexes {
13. Entity Attribute Value Time {
14. {:entity  User,    :attribute  "name",    :value  "Hector",    :time  (System/currentTimeMillis)}
15. User,name,Hector,1389288086560   User,name,Batman,1389288086561   User,name,Joker,1389288086562
16. name,User,Hector,1389288086560   name,User,Batman,1389288086561   name,User,Joker,1389288086562
17. AEVT and VAET (optional)
19. Distributed, incrementally scalable, masterless, highly- available key/value store.
20. Distributed, incrementally scalable, masterless, highly- available key/value store.
21. Distributed, incrementally scalable, masterless, highly- available key/value store.
22. Distributed, incrementally scalable, masterless, highly- available key/value store.
23. Distributed, incrementally scalable, masterless, highly- available key/value store.
24. Distributed, incrementally scalable, masterless, highly- available key/value store.
26. Data log and index root references require CAS.
27. Index Reference Log Reference
29. Zookeeper Riak Index Reference Log Reference
30. Strong Consistency Available on a per-bucket basis in Riak 2.0.
31. N=3   DW=2   R=1   not_found_ok=false {
33. Installation Steps
34. Peer Library
35. $  unzip  datomic-­‐pro-­‐0.9.4360.zip   $  cd  datomic-­‐pro-­‐0.9.4360   $  ./bin/maven-­‐install
36. <dependency>      <groupId>com.datomic</groupId>      <artifactId>datomic-­‐pro</artifactId>      <version>0.9.4360</version>   </dependency>
37. ;  Add  to  collection  under  :dependencies   [com.datomic/datomic-­‐pro  "4699"]
38. Transactor and Storage
39. Transactor service (2) Zookeeper cluster (3-5) Riak cluster (5+) {
40. Transactor service (2) Zookeeper cluster (3-5) Riak cluster (5+) {
41. $  ./bin/transactor  ./config/transactor.properties
42. Transactor service (2) Zookeeper cluster (3-5) Riak cluster (5+) {
43. $  ./bin/zkServer.sh  start-­‐foreground  ./conf/zoo.cfg
44. Transactor service (2) Zookeeper cluster (3-5) Riak cluster (5+) {
45. $  riak-­‐admin  cluster  join  riak@33.33.33.20   $  riak-­‐admin  cluster  join  riak@33.33.33.30   $  riak-­‐admin  cluster  join  riak@33.33.33.40   $  riak-­‐admin  cluster  join  riak@33.33.33.50   $  riak-­‐admin  cluster  plan   $  riak-­‐admin  cluster  commit
46. $  curl  "http://127.0.0.1:8098/buckets/datomic/keys/config\\zookeeper"  /     -­‐X  PUT     -­‐d  "127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183"  /     -­‐H  "Content-­‐Type:  text/plain;  charset=utf-­‐8"
47. $  curl  "http://127.0.0.1:8098/buckets/datomic/keys/config\\zookeeper"  /     -­‐X  PUT     -­‐d  "127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183"  /     -­‐H  "Content-­‐Type:  text/plain;  charset=utf-­‐8"
48. $  curl  "http://127.0.0.1:8098/buckets/datomic/keys/config\\zookeeper"  /     -­‐X  PUT     -­‐d  "127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183"  /     -­‐H  "Content-­‐Type:  text/plain;  charset=utf-­‐8"
49. $  curl  "http://127.0.0.1:8098/buckets/datomic/keys/config\\zookeeper"  /     -­‐X  PUT     -­‐d  "127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183"  /     -­‐H  "Content-­‐Type:  text/plain;  charset=utf-­‐8"
51. Let’s automate that.
52. Vagrant Chef Clojure {
53. Riak Community Hangout 006 https://www.youtube.com/watch?v=VD9UCfQohQE
54. Thanks for listening.
    E-mail: hector@basho.com
    Twitter: @hectcastro
    Web: http://docs.basho.com