Local Development with Riak CS

Riak CS (Cloud Storage) [0] is simple, open source storage software built on top of Riak [1]. Riak CS provides multi-tenant, large object storage that can be used to build public or private clouds, or utilized as highly-available storage to power applications and services.

But for today, all you’ll care about is how to get Riak CS running locally so that you can bootstrap an app and talk to it via its S3-compatible API. This talk will provide an introduction to Riak CS, examples of how to talk to it from different programming languages and tools, and a walk-through of how to spin up a local instance using Vagrant [2] and Chef [3].

[0] http://basho.com/riak-cloud-storage/
[1] http://basho.com/riak/
[2] http://www.vagrantup.com/
[3] http://www.opscode.com/chef/

Transcript

3. // What is Riak CS?
4. // What is Riak CS? * An Apache 2.0-licensed project
5. // What is Riak CS? * An Apache 2.0-licensed project * Built on top Riak
6. // What is Riak CS? * An Apache 2.0-licensed project * Built on top Riak * Presents a S3-compatible API
7. // What is Riak CS? * An Apache 2.0-licensed project * Built on top Riak * Presents a S3-compatible API * Supports large object storage and multipart upload
8. // What is Riak CS? * An Apache 2.0-licensed project * Built on top Riak * Presents a S3-compatible API * Supports large object storage and multipart upload * Provides multi-tenancy and per-user reporting
9. // What is Riak CS? * An Apache 2.0-licensed project * Built on top Riak * Presents a S3-compatible API * Supports large object storage and multipart upload * Provides multi-tenancy and per-user reporting * Pluggable authentication
11. "Distributed, masterless, highly-available key/value store."
12. "Distributed, masterless, highly-available key/value store." * Deployed as cluster of nodes (>= 5), scales horizontally
13. "Distributed, masterless, highly-available key/value store." * Deployed as cluster of nodes (>= 5), scales horizontally * Any node can coordinate requests, data replicated to 3 nodes by default, no SPOF
14. "Distributed, masterless, highly-available key/value store." * Deployed as cluster of nodes (>= 5), scales horizontally * Any node can coordinate requests, data replicated to 3 nodes by default, no SPOF * Eventually consistent with automatic failover
15. "Distributed, masterless, highly-available key/value store." * Deployed as cluster of nodes (>= 5), scales horizontally * Any node can coordinate requests, data replicated to 3 nodes by default, no SPOF * Eventually consistent with automatic failover * Key/value model with additional query methods
17. +---------------+ | | | Large object | | | +---------------+
18. +---------------+ | | | Large object | | | +---------------+ | | +----------+ +----------+ +----------+ | S3 API | | S3 API | | S3 API | |----------| |----------| |----------| | | | | | | | RiakCS | | RiakCS | | RiakCS | | | | | | | +----------+ +----------+ +----------+
19. +---------------+ | | | Large object | | | +---------------+ | | +----------+ +----------+ +----------+ | S3 API | | S3 API | | S3 API | |----------| |----------| |----------| | | | | | | | RiakCS | | RiakCS | | RiakCS | | | | | | | +----------+ +----------+ +----------+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ |1MB| |1MB| |1MB| |1MB| |1MB| |1MB| |1MB| |1MB| +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+
20. +---------------+ | | | Large object | | | +---------------+ | | +----------+ +----------+ +----------+ | S3 API | | S3 API | | S3 API | |----------| |----------| |----------| | | | | | | | RiakCS | | RiakCS | | RiakCS | | | | | | | +----------+ +----------+ +----------+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ |1MB| |1MB| |1MB| |1MB| |1MB| |1MB| |1MB| |1MB| +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ +------+ +------+ +------+ | | | | | | | Riak | | Riak | | Riak | | | | | | | +------+ +------+ +------+
21. // Objects
22. // Objects * Contents are opaque
23. // Objects * Contents are opaque * Store any file type
24. // Buckets
25. // Buckets * Like folders on a filesystem
26. // Buckets * Like folders on a filesystem * Can house multiple objects
27. // Buckets * Like folders on a filesystem * Can house multiple objects * Names are globally unique
28. // ACLs
29. // ACLs * Object: read, write, read/write ACL, full control
30. // ACLs * Object: read, write, read/write ACL, full control * Bucket: list, create/delete/modify objects, read/write ACL, full control
31. // ACLs * Object: read, write, read/write ACL, full control * Bucket: list, create/delete/modify objects, read/write ACL, full control * Defaults to creator ownership and full access control, denying access to all others
32. // Future work
33. // Future work * Swift API
34. // Future work * Swift API * Keystone integration
35. // Future work * Swift API * Keystone integration * S3 COPY object
36. // Future work * Swift API * Keystone integration * S3 COPY object * S3 object versioning
37. // Future work * Swift API * Keystone integration * S3 COPY object * S3 object versioning * Server-side encryption
39. // Common configuration settings
40. // Common configuration settings * Proxy: localhost:8080
41. // Common configuration settings * Proxy: localhost:8080 * Host: localhost
42. // Common configuration settings * Proxy: localhost:8080 * Host: localhost * Port: 8080
43. // Common configuration settings * Proxy: localhost:8080 * Host: localhost * Port: 8080 * Protocol: HTTP
44. // A good way to get started?
45. // A good way to get started? $ git clone https://github.com/basho/riak_cs.git
46. // A good way to get started? $ git clone https://github.com/basho/riak_cs.git Cloning into 'riak_cs'... remote: Counting objects: 11546, done. remote: Compressing objects: 100% (5131/5131), done. remote: Total 11546 (delta 6736), reused 11057 (delta 6291) Receiving objects: 100% (1/1), 5.62 MiB | 1.19 MiB/s, done. Resolving deltas: 100% (6736/6736), done. $
47. // A good way to get started? $ cd riak_cs
48. // A good way to get started? $ cd riak_cs $
49. // A good way to get started? $ cd riak_cs $ ls -l client_tests
50. // A good way to get started? $ cd riak_cs $ ls -l client_tests total 0 drwxr-xr-x 3 hector staff 102 Apr 19 10:20 clojure drwxr-xr-x 3 hector staff 102 Apr 19 10:20 erlang drwxr-xr-x 12 hector staff 408 Apr 19 11:03 php drwxr-xr-x 6 hector staff 204 Apr 19 11:49 python drwxr-xr-x 11 hector staff 374 Apr 19 11:30 ruby $
51. // A good way to get started? $ cd riak_cs $ ls -l client_tests total 0 drwxr-xr-x 3 hector staff 102 Apr 19 10:20 clojure drwxr-xr-x 3 hector staff 102 Apr 19 10:20 erlang drwxr-xr-x 12 hector staff 408 Apr 19 11:03 php drwxr-xr-x 6 hector staff 204 Apr 19 11:49 python drwxr-xr-x 11 hector staff 374 Apr 19 11:30 ruby $ make test-client
52. // A good way to get started? $ cd riak_cs $ ls -l client_tests total 0 drwxr-xr-x 3 hector staff 102 Apr 19 10:20 clojure drwxr-xr-x 3 hector staff 102 Apr 19 10:20 erlang drwxr-xr-x 12 hector staff 408 Apr 19 11:03 php drwxr-xr-x 6 hector staff 204 Apr 19 11:49 python drwxr-xr-x 11 hector staff 374 Apr 19 11:30 ruby $ make test-client <<TEST OUTPUT>>
53. // Ruby
54. // Ruby $ gem install aws-sdk
55. // Ruby $ gem install aws-sdk Successfully installed aws-sdk-1.9.1 $
56. # General configuration. AWS.config( access_key_id: "XDULEEEXPMOQQFDUCWPV", secret_access_key: "Px9nfHwrHif9dNumgTi5ZAJLC-lBBstoxQIF-A==", s3_endpoint: "localhost", use_ssl: false ) # Override default port. class AWS::Core::Http::Request def port; 8080; end end
57. // Ruby
58. // Ruby $ gem install fog
59. // Ruby $ gem install fog Successfully installed fog-1.10.1 $
60. # User creation. client = Fog::RiakCS::Provisioning.new( riakcs_access_key_id: "XDULEEEXPMOQQFDUCWPV", riakcs_secret_access_key: "Px9nfHwrHif9dNumgTi5ZAJLC-lBBstoxQIF-A==", host: "localhost", port: 8080 ) # Usage statistics. client = Fog::RiakCS::Usage.new( riakcs_access_key_id: "XDULEEEXPMOQQFDUCWPV", riakcs_secret_access_key: "Px9nfHwrHif9dNumgTi5ZAJLC-lBBstoxQIF-A==", host: "localhost", port: 8080 )
61. // Python
62. // Python $ pip install boto
63. // Python $ pip install boto Successfully installed boto $
64. from boto.s3.connection import S3Connection, OrdinaryCallingFormat # Connect as user. S3Connection('XDULEEEXPMOQQFDUCWPV', 'P9nfHwrHif9dNumgTi5ZAJLC-lBBstoxQIF-A==', is_secure=False, host='localhost', port=8080, debug=False, calling_format=OrdinaryCallingFormat())
65. // Clojure
66. // Clojure $ cat project.clj
67. // Clojure $ cat project.clj (defproject java-s3-tests/java-s3-tests "0.0.1" :dependencies [[org.clojure/clojure "1.4.0"] [com.amazonaws/aws-java-sdk "1.3.11"] [com.reiddraper/clj-aws-s3 "0.4.0"]] :profiles {:dev {:dependencies [[midje "1.4.0"]]}} :plugins [[lein-midje "2.0.1"]] :description "integration tests for Riak CS using the offical Java SDK") $
68. // Clojure
69. // Clojure $ lein deps
70. // Clojure $ lein deps <<DEPENDENCY RESOLUTION OUTPUT>> $
71. (ns java-s3-tests.test.client (:require [aws.sdk.s3 :as s3])) (s3/client "XDULEEEXPMOQQFDUCWPV" "P9nfHwrHif9dNumgTi5ZAJLC-lBBstoxQIF-A==" {:proxy-host "localhost" :proxy-port 8080 :protocol :http})
72. // Vagrant and Chef
73. // Vagrant and Chef $ git clone https://github.com/basho/vagrant-riak-cs-cluster.git
74. // Vagrant and Chef $ git clone https://github.com/basho/vagrant-riak-cs-cluster.git Cloning into 'vagrant-riak-cs-cluster'... remote: Counting objects: 72, done. remote: Compressing objects: 100% (45/45), done. remote: Total 72 (delta 34), reused 55 (delta 23) Unpacking objects: 100% (72/72), done. $
75. // Vagrant and Chef $ git clone https://github.com/basho/vagrant-riak-cs-cluster.git Cloning into 'vagrant-riak-cs-cluster'... remote: Counting objects: 72, done. remote: Compressing objects: 100% (45/45), done. remote: Total 72 (delta 34), reused 55 (delta 23) Unpacking objects: 100% (72/72), done. $ cd vagrant-riak-cs-cluster $
76. // Vagrant and Chef $ git clone https://github.com/basho/vagrant-riak-cs-cluster.git Cloning into 'vagrant-riak-cs-cluster'... remote: Counting objects: 72, done. remote: Compressing objects: 100% (45/45), done. remote: Total 72 (delta 34), reused 55 (delta 23) Unpacking objects: 100% (72/72), done. $ cd vagrant-riak-cs-cluster $ RIAK_CS_CREATE_ADMIN_USER=1 vagrant up | grep "Riak CS"
77. // Vagrant and Chef $ git clone https://github.com/basho/vagrant-riak-cs-cluster.git Cloning into 'vagrant-riak-cs-cluster'... remote: Counting objects: 72, done. remote: Compressing objects: 100% (45/45), done. remote: Total 72 (delta 34), reused 55 (delta 23) Unpacking objects: 100% (72/72), done. $ cd vagrant-riak-cs-cluster $ RIAK_CS_CREATE_ADMIN_USER=1 vagrant up | grep "Riak CS" INFO: Riak CS Key: XDULEEEXPMOQQFDUCWPV INFO: Riak CS Secret: Px9nfHwrHif9dNumgTi5ZAJLC-lBBstoxQIF-A==
78. // Hackathon project ideas
79. // Hackathon project ideas * Specialized Riak CS CLI client [0]
80. // Hackathon project ideas * Specialized Riak CS CLI client [0] * User asset storage (profile photo, etc)
81. // Hackathon project ideas * Specialized Riak CS CLI client [0] * User asset storage (profile photo, etc) * Dropbox clone [1]
82. // Hackathon project ideas * Specialized Riak CS CLI client [0] * User asset storage (profile photo, etc) * Dropbox clone [1] * Vagrant + Chef local workflow improvements [2]
83. // Hackathon project ideas * Specialized Riak CS CLI client [0] * User asset storage (profile photo, etc) * Dropbox clone [1] * Vagrant + Chef local workflow improvements [2] [0] https://github.com/hectcastro/riakcs [1] https://github.com/cmeiklejohn/consus [2] https://github.com/basho/vagrant-riak-cs-cluster
84. // Atlanta Riak Meetup * First meeting April 23rd @ Atlanta Tech Village * John E. Vincent (@lusis) on "Why Riak Matters" * Want to speak? Let's talk.
85. // RICON East * May 13-14th in New York City * Talks, hacking, parties * Dedicated to future of Riak and distributed systems in production * Coupon code: 25offRICON * Site: http://ricon.io/east.html
86. // Thank you * E-mail: hector@basho.com * Twitter: @hectcastro * Web: http://docs.basho.com