Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Local Development with Riak CS

Local Development with Riak CS

Riak CS (Cloud Storage) [0] is simple, open source storage software built on top of Riak [1]. Riak CS provides multi-tenant, large object storage that can be used to build public or private clouds, or utilized as highly-available storage to power applications and services.

But for today, all you’ll care about is how to get Riak CS running locally so that you can bootstrap an app and talk to it via its S3-compatible API. This talk will provide an introduction to Riak CS, examples of how to talk to it from different programming languages and tools, and a walk-through of how to spin up a local instance using Vagrant [2] and Chef [3].

[0] http://basho.com/riak-cloud-storage/
[1] http://basho.com/riak/
[2] http://www.vagrantup.com/
[3] http://www.opscode.com/chef/

Hector Castro

April 20, 2013
Tweet

More Decks by Hector Castro

Other Decks in Technology

Transcript

  1. // What is Riak CS? * An Apache 2.0-licensed project

    * Built on top Riak * Presents a S3-compatible API
  2. // What is Riak CS? * An Apache 2.0-licensed project

    * Built on top Riak * Presents a S3-compatible API * Supports large object storage and multipart upload
  3. // What is Riak CS? * An Apache 2.0-licensed project

    * Built on top Riak * Presents a S3-compatible API * Supports large object storage and multipart upload * Provides multi-tenancy and per-user reporting
  4. // What is Riak CS? * An Apache 2.0-licensed project

    * Built on top Riak * Presents a S3-compatible API * Supports large object storage and multipart upload * Provides multi-tenancy and per-user reporting * Pluggable authentication
  5. "Distributed, masterless, highly-available key/value store." * Deployed as cluster of

    nodes (>= 5), scales horizontally * Any node can coordinate requests, data replicated to 3 nodes by default, no SPOF
  6. "Distributed, masterless, highly-available key/value store." * Deployed as cluster of

    nodes (>= 5), scales horizontally * Any node can coordinate requests, data replicated to 3 nodes by default, no SPOF * Eventually consistent with automatic failover
  7. "Distributed, masterless, highly-available key/value store." * Deployed as cluster of

    nodes (>= 5), scales horizontally * Any node can coordinate requests, data replicated to 3 nodes by default, no SPOF * Eventually consistent with automatic failover * Key/value model with additional query methods
  8. +---------------+ | | | Large object | | | +---------------+

    | | +----------+ +----------+ +----------+ | S3 API | | S3 API | | S3 API | |----------| |----------| |----------| | | | | | | | RiakCS | | RiakCS | | RiakCS | | | | | | | +----------+ +----------+ +----------+
  9. +---------------+ | | | Large object | | | +---------------+

    | | +----------+ +----------+ +----------+ | S3 API | | S3 API | | S3 API | |----------| |----------| |----------| | | | | | | | RiakCS | | RiakCS | | RiakCS | | | | | | | +----------+ +----------+ +----------+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ |1MB| |1MB| |1MB| |1MB| |1MB| |1MB| |1MB| |1MB| +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+
  10. +---------------+ | | | Large object | | | +---------------+

    | | +----------+ +----------+ +----------+ | S3 API | | S3 API | | S3 API | |----------| |----------| |----------| | | | | | | | RiakCS | | RiakCS | | RiakCS | | | | | | | +----------+ +----------+ +----------+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ |1MB| |1MB| |1MB| |1MB| |1MB| |1MB| |1MB| |1MB| +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ +------+ +------+ +------+ | | | | | | | Riak | | Riak | | Riak | | | | | | | +------+ +------+ +------+
  11. // Buckets * Like folders on a filesystem * Can

    house multiple objects * Names are globally unique
  12. // ACLs * Object: read, write, read/write ACL, full control

    * Bucket: list, create/delete/modify objects, read/write ACL, full control
  13. // ACLs * Object: read, write, read/write ACL, full control

    * Bucket: list, create/delete/modify objects, read/write ACL, full control * Defaults to creator ownership and full access control, denying access to all others
  14. // Future work * Swift API * Keystone integration *

    S3 COPY object * S3 object versioning
  15. // Future work * Swift API * Keystone integration *

    S3 COPY object * S3 object versioning * Server-side encryption
  16. // A good way to get started? $ git clone

    https://github.com/basho/riak_cs.git
  17. // A good way to get started? $ git clone

    https://github.com/basho/riak_cs.git Cloning into 'riak_cs'... remote: Counting objects: 11546, done. remote: Compressing objects: 100% (5131/5131), done. remote: Total 11546 (delta 6736), reused 11057 (delta 6291) Receiving objects: 100% (1/1), 5.62 MiB | 1.19 MiB/s, done. Resolving deltas: 100% (6736/6736), done. $
  18. // A good way to get started? $ cd riak_cs

    $ ls -l client_tests total 0 drwxr-xr-x 3 hector staff 102 Apr 19 10:20 clojure drwxr-xr-x 3 hector staff 102 Apr 19 10:20 erlang drwxr-xr-x 12 hector staff 408 Apr 19 11:03 php drwxr-xr-x 6 hector staff 204 Apr 19 11:49 python drwxr-xr-x 11 hector staff 374 Apr 19 11:30 ruby $
  19. // A good way to get started? $ cd riak_cs

    $ ls -l client_tests total 0 drwxr-xr-x 3 hector staff 102 Apr 19 10:20 clojure drwxr-xr-x 3 hector staff 102 Apr 19 10:20 erlang drwxr-xr-x 12 hector staff 408 Apr 19 11:03 php drwxr-xr-x 6 hector staff 204 Apr 19 11:49 python drwxr-xr-x 11 hector staff 374 Apr 19 11:30 ruby $ make test-client
  20. // A good way to get started? $ cd riak_cs

    $ ls -l client_tests total 0 drwxr-xr-x 3 hector staff 102 Apr 19 10:20 clojure drwxr-xr-x 3 hector staff 102 Apr 19 10:20 erlang drwxr-xr-x 12 hector staff 408 Apr 19 11:03 php drwxr-xr-x 6 hector staff 204 Apr 19 11:49 python drwxr-xr-x 11 hector staff 374 Apr 19 11:30 ruby $ make test-client <<TEST OUTPUT>>
  21. # User creation. client = Fog::RiakCS::Provisioning.new( riakcs_access_key_id: "XDULEEEXPMOQQFDUCWPV", riakcs_secret_access_key: "Px9nfHwrHif9dNumgTi5ZAJLC-lBBstoxQIF-A==",

    host: "localhost", port: 8080 ) # Usage statistics. client = Fog::RiakCS::Usage.new( riakcs_access_key_id: "XDULEEEXPMOQQFDUCWPV", riakcs_secret_access_key: "Px9nfHwrHif9dNumgTi5ZAJLC-lBBstoxQIF-A==", host: "localhost", port: 8080 )
  22. from boto.s3.connection import S3Connection, OrdinaryCallingFormat # Connect as user. S3Connection('XDULEEEXPMOQQFDUCWPV',

    'P9nfHwrHif9dNumgTi5ZAJLC-lBBstoxQIF-A==', is_secure=False, host='localhost', port=8080, debug=False, calling_format=OrdinaryCallingFormat())
  23. // Clojure $ cat project.clj (defproject java-s3-tests/java-s3-tests "0.0.1" :dependencies [[org.clojure/clojure

    "1.4.0"] [com.amazonaws/aws-java-sdk "1.3.11"] [com.reiddraper/clj-aws-s3 "0.4.0"]] :profiles {:dev {:dependencies [[midje "1.4.0"]]}} :plugins [[lein-midje "2.0.1"]] :description "integration tests for Riak CS using the offical Java SDK") $
  24. // Vagrant and Chef $ git clone https://github.com/basho/vagrant-riak-cs-cluster.git Cloning into

    'vagrant-riak-cs-cluster'... remote: Counting objects: 72, done. remote: Compressing objects: 100% (45/45), done. remote: Total 72 (delta 34), reused 55 (delta 23) Unpacking objects: 100% (72/72), done. $
  25. // Vagrant and Chef $ git clone https://github.com/basho/vagrant-riak-cs-cluster.git Cloning into

    'vagrant-riak-cs-cluster'... remote: Counting objects: 72, done. remote: Compressing objects: 100% (45/45), done. remote: Total 72 (delta 34), reused 55 (delta 23) Unpacking objects: 100% (72/72), done. $ cd vagrant-riak-cs-cluster $
  26. // Vagrant and Chef $ git clone https://github.com/basho/vagrant-riak-cs-cluster.git Cloning into

    'vagrant-riak-cs-cluster'... remote: Counting objects: 72, done. remote: Compressing objects: 100% (45/45), done. remote: Total 72 (delta 34), reused 55 (delta 23) Unpacking objects: 100% (72/72), done. $ cd vagrant-riak-cs-cluster $ RIAK_CS_CREATE_ADMIN_USER=1 vagrant up | grep "Riak CS"
  27. // Vagrant and Chef $ git clone https://github.com/basho/vagrant-riak-cs-cluster.git Cloning into

    'vagrant-riak-cs-cluster'... remote: Counting objects: 72, done. remote: Compressing objects: 100% (45/45), done. remote: Total 72 (delta 34), reused 55 (delta 23) Unpacking objects: 100% (72/72), done. $ cd vagrant-riak-cs-cluster $ RIAK_CS_CREATE_ADMIN_USER=1 vagrant up | grep "Riak CS" INFO: Riak CS Key: XDULEEEXPMOQQFDUCWPV INFO: Riak CS Secret: Px9nfHwrHif9dNumgTi5ZAJLC-lBBstoxQIF-A==
  28. // Hackathon project ideas * Specialized Riak CS CLI client

    [0] * User asset storage (profile photo, etc)
  29. // Hackathon project ideas * Specialized Riak CS CLI client

    [0] * User asset storage (profile photo, etc) * Dropbox clone [1]
  30. // Hackathon project ideas * Specialized Riak CS CLI client

    [0] * User asset storage (profile photo, etc) * Dropbox clone [1] * Vagrant + Chef local workflow improvements [2]
  31. // Hackathon project ideas * Specialized Riak CS CLI client

    [0] * User asset storage (profile photo, etc) * Dropbox clone [1] * Vagrant + Chef local workflow improvements [2] [0] https://github.com/hectcastro/riakcs [1] https://github.com/cmeiklejohn/consus [2] https://github.com/basho/vagrant-riak-cs-cluster
  32. // Atlanta Riak Meetup * First meeting April 23rd @

    Atlanta Tech Village * John E. Vincent (@lusis) on "Why Riak Matters" * Want to speak? Let's talk.
  33. // RICON East * May 13-14th in New York City

    * Talks, hacking, parties * Dedicated to future of Riak and distributed systems in production * Coupon code: 25offRICON * Site: http://ricon.io/east.html