Riak CS Usage and Billing Statistics

A brief overview of Riak [0] and Riak CS [1], followed by details on multi-tenant usage and billing statistics.

After the slides, there is an interactive demonstration [2] on how to consume the statistics from Riak CS.

[0] http://basho.com/riak/
[1] http://basho.com/riak-cloud-storage/
[2] https://gist.github.com/hectcastro/126b5657f228096775c6

Transcript

2. // Who are we? * Founded in 2008 by group of ex-Akamai, Mitre, Apple * 120+ employees; > 50% eng; distributed company * Sponsors of Riak, the Apache 2.0-licensed project * Basho sells add-ons to Riak: Riak EDS and Riak CS
4. // What is Riak CS?
5. // What is Riak CS? * An Apache 2.0-licensed project
6. // What is Riak CS? * An Apache 2.0-licensed project * Built on top Riak
7. // What is Riak CS? * An Apache 2.0-licensed project * Built on top Riak * S3-compatible API (Swift coming soon)
8. // What is Riak CS? * An Apache 2.0-licensed project * Built on top Riak * S3-compatible API (Swift coming soon) * Large object support and multipart upload
9. // What is Riak CS? * An Apache 2.0-licensed project * Built on top Riak * S3-compatible API (Swift coming soon) * Large object support and multipart upload * Multi-tenancy and per-user reporting
11. "Distributed, masterless, highly-available key/value store."
12. "Distributed, masterless, highly-available key/value store." * Deployed as cluster of nodes (>= 5), scales horizontally
13. "Distributed, masterless, highly-available key/value store." * Deployed as cluster of nodes (>= 5), scales horizontally * Any node can coordinate requests, data replicated to 3 nodes by default, no SPOF
14. "Distributed, masterless, highly-available key/value store." * Deployed as cluster of nodes (>= 5), scales horizontally * Any node can coordinate requests, data replicated to 3 nodes by default, no SPOF * Eventually consistent with automatic failover
15. "Distributed, masterless, highly-available key/value store." * Deployed as cluster of nodes (>= 5), scales horizontally * Any node can coordinate requests, data replicated to 3 nodes by default, no SPOF * Eventually consistent with automatic failover * Key/value model with additional query methods
16. +---------------+ | | | Large object | | | +---------------+
17. +---------------+ | | | Large object | | | +---------------+ | | +----------+ +----------+ +----------+ | S3 API | | S3 API | | S3 API | |----------| |----------| |----------| | | | | | | | RiakCS | | RiakCS | | RiakCS | | | | | | | +----------+ +----------+ +----------+
18. +---------------+ | | | Large object | | | +---------------+ | | +----------+ +----------+ +----------+ | S3 API | | S3 API | | S3 API | |----------| |----------| |----------| | | | | | | | RiakCS | | RiakCS | | RiakCS | | | | | | | +----------+ +----------+ +----------+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ |1MB| |1MB| |1MB| |1MB| |1MB| |1MB| |1MB| |1MB| +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+
19. +---------------+ | | | Large object | | | +---------------+ | | +----------+ +----------+ +----------+ | S3 API | | S3 API | | S3 API | |----------| |----------| |----------| | | | | | | | RiakCS | | RiakCS | | RiakCS | | | | | | | +----------+ +----------+ +----------+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ |1MB| |1MB| |1MB| |1MB| |1MB| |1MB| |1MB| |1MB| +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ +------+ +------+ +------+ | | | | | | | Riak | | Riak | | Riak | | | | | | | +------+ +------+ +------+
20. // Multi-tenant reporting
21. // Multi-tenant reporting * Access and storage statistics
22. // Multi-tenant reporting * Access and storage statistics * As users interact with the API, Riak CS collects per- user statistics on operations
23. // Multi-tenant reporting * Access and storage statistics * As users interact with the API, Riak CS collects per- user statistics on operations * Statistics are sent to an aggregation subsystem
24. // Multi-tenant reporting * Access and storage statistics * As users interact with the API, Riak CS collects per- user statistics on operations * Statistics are sent to an aggregation subsystem * Aggregation subsystem periodically sends its accumulated log to be archived
25. // Multi-tenant reporting * Access and storage statistics * As users interact with the API, Riak CS collects per- user statistics on operations * Statistics are sent to an aggregation subsystem * Aggregation subsystem periodically sends its accumulated log to be archived * Archival subsystem sums all recorded accesses per-user and then stores a time-sliced rollup
26. // Access statistics
27. // Access statistics * Count * BytesIn * BytesOut
28. // Access statistics * Count * BytesIn * BytesOut These statistics are available for operations against:
29. // Access statistics * Count * BytesIn * BytesOut These statistics are available for operations against: * Buckets * Keys * ACLs
30. // Storage statistics
31. // Storage statistics * Objects * Bytes
32. // Storage statistics * Objects * Bytes Note: These statistics are separated by bucket.
33. { "Access": "not_requested", "Storage": [ { "StartTime": "20120316T123318Z", "EndTime": "20120316T123319Z", "fooBucket": { "Objects": 1, "Bytes": 3243225 }, "barBucket": { "Objects": 4, "Bytes": 234326234 } }, { "Errors": [] } ] }
34. // Usage interfaces
35. // Usage interfaces * HTTP * Magic usage bucket
36. HTTP: $ curl http://localhost:8080/riak-cs/usage/8NK4FH2SGKJJM8JIP2GU
37. HTTP: $ curl http://localhost:8080/riak-cs/usage/8NK4FH2SGKJJM8JIP2GU Usage bucket: $ s3cmd get s3://riak-cs/usage/8NK4FH2SGKJJM8JIP2GU/bx
38. // Result formats
39. // Result formats * XML * JSON
40. JSON: { "Access": "not_requested", "Storage": "not_requested" }
41. JSON: { "Access": "not_requested", "Storage": "not_requested" } XML: <?xml version="1.0" encoding="UTF-8"?> <Usage> <Access>not_requested</Access> <Storage>not_requested</Storage> </Usage>
42. // Future work
43. // Future work * Swift API
44. // Future work * Swift API * Keystone integration
45. // Future work * Swift API * Keystone integration * S3 COPY object
46. // Future work * Swift API * Keystone integration * S3 COPY object * S3 object versioning
47. // Future work * Swift API * Keystone integration * S3 COPY object * S3 object versioning * Server-side encryption
48. // Demo
49. // Thank you * E-mail: hector@basho.com * Twitter: @hectcastro * Web: http://docs.basho.com