Riak & Riak CS_ Gaming

Transcript

1. 1 Riak Enterprise Grade NoSQL Distributed Data Store Pavan@basho.com
2. AGENDA •  Riak Overview •  Riak Architecture •  Use Cases •  Riak EDS •  Riak CS •  Q & A
3. About Basho Our Mission is to Be The Leader in Distributed Systems •  Founded January 2008 •  115+ employees •  Headquartered in Cambridge, with regional offices in San Francisco, Washington DC, London and Tokyo •  Makers of Riak- A popular distributed key- value store •  Thousands of Users Worldwide including over 20% of the Fortune 50 •  30,000+ downloads per month now up from 19,500 in Dec 2011 •  Strategic Partners include Citrix, IDC Frontier, Yahoo! Japan, and Microsoft
4. Basho’s Product Family Distributed Data Technology is Our Passion EnterpriseDS Open Source Distributed Database Commercial Distributed Database Distributed Cloud Storage Platform •  Always-available, scalable, low-cost NoSQL database •  Over 35,000 Downloads per Month •  Thousands of users worldwide •  Available Since Sept 2009 •  Version 1.0 unveiled September 2011 •  Subsequent versions released along side Riak EDS •  Adds multi-data center replication, monitoring & 24x7 support •  Requires commercial contract and secure download •  Version 1.1 launched with Riak Control in Feb 2012 •  Version 1.2 launched in August 2012 •  Version 1.3 launched in February 2013 •  Expands with multi- tenancy, large object support, metering and Amazon S3 API •  Launched on March 27, 2012 •  Used by multiple global cloud operators •  Software released to open source on March 20th •  Available in commercial version as well (multi- data center replication and 24/7 support)
5. Riak •  Master-Slave architecture •  Application sharding •  Distributed model •  Active-Active with write scalability •  Active-Passive Say “NO” to Say “Yes” to
6. Riak Riak is a distributed NoSQL key-value store. Simple operations •  Get •  Put •  Delete
7. Key-Value Data Model •  Keys are grouped into buckets. •  All data (objects) are referenced by keys •  Object is composed of metadata and value Object/key Operations KEY VALUE KEY VALUE KEY VALUE bucket
8. Masterless & Highly Available Any node can serve client requests Fallbacks are used when nodes are down Always accepts read and write requests Per-request quorums
9. Consistent Hashing & The Ring •  160-bit integer keyspace •  Divided into fixed number of evenly-sized partitions •  Partitions are claimed by nodes in the cluster •  Replicas go to the N partitions following the key 32 partitions N=3 node 0 node 1 node 2 node 3 hash(“product/iphone”) 2160/4 2160/2 0
10. Failure Scenario •  Node fails •  Requests go to fallback nodes hash(“product/iphone”) node 0 node 1 node 2 node 3 X X X X X X X X
11. Hinted Handoff •  Node comes back •  “Handoff” - data returns to recovered node •  Normal operations resume hash(“product/iphone”) node 0 node 1 node 2 node 3
12. Riak’s core capability Scalable Add commodity hardware to get more [throughput | processing | storage]
13. Riak’s core capability Fault Tolerant All nodes participate equally (no SPOF) All data is replicated (n=3 by default) Cluster transparently survives node failure & network partition
14. Tunable Consistency •  n_val - number of replica to store; bucket-level setting. Defaults to “3”. •  w - number of replicas required for a successful write; Defaults to “2”. •  r - number of replica acks required for a successful read. request-level setting. Defaults to “2”. •  pr, pw & dw • Tweak consistency vs. availability
15. Two APIs HTTP (just like the web) Protocol Bu"ers (thank you, Google) Client Libraries Ruby, Node.js, Java, Python, Perl, Erlang, PHP, C, Scala, Haskell, Lisp,.NET, Play, and more (supported by either Basho or the community).
16. Riak Backend • Riak has a pluggable backend architecture • Bitcask, LevelDB are used the most in production depending on use-case • All writes are appends to a file • This provides crash safety and fast writes
17. Accessing Data in Riak Retrieving Single Objects •  Support for retrieving the object associated with a particular bucket / key •  Support for retrieving all of the keys associated with a particular bucket Object/Key Operations Collecting, Parsing, and Storing Data •  Distributed, full-text search engine with an easy-to-use query language, a Solr-like HTTP interface and a Apache Lucene-style query syntax •  Support for a wide variety of mime types, including JSON, plain text, XML and Erlang) •  Ideal for indexing JSON documents, as indexes are built automatically from a schema. Riak Search Seeking Reverse Lookups on Data Stored •  Provides the ability, at write time, to tag an object stored in Riak with one or more values (key/value metadata), which can then be queried •  Useful for finding data that is based on terms other than an objects’ bucket/key pair, or for adding metadata values to a binary object or opaque blob Secondary Indexes (Riak 2i) Processing a Large Dataset •  Provides the general ability to analyze and aggregate data in phases with data locality •  Features Javascript support and Erlang for performance benefit MapReduce Riak Search and 2I Query Results Can be Used as an Input to MapReduce
18. Always-Available Scale-Out Fast Ultra-Low Latency Any data types Programmer ease-of-use Riak Technical Architecture Riak Clients! (Ruby, Java, Node.js, PHP, .NET, etc)! Webmachine! (HTTP)! Protocol Buffers! (Binary, Persistent Connect)! MDC Replication! * Riak EnterpriseDS Only! Riak SNMP / JMX! * Riak EnterpriseDS Only! Riak KVS" (with 2i, " MapReduce)! Riak Search" (Distributed, Full- Text)! Riak Pipe" ! Riak Core" (Ring Management, Partitioning, Anti-Entropy, Replication, Version Control)! Pluggable Storage Backends! Bitcask! LevelDB! Riak Control!
19. What Riak is good for ?
20. What Riak Isn’t •  NOT Relational •  No fixed schema •  NOT Right for Every Project •  Large Objects (Riak CS is a good fit here) •  Dynamic Queries(SQL)
21. Modeling Applications in Riak
22. Ideal Riak Scenarios •  When you have enough data to require >1 physical machine (preferably > 4) •  When availability is the top requirement •  When your data can be modeled as keys and values When to Use Popular Use Cases •  Ad Networks •  Digital Media •  On-Line Games •  Social Networks •  Social Analysis •  Cloud Operators •  Messaging Services •  Product Catalogs •  Document Management •  Health Care Information Management
23. Riak Production Users- growing & growing …. Mobile, Retail & Social Cloud Computing & Advertising Security and Others Gaming, Payments and Others
24. Riak at Rovio
25. Riak Sizing
26. Riak Sizing
27. Riak Sizing
28. Web / Mobile App Growth Case Study for Top Rated Apple App Store App •  #4 most popular Apple App Store Social Networking App at EOY behind Facebook, Skype and Twitter •  Truly Viral Growth: Scaled 10x between Thanksgiving and New Years Day •  Required scaling across multiple IaaS / hosting providers •  Surpassed one billion operations per day
29. Riak MDC- (EDS) Cloud Mobile Social Data Center #2 Data Center #3 Data Center #1 Multi-Data Center Replication Applications, Users and Machines Generate Data 1 2 Riak Stores and Manages Data Efficiently and Effectively •  Clusters are local to regional users to solve latency •  Replication is uni-directional, remote clusters can be setup to replicate data back to a primary cluster, thus synchronizing bi-directionally. •  Easily deploy in many regional zones •  Write everywhere solution •  Easy to scale, can easily add additional data centers
30. Full-sync Replication
31. Real-time Replication
32. Product Information Repository
33. Multi-Device Session Store Case Study Showcases Seamless User Experience The Global Session Store Manages a Seamless User Session throughout a Customer’s multi-mode experience, from Web to device Philadelphia Data Center Denver
34. Riak 1.3 GA Now •  Active Anti Entropy •  Replication enhancements for MDC •  IPv6 support •  New Look for Riak Control
35. Riak 1.4 (Roadmap) •  Dynamic Ring Sizing •  2i Pagination •  Performance and Scaling Improvements
36. Riak Sizing AWS cr1.8xlarge 16-core/32-thread, 250GB SSD, 240GB ram 5-node cluster, 120 concurrent connections >15k ops/sec median latency 5ms, mean 8ms, 95th percentile ~ 16ms 80% writes 10kb object size
37. Riak Sizing 1 x quad core 2.4Ghz E5620 w/ 32GB RAM and 24 x 1TB 7200 RPM drives 200 concurrent connections 7k ops/sec ~10-20ms median latency Get/put mix 10kb object size The above numbers are for 1 machine only.
38. Stats and Monitoring (1) • Riak exposes data about current operating status (counters, histograms, etc.) via the HTTP /stats endpoint or ‘riak-admin status’ • Anything that speaks HTTP can be plugged into Riak • Plugins exist for most OSS monitoring tools (munin, cacti, nagios, graphite, statsd)
39. Stats and Monitoring (2) •  ‘riaknostic’ is a suite of diagnostic checks that can used to debug your cluster before it’s in production; checks for common misconfigurations •  Riak Control is a full-fledged management GUI that Basho develops and maintains.
40. What is Riak CS? Key features: •  Multi-Tenant support •  User Authentication and Authorization •  Amazon S3 API-compatibility •  Per-Tenant visibility •  Provisioning, Metering, Billing and Reporting •  Multi part upload up to 5TB
41. Riak CS in Action
42. Reporting Large Objects AuthZ Riak CS Use Cases Storage for Cloud Computing S3 Without AWS Cloud Drive (General Content Storage) Backup-as-a- Service Archival and Preservation Integration with Workflow Multi-Tenancy
43. Multi-Datacenter Replication •  Multi-site storage replication •  Data locality •  Availability in disaster scenarios •  Active backup
44. Multi-Datacenter Replication •  Global information for users, buckets and manifests is streamed in real- time •  Objects are replicated in full or real- time sync mode •  If a client requests an object from a site but not all of the blocks that constitute that object have been replicated to that site, missing blocks will be requested and streamed from the “origin” cluster How It Works
45. Sign up here- http://ricon.io/west.html
46. Q & A