Share
Embed
Multi dimensional scaling with CouchBase
Presented by Clarence Tauro at the Singapore Spring User Group on August 20th
SingaSUG
August 20, 2015
Transcript
25 slides
-
MULTI-DIMENSIONAL SCALING – A NEW ARCHITECTURE FOR SCALING BIG DATA
APPLICATION NoSQL Night Singapore -
©2015 Couchbase Inc. 2 Agenda § Brief history
of scaling in database § Scaling up § Scaling out § NoSQL workloads and scalability model § Core data operations, indexing and querying § Homogenous HW scaling § Introducing multi-‐dimensional scaling § Isolation of workloads through a services architecture § Independent HW scaling § Q & A -
©2015 Couchbase Inc. 3 Disclaimer & Notes
§ Disclaimer: The views expressed in this presentation are our own and do not necessarily reflect the views of Couchbase § Most of the content in this presentation is originally created by Anil, Product Manager, Couchbase § Thanks Anil, for allowing me to re-‐use your slides J 3 -
©2015 Couchbase Inc. 4 About the Speaker
§ Clarence J M Tauro – [email protected] - Senior Instructor, Couchbase - ~11 Years Professional Teaching and Consulting Experience - Worked at Pivotal – Instructor/Consultant for Spring/Spring Security/Spring Web/Enterprise Integration with Spring/Spring JMS/Spring Web/Spring Batch, Pivotal Hadoop/Cloud Foundry - PhD in Computer Science from Christ University [thesis accepted] - Hard-core Dog lover 4 -
©2015 Couchbase Inc. 6 Scaling up § Scale-‐up
architecture § Cluster processors – hyper-‐threading to cores § Locally partition workload among processors § Communicate over memory -
©2015 Couchbase Inc. 7 Scaling Up – Pros
& Cons Pros § Can result in major performance improvement § Machines can now support having many cores and terabytes of RAM Cons § Expensive § Requires downtime § Performance bounded – at some point the database engine itself becomes the bottleneck § Limited in scalability and elasticity -
©2015 Couchbase Inc. 8 Scaling out § Scale-‐out
architecture § Cluster of commodity HW § Horizontal partitioning of data on cluster nodes § Communicate over network -
©2015 Couchbase Inc. 9 Scaling Out – Pros
& Cons Pros § Simple and easily scalable § Data evenly split across cluster of nodes § Scales linearly with throughput § Highly available § No single point of failure Cons § Not great for all workloads – data, index and query -
©2015 Couchbase Inc. 10 So Which Model is
the Right Model ? Scale up or scale out? -
©2015 Couchbase Inc. 11 Scaling Up vs. Scale
Out Link to whitepaper -‐ http://www.msr-‐ waypoint.com/pubs/204499/ a20-‐appuswamy.pdf -
©2015 Couchbase Inc. 13 NoSQL Workloads § One
database, many workloads § Core data processing: GETs & SETs for a given key § Indexing: Index maintenance and lookups § Querying: Combine index and data with complex just-‐in-‐time data re-‐shaping, ordering, grouping, aggregations, and more Varying resource requirements -‐ CPU, RAM, I/O, Network Varying methods to optimize latency & throughput for each -
©2015 Couchbase Inc. 14 Scalability Model Today
Homogenous Scaling § Each node get a slice of the workload § Simple to do… But... § Workloads compete and interfere with each other § Can’t fine tune each workload § Core Data operation are partition-‐able so great with wider fan-‐out § Indexing and queries aren’t always partitionable, so worse with wider fan-‐out Index Workload Couchbase Cluster Query Workload Data Workload node1 node8 -
©2015 Couchbase Inc. 16 Modern Architecture What
is Multi-‐Dimensional Scalability? MDS is the architecture that enables independent scaling of data, query and indexing workloads. Index Service Couchbase Cluster Query Service Data Service node1 node8 -
©2015 Couchbase Inc. 17 Couchbase Cluster node1
node8 Modern Architecture § Isolated Service for minimized interference § Independent “zones” for Query, Index and Data Services Minimize indexing and query overhead on core key-‐value operations. Index Service Query Service Data Service -
©2015 Couchbase Inc. 18 Modern Architecture §
Independent Scalability for Best Computational Capacity per Service Heavier indexing (index more fields) : scale up index service nodes More RAM for query processing: scale up query service nodes Couchbase Cluster node1 node8 node9 Data Service Index Service Query Service -
©2015 Couchbase Inc. 20 Full Cluster Architecture
STORAGE Couchbase Server 1 SHARD 7 SHARD 9 SHARD 5 SHARD SHARD SHARD Managed Cache Cluster Manager Cluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 2 SHARD 7 SHARD 9 SHARD 5 SHARD SHARD SHARD Managed Cache Cluster Manager Cluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 3 SHARD 7 SHARD 9 SHARD 5 SHARD SHARD SHARD Managed Cache Cluster Manager Cluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 4 SHARD 7 SHARD 9 SHARD 5 SHARD SHARD SHARD Managed Cache Cluster Manager Cluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 5 SHARD 7 SHARD 9 SHARD 5 SHARD SHARD SHARD Managed Cache Cluster Manager Cluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 6 SHARD 7 SHARD 9 SHARD 5 SHARD SHARD SHARD Managed Cache Cluster Manager Cluster Manager Managed Cache Storage Data Service Index Service Query Service -
©2015 Couchbase Inc. 21 Full Cluster Architecture
STORAGE Couchbase Server 1 SHARD 7 SHARD 9 SHARD 5 SHARD SHARD SHARD Managed Cache Cluster Manager Cluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 2 SHARD 7 SHARD 9 SHARD 5 SHARD SHARD SHARD Managed Cache Cluster Manager Cluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 3 SHARD 7 SHARD 9 SHARD 5 SHARD SHARD SHARD Managed Cache Cluster Manager Cluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 4 SHARD 7 SHARD 9 SHARD 5 SHARD SHARD SHARD Managed Cache Cluster Manager Cluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 5 SHARD 7 SHARD 9 SHARD 5 SHARD SHARD SHARD Managed Cache Cluster Manager Cluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 6 SHARD 7 SHARD 9 SHARD 5 SHARD SHARD SHARD Managed Cache Cluster Manager Cluster Manager Managed Cache Storage Data Service Index Service Query Service -
©2015 Couchbase Inc. 22 Storage Engine Cluster
Manager Data Service Projector & Router New Services in Couchbase Server 4.0 Query Service Index Service Supervisor Index maintenance & Scan coordinator Index#2 Index#1 Query Processor cbq-‐engine Bucket#1 Bucket#2 DCP Stream Index#4 Index#3 ... Bucket#2 18093 11211 18901 Managed Cache -
©2015 Couchbase Inc. 23 Recap § MDS enables
unprecedented control of scalability with Couchbase Server § Separate out competing workloads to independent services § Independently scale each service “zone” within the cluster § Couchbase Server with MDS maximizes scalability and performance § Improves scale and performance to degrees not possible with other NoSQL or big-‐data engines on premise or in the cloud § Improved price/performance and squeezes more performance and throughput for mission-‐critical systems -
Get Started with Couchbase Server 4.0: www.couchbase.com/beta Get Trained
on Couchbase: http://training.couchbase.com CD220: Developing Couchbase NoSQL Applications Oct 20 – Oct 23 2015 CS300: Couchbase NoSQL Server Administration Nov 17 – Nov 20 Enroll Today!