FOSDEM 2013 : Getting Started with Couchhbase Server 2.0

Monday, February 4, 13

Technical Evangelist twi0er: @tgrall email: [email protected] Tugdual “Tug” Grall Ge#ng
Started with Couchbase Monday, February 4, 13

Couchbase Server Core Principles Easy Scalability Consistent High
Performance Always On 24x365 Grow cluster without applica7on changes, without down7me with a single click Consistent sub-‐millisecond read and write response 7mes with consistent high throughput No down7me for so>ware upgrades, hardware maintenance, etc. Flexible Data Model JSON document model with no ﬁxed schema. JSON JSON JSON JSON JSON PERFORMANCE Monday, February 4, 13

Couchbase 2.0 New Features JSON support Indexing and Querying Cross
data center replication Incremental Map Reduce Monday, February 4, 13

Couchbase Handles Real World Scale Monday, February 4, 13

Sub:tle Couchbase Server 2.0 Architecture Heartbeat Process monitor Global singleton
supervisor Conﬁgura7on manager on each node Rebalance orchestrator Node health monitor one per cluster vBucket state and replica7on manager hDp REST management API/Web UI HTTP 8091 Erlang port mapper 4369 Distributed Erlang 21100 -‐ 21199 Erlang/OTP storage interface Couchbase EP Engine 11210 Memcapable 2.0 Moxi 11211 Memcapable 1.0 Memcached New Persistence Layer 8092 Query API Query Engine Data Manager Cluster Manager Monday, February 4, 13

Couchbase Server 2.0 Architecture New Persistence Layer storage interface Couchbase
EP Engine 11210 Memcapable 2.0 Moxi 11211 Memcapable 1.0 Object-‐level Cache Disk Persistence 8092 Query API Query Engine HTTP 8091 Erlang port mapper 4369 Distributed Erlang 21100 -‐ 21199 Heartbeat Process monitor Global singleton supervisor Conﬁgura7on manager on each node Rebalance orchestrator Node health monitor one per cluster vBucket state and replica7on manager hDp REST management API/Web UI Erlang/OTP Server/Cluster Management & CommunicaYon (Erlang) RAM Cache, Indexing & Persistence Management (C & V8) The Unreasonable Effectiveness of C by Damien Katz Monday, February 4, 13

Apache 2.0 Open Source Project hIps://github.com/couchbase/ hIps://github.com/couchbaselabs/ hIp://review.couchbase.org/ Gerrit: Monday,
February 4, 13

SETTING UP TO DEVELOP Monday, February 4, 13

Install Couchbase Server 2.0 Ubuntu RedHat Mac OS X Windows
or build from sources Monday, February 4, 13

Oﬃcial SDKs www.couchbase.com/develop Clojure Python Ruby libcouchbase Go Monday, February
4, 13

COUCHBASE OPERATIONS Monday, February 4, 13

DEMONSTRATION Monday, February 4, 13

Write OperaJon 3 3 2 Managed Cache Disk Queue Disk
Replica7on Queue App Server Couchbase Server Node Doc 1 To other node Monday, February 4, 13

Write OperaJon 3 3 2 Managed Cache Disk Queue Disk
Replica7on Queue App Server Couchbase Server Node Doc 1 Doc 1 Doc 1 To other node Monday, February 4, 13

Basic OperaJons COUCHBASE SERVER CLUSTER • Docs distributed evenly
across servers • Each server stores both acYve and replica docs Only one doc ac7ve at a 7me • Client library provides app with simple interface to database • Cluster map provides map to which server doc is on App never needs to know • App reads, writes, updates docs • MulYple app servers can access same document at same Yme ACTIVE Doc 5 Doc 2 Doc Doc Doc SERVER 1 ACTIVE Doc 4 Doc 7 Doc Doc Doc SERVER 2 Doc 8 ACTIVE Doc 1 Doc 2 Doc Doc Doc SERVER 3 Doc 6 Doc 9 Monday, February 4, 13

across servers • Each server stores both acYve and replica docs Only one doc ac7ve at a 7me • Client library provides app with simple interface to database • Cluster map provides map to which server doc is on App never needs to know • App reads, writes, updates docs • MulYple app servers can access same document at same Yme ACTIVE Doc 5 Doc 2 Doc Doc Doc SERVER 1 ACTIVE Doc 4 Doc 7 Doc Doc Doc SERVER 2 Doc 8 ACTIVE Doc 1 Doc 2 Doc Doc Doc REPLICA Doc 4 Doc 1 Doc 8 Doc Doc Doc REPLICA Doc 6 Doc 3 Doc 2 Doc Doc Doc REPLICA Doc 7 Doc 9 Doc 5 Doc Doc Doc SERVER 3 Doc 6 Doc 9 Monday, February 4, 13

across servers • Each server stores both acYve and replica docs Only one doc ac7ve at a 7me • Client library provides app with simple interface to database • Cluster map provides map to which server doc is on App never needs to know • App reads, writes, updates docs • MulYple app servers can access same document at same Yme ACTIVE Doc 5 Doc 2 Doc Doc Doc SERVER 1 ACTIVE Doc 4 Doc 7 Doc Doc Doc SERVER 2 Doc 8 ACTIVE Doc 1 Doc 2 Doc Doc Doc REPLICA Doc 4 Doc 1 Doc 8 Doc Doc Doc REPLICA Doc 6 Doc 3 Doc 2 Doc Doc Doc REPLICA Doc 7 Doc 9 Doc 5 Doc Doc Doc SERVER 3 Doc 6 APP SERVER 1 COUCHBASE Client Library CLUSTER MAP COUCHBASE Client Library CLUSTER MAP APP SERVER 2 Doc 9 Monday, February 4, 13

across servers • Each server stores both acYve and replica docs ACTIVE Doc 5 Doc 2 Doc Doc Doc SERVER 1 ACTIVE Doc 4 Doc 7 Doc Doc Doc SERVER 2 Doc 8 ACTIVE Doc 1 Doc 2 Doc Doc Doc REPLICA Doc 4 Doc 1 Doc 8 Doc Doc Doc REPLICA Doc 6 Doc 3 Doc 2 Doc Doc Doc REPLICA Doc 7 Doc 9 Doc 5 Doc Doc Doc SERVER 3 Doc 6 APP SERVER 1 COUCHBASE Client Library CLUSTER MAP COUCHBASE Client Library CLUSTER MAP APP SERVER 2 Doc 9 Monday, February 4, 13

across servers • Each server stores both acYve and replica docs Only one doc ac7ve at a 7me ACTIVE Doc 5 Doc 2 Doc Doc Doc SERVER 1 ACTIVE Doc 4 Doc 7 Doc Doc Doc SERVER 2 Doc 8 ACTIVE Doc 1 Doc 2 Doc Doc Doc REPLICA Doc 4 Doc 1 Doc 8 Doc Doc Doc REPLICA Doc 6 Doc 3 Doc 2 Doc Doc Doc REPLICA Doc 7 Doc 9 Doc 5 Doc Doc Doc SERVER 3 Doc 6 APP SERVER 1 COUCHBASE Client Library CLUSTER MAP COUCHBASE Client Library CLUSTER MAP APP SERVER 2 Doc 9 Monday, February 4, 13

across servers • Each server stores both acYve and replica docs Only one doc ac7ve at a 7me • Client library provides app with simple interface to database ACTIVE Doc 5 Doc 2 Doc Doc Doc SERVER 1 ACTIVE Doc 4 Doc 7 Doc Doc Doc SERVER 2 Doc 8 ACTIVE Doc 1 Doc 2 Doc Doc Doc REPLICA Doc 4 Doc 1 Doc 8 Doc Doc Doc REPLICA Doc 6 Doc 3 Doc 2 Doc Doc Doc REPLICA Doc 7 Doc 9 Doc 5 Doc Doc Doc SERVER 3 Doc 6 APP SERVER 1 COUCHBASE Client Library CLUSTER MAP COUCHBASE Client Library CLUSTER MAP APP SERVER 2 Doc 9 Monday, February 4, 13

across servers • Each server stores both acYve and replica docs Only one doc ac7ve at a 7me • Client library provides app with simple interface to database • Cluster map provides map to which server doc is on ACTIVE Doc 5 Doc 2 Doc Doc Doc SERVER 1 ACTIVE Doc 4 Doc 7 Doc Doc Doc SERVER 2 Doc 8 ACTIVE Doc 1 Doc 2 Doc Doc Doc REPLICA Doc 4 Doc 1 Doc 8 Doc Doc Doc REPLICA Doc 6 Doc 3 Doc 2 Doc Doc Doc REPLICA Doc 7 Doc 9 Doc 5 Doc Doc Doc SERVER 3 Doc 6 APP SERVER 1 COUCHBASE Client Library CLUSTER MAP COUCHBASE Client Library CLUSTER MAP APP SERVER 2 Doc 9 Monday, February 4, 13

across servers • Each server stores both acYve and replica docs Only one doc ac7ve at a 7me • Client library provides app with simple interface to database • Cluster map provides map to which server doc is on App never needs to know ACTIVE Doc 5 Doc 2 Doc Doc Doc SERVER 1 ACTIVE Doc 4 Doc 7 Doc Doc Doc SERVER 2 Doc 8 ACTIVE Doc 1 Doc 2 Doc Doc Doc REPLICA Doc 4 Doc 1 Doc 8 Doc Doc Doc REPLICA Doc 6 Doc 3 Doc 2 Doc Doc Doc REPLICA Doc 7 Doc 9 Doc 5 Doc Doc Doc SERVER 3 Doc 6 APP SERVER 1 COUCHBASE Client Library CLUSTER MAP COUCHBASE Client Library CLUSTER MAP APP SERVER 2 Doc 9 Monday, February 4, 13

across servers • Each server stores both acYve and replica docs Only one doc ac7ve at a 7me • Client library provides app with simple interface to database • Cluster map provides map to which server doc is on App never needs to know • App reads, writes, updates docs ACTIVE Doc 5 Doc 2 Doc Doc Doc SERVER 1 ACTIVE Doc 4 Doc 7 Doc Doc Doc SERVER 2 Doc 8 ACTIVE Doc 1 Doc 2 Doc Doc Doc REPLICA Doc 4 Doc 1 Doc 8 Doc Doc Doc REPLICA Doc 6 Doc 3 Doc 2 Doc Doc Doc REPLICA Doc 7 Doc 9 Doc 5 Doc Doc Doc SERVER 3 Doc 6 APP SERVER 1 COUCHBASE Client Library CLUSTER MAP COUCHBASE Client Library CLUSTER MAP APP SERVER 2 Doc 9 Monday, February 4, 13

across servers • Each server stores both acYve and replica docs Only one doc ac7ve at a 7me • Client library provides app with simple interface to database • Cluster map provides map to which server doc is on App never needs to know • App reads, writes, updates docs • MulYple app servers can access same document at same Yme READ/WRITE/UPDATE ACTIVE Doc 5 Doc 2 Doc Doc Doc SERVER 1 ACTIVE Doc 4 Doc 7 Doc Doc Doc SERVER 2 Doc 8 ACTIVE Doc 1 Doc 2 Doc Doc Doc REPLICA Doc 4 Doc 1 Doc 8 Doc Doc Doc REPLICA Doc 6 Doc 3 Doc 2 Doc Doc Doc REPLICA Doc 7 Doc 9 Doc 5 Doc Doc Doc SERVER 3 Doc 6 APP SERVER 1 COUCHBASE Client Library CLUSTER MAP COUCHBASE Client Library CLUSTER MAP APP SERVER 2 Doc 9 Monday, February 4, 13

• get (key) – Retrieve a document • set (key,
value) – Store a document, overwrites if exists • add (key, value) – Store a document, error/excep7on if exists • replace (key, value) – Store a document, error/excep7on if doesn’t exist • cas (key, value, cas) – Compare and swap, mutate document only if it hasn’t changed while execu7ng this opera7on Basic OperaJons Monday, February 4, 13

Fail Over Node REPLICA ACTIVE Doc 5 Doc 2 Doc
Doc Doc 4 Doc 1 Doc Doc SERVER 1 REPLICA ACTIVE Doc 4 Doc 7 Doc Doc Doc 6 Doc 3 Doc Doc SERVER 2 REPLICA ACTIVE Doc 1 Doc 2 Doc Doc Doc 7 Doc 9 Doc Doc SERVER 3 SERVER 4 SERVER 5 REPLICA ACTIVE REPLICA ACTIVE Doc 9 Doc 8 Doc Doc 6 Doc Doc Doc 5 Doc Doc 2 Doc 8 Doc Doc • App servers accessing docs • Requests to Server 3 fail • Cluster detects server failed Promotes replicas of docs to ac7ve Updates cluster map • Requests for docs now go to appropriate server • Typically rebalance would follow APP SERVER 1 COUCHBASE Client Library CLUSTER MAP COUCHBASE Client Library CLUSTER MAP APP SERVER 2 User Conﬁgured Replica Count = 1 COUCHBASE SERVER CLUSTER Monday, February 4, 13

Doc Doc 4 Doc 1 Doc Doc SERVER 1 REPLICA ACTIVE Doc 4 Doc 7 Doc Doc Doc 6 Doc 3 Doc Doc SERVER 2 REPLICA ACTIVE Doc 1 Doc 2 Doc Doc Doc 7 Doc 9 Doc Doc SERVER 3 SERVER 4 SERVER 5 REPLICA ACTIVE REPLICA ACTIVE Doc 9 Doc 8 Doc Doc 6 Doc Doc Doc 5 Doc Doc 2 Doc 8 Doc Doc • App servers accessing docs • Requests to Server 3 fail APP SERVER 1 COUCHBASE Client Library CLUSTER MAP COUCHBASE Client Library CLUSTER MAP APP SERVER 2 User Conﬁgured Replica Count = 1 COUCHBASE SERVER CLUSTER Monday, February 4, 13

Doc Doc 4 Doc Doc SERVER 1 REPLICA ACTIVE Doc 4 Doc 7 Doc Doc Doc 6 Doc Doc SERVER 2 REPLICA ACTIVE Doc 1 Doc 2 Doc Doc Doc 7 Doc 9 Doc Doc SERVER 3 SERVER 4 SERVER 5 REPLICA ACTIVE REPLICA ACTIVE Doc 9 Doc 8 Doc Doc 6 Doc Doc Doc 5 Doc 2 Doc 8 Doc Doc • App servers accessing docs • Requests to Server 3 fail • Cluster detects server failed Promotes replicas of docs to ac7ve Updates cluster map • Requests for docs now go to appropriate server • Typically rebalance would follow Doc Doc 1 Doc 3 APP SERVER 1 COUCHBASE Client Library CLUSTER MAP COUCHBASE Client Library CLUSTER MAP APP SERVER 2 User Conﬁgured Replica Count = 1 COUCHBASE SERVER CLUSTER Monday, February 4, 13

Indexing and Querying COUCHBASE SERVER CLUSTER ACTIVE Doc 5
Doc 2 Doc Doc Doc SERVER 1 REPLICA Doc 4 Doc 1 Doc 8 Doc Doc Doc APP SERVER 1 COUCHBASE Client Library CLUSTER MAP COUCHBASE Client Library CLUSTER MAP APP SERVER 2 Doc 9 • Indexing work is distributed amongst nodes • Large data set possible • Parallelize the eﬀort • Each node has index for data stored on it • Queries combine the results from required nodes ACTIVE Doc 5 Doc 2 Doc Doc Doc SERVER 2 REPLICA Doc 4 Doc 1 Doc 8 Doc Doc Doc Doc 9 ACTIVE Doc 5 Doc 2 Doc Doc Doc SERVER 3 REPLICA Doc 4 Doc 1 Doc 8 Doc Doc Doc Doc 9 Monday, February 4, 13

Doc 2 Doc Doc Doc SERVER 1 REPLICA Doc 4 Doc 1 Doc 8 Doc Doc Doc APP SERVER 1 COUCHBASE Client Library CLUSTER MAP COUCHBASE Client Library CLUSTER MAP APP SERVER 2 Doc 9 • Indexing work is distributed amongst nodes • Large data set possible • Parallelize the eﬀort • Each node has index for data stored on it • Queries combine the results from required nodes ACTIVE Doc 5 Doc 2 Doc Doc Doc SERVER 2 REPLICA Doc 4 Doc 1 Doc 8 Doc Doc Doc Doc 9 ACTIVE Doc 5 Doc 2 Doc Doc Doc SERVER 3 REPLICA Doc 4 Doc 1 Doc 8 Doc Doc Doc Doc 9 Query Monday, February 4, 13

Doc 2 Doc Doc Doc SERVER 1 REPLICA Doc 4 Doc 1 Doc 8 Doc Doc Doc APP SERVER 1 COUCHBASE Client Library CLUSTER MAP COUCHBASE Client Library CLUSTER MAP APP SERVER 2 Doc 9 • Indexing work is distributed amongst nodes • Large data set possible • Parallelize the eﬀort • Each node has index for data stored on it • Queries combine the results from required nodes ACTIVE Doc 5 Doc 2 Doc Doc Doc SERVER 2 REPLICA Doc 4 Doc 1 Doc 8 Doc Doc Doc Doc 9 ACTIVE Doc 5 Doc 2 Doc Doc Doc SERVER 3 REPLICA Doc 4 Doc 1 Doc 8 Doc Doc Doc Doc 9 Monday, February 4, 13

Cross Data Center ReplicaJon (XDCR) RAM CACHE Doc 1 Doc
2 Doc Doc Doc Doc Doc Doc Doc Doc Doc SERVER 1 Doc 6 DISK RAM CACHE Doc 1 Doc 2 Doc Doc Doc Doc Doc Doc Doc Doc Doc SERVER 2 Doc 6 DISK RAM CACHE Doc 1 Doc 2 Doc Doc Doc Doc Doc Doc Doc Doc Doc SERVER 3 Doc 6 DISK Couchbase Cluster West Coast Data Center RAM CACHE Doc 1 Doc 2 Doc Doc Doc Doc Doc Doc Doc Doc Doc SERVER 1 Doc 6 DISK RAM CACHE Doc 1 Doc 2 Doc Doc Doc Doc Doc Doc Doc Doc Doc SERVER 2 Doc 6 DISK RAM CACHE Doc 1 Doc 2 Doc Doc Doc Doc Doc Doc Doc Doc Doc SERVER 3 Doc 6 DISK Couchbase Cluster East Coast Data Center Monday, February 4, 13

GeoCouch R-‐Tree Index • Optimized for bulk loading of large
data sets • Simpliﬁed query model (bounding box, nearest neighbor) Monday, February 4, 13

ElasJc Search Adaptor • Elastic Search is good for ad-hoc
queries and faceted browsing • Our adapter is aware of changing Couchbase topology • Indexed by Elastic Search after stored to disk in Couchbase ElasYcSearch Monday, February 4, 13

I’m Excited to See What You Build, Q & A
Contact me on TwiDer @tgrall Contact me by Email [email protected] Learn More About Design PaDerns CouchbaseModels.com Seeng up for Ruby on Rails CouchbaseOnRails.com Couchbase Docs www.couchbase.com/docs/index-‐full.html Couchbase Forums www.couchbase.com/forums IRC #couchbase #libcouchbase Monday, February 4, 13

Monitoring Monday, February 4, 13

FOSDEM 2013 : Getting Started with Couchhbase Server 2.0

FOSDEM 2013 : Getting Started with Couchhbase Server 2.0

More Decks by Tugdual Grall

Other Decks in Technology

Featured

Transcript