Introduction to NoSQL Solutions with Couchbase and .NET Core

1. Baris Ceviz | Software Developer @Trendyol Twitter – Github – LinkedIn : @PeaceCwz Mail: baris.ceviz@trendyol.com Introduction to NoSQL Solutions with Couchbase and .NET Core
2. AGENDA 01/ 02/ 03/ 04/ What’s Couchbase Clusters and nodes Key-Value Database Couchbase .NET SDK
3. A Cluster Application SDK App Server Application SDK App Server 3 CLUSTER of COUCHBASE NODES
4. A Cluster and Sync Gateway Application SDK App Server Application SDK App Server Application SDK App Server Application SDK App Server Sync Gateway Application SDK App Server Application SDK App Server Sync Gateway 4 CLUSTER of COUCHBASE NODES
5. App Server SDK Application Application SDK App Server Application SDK App Server Sync Gateway Sync Gateway Application SDK App Server Application SDK App Server Application SDK App Server 5 A Cluster and Sync Gateway CLUSTER of COUCHBASE NODES
6. Application SDK App Server Application SDK App Server Application SDK App Server Sync Gateway Sync Gateway Application SDK App Server Application SDK App Server Application SDK App Server 6 A Cluster and Sync Gateway Kubernetes CLUSTER of COUCHBASE NODES
7. Application SDK App Server Application SDK App Server Application SDK App Server Sync Gateway Sync Gateway Application SDK App Server Application SDK App Server Application SDK App Server A Cluster and Sync Gateway CLUSTER of COUCHBASE NODES 7
8. Kubernetes Autonomous Operator A Cluster and Sync Gateway and Kubernetes Autonomous Application SDK App Server Application SDK App Server Application SDK App Server Application SDK App Server Sync Gateway Application SDK App Server Application SDK App Server Sync Gateway 8 CLUSTER of COUCHBASE NODES
9. Nodes COUCHBASE SERVER NODE Analytics Query Indexing Full Text Eventing Key Value Cluster Manager COUCHBASE SERVER NODE Analytics Query Indexing Full Text Eventing Key Value Cluster Manager COUCHBASE SERVER NODE Analytics Query Indexing Full Text Eventing Key Value Cluster Manager COUCHBASE SERVER NODE Analytics Query Indexing Full Text Eventing Key Value Cluster Manager
10. Nodes COUCHBASE SERVER NODE Analytics Query Indexing Full Text Eventing Key Value Cluster Manager COUCHBASE SERVER NODE Analytics Query Indexing Full Text Eventing Key Value Cluster Manager COUCHBASE SERVER NODE Analytics Query Indexing Full Text Eventing Key Value Cluster Manager COUCHBASE SERVER NODE Analytics Query Indexing Full Text Eventing Key Value Cluster Manager
11. Mix & match subsets of services (Multidimensional Scaling) Analytics Query Indexing Full Text Eventing Key Value Cluster Manager Analytics Query Indexing Full Text Eventing Key Value Cluster Manager Analytics Query Indexing Full Text Eventing Key Value Cluster Manager Analytics Query Indexing Full Text Eventing Key Value Cluster Manager COUCHBASE SERVER NODE COUCHBASE SERVER NODE COUCHBASE SERVER NODE COUCHBASE SERVER NODE
12. Mix & match subsets of services (Multidimensional Scaling) Analytics Query Indexing Full Text Eventing Key Value Cluster Manager Analytics Query Indexing Full Text Eventing Key Value Cluster Manager Analytics Query Indexing Full Text Eventing Key Value Cluster Manager Analytics Query Indexing Full Text Eventing Key Value Cluster Manager COUCHBASE SERVER NODE COUCHBASE SERVER NODE COUCHBASE SERVER NODE COUCHBASE SERVER NODE
13. Cluster Manager Cluster Manager Cluster Manager Cluster Manager Cluster Manager Cluster Manager Cluster Manager Cluster Manager Analytics Search Key Value Query Indexing Key Value Key Value Mix & match subsets of services (Multidimensional Scaling) Analytics
14. The Cluster Manager The control plane of the cluster: • Admin APIs & User Interface • Authentication / RBAC • Master Services • Rebalance and Auto-Failover • Automatic shard placement • Bucket Services • Per-node Services • Generic facilities per-node-&-bucket services Generic distributed facilities Generic local facilities Logging and other services Distributed node discovery Master Services • Cluster-wide operations • Rebalance • Auto-failover • Automatic shard placement Global config (gossip replication) Local config Store Per-node Services • Heartbeats • Babysitter Cluster Manager Bucket services • Replica stream management • Stats collection & monitoring 14 Admin Console & REST API Authentication
15. Automatic Sharding 26
16. KEY VALUE DATA SERVICE App Server SDK Application HASH("user_profile-1234") ==> 27 VB Node 0 A 1 B 2 C … 27 C 1023 D { … } Automatic Sharding and Rebalancing NODE A NODE B NODE C NODE D CLUSTER MANAGER 16 user_profile-1234
17. App Server SDK Application VB Node 0 A 1 B 2 C … 27 C 1023 D Automatic Sharding and Rebalancing CLUSTER MANAGER KEY VALUE DATA SERVICE NODE A NODE B NODE C NODE D CLUSTER MANAGER 17 user_profile-1234 { … }
18. App Server SDK Application VB Node 0 A 1 B 2 C … 27 C 1023 D Automatic Sharding and Rebalancing KEY VALUE DATA SERVICE NODE A NODE B NODE C NODE D NODE E CLUSTER MANAGER 18 user_profile-1234 { … }
19. C L DATA SERVICE App Server SDK Application VB Nod e 0 A 1 B 2 C … 27 C 1023 D { … } NODE A NODE B NODE C NODE D VB Nod e 0 A 1 B 2 C … 27 E 1023 D NODE E Automatic Sharding and Rebalancing CLUSTER MANAGER KEY VALUE DATA SERVICE HASH(“user_profile-1234”) ==> 27 user_profile-1234 19
20. Efficient Data Propagation
21. • High Performance / In-Memory • De-Duplication • Ordered, predictable and consistent • Restartable 21 The Database Change Protocol
22. • High Performance / In-Memory • De-Duplication • Ordered, predictable and consistent • Restartable Ke 22 y Value D ata Service The Database Change Protocol
23. • High Performance / In-Memory • De-Duplication • Ordered, predictable and consistent • Restartable Ke y Value Data Service The Database Change Protocol DCP 23
24. • High Performance / In-Memory • De-Duplication • Ordered, predictable and consistent • Restartable Ke y Value Data Service The Database Change Protocol DCP 24
25. • High Performance / In-Memory • De-Duplication • Ordered, predictable and consistent • Restartable Ke y Value D ata Service The Database Change Protocol DCP DCP 25
26. • High Performance / In-Memory • De-Duplication • Ordered, predictable and consistent • Restartable The Database Change Protocol DCP Key Value Data Service DCP 26
27. • High Performance / In-Memory • De-Duplication • Ordered, predictable and consistent • Restartable Ke y Value D ata Service The Database Change Protocol OTHER FEATURES & SERVICES • XDCR & Backup • Sync Gateway DCP DCP 27
28. • High Performance / In-Memory • De-Duplication • Ordered, predictable and consistent • Restartable Key Value Data Service DCP The Database Change Protocol INTEGRATIONS • hadoop, kafka & more OTHER FEATURES & SERVICES • XDCR & Backup • Sync Gateway DCP 28
29. Key Value … Networking I/O & handler threads VBucket Manager Item Pager Expiry Pager Checkpoint Manager Cache . . Partition Hash Tables Fetcher Scheduler Reader IO Writer IO Generic Tasks Flusher The Key Value Data Service Authentication 29 Analytics Query Indexing Full Text Eventing Key Value Cluster Manager
30. Key Value … Networking I/O & handler threads VBucket Manager Item Pager Expiry Pager Checkpoint Manager Cache . . H Partition ash Tables Fetcher Scheduler Reader IO Writer IO Generic Tasks Flusher The Key Value Data Service The KV "Data Plane" of a Couchbase cluster: • High performance memcached DNA • With data partitioning ("vbuckets") • Sub-Document operations • Authentication / RBAC And also... • Asynch task workers and scheduler • For I/O: persistence, backfill • For expiry & cleanup • For replication • The source of Database Change Protocol • A consumer of Database Change Protocol 30 Authentication
31. Multiple Clusters in Different Regions
32. zooming out 32 ASIA CLUSTER US WEST CLUSTER US EAST CLUSTER
33. zooming out 33 ASIA CLUSTER US WEST CLUSTER US EAST CLUSTER XDCR XDCR XDCR
34. Couchbase SDK with .NET Core
35. e 2017. All rights reserved. May the sorce be with you