MongoDB UK 2012: Overview of Sharding

Daniel Roberts, 10gen
Primer on horizontal scaling with MongoDB. Introduces the core concepts behind storing and querying large datasets in addition to presenting the architecture, mechanics, and configuration of a sharded cluster.

Transcript

1. Overview Of Sharding
2. Solution(Architect Based(in(London http://www.10gen.com/ @dmroberts daniel.roberts@10gen.com sdf
3. Agenda Architecture How it Works Balancing the cluster Routing requests • • • • 3
4. http://community.qlikview.com/cfs- filesystemfile.ashx/__key/CommunityServer.Blogs.Components.WeblogFiles/theqlikviewblog/Cutting-Grass- with-Scissors-_2D00_-2.jpg
5. http://www.bitquill.net/blog/wp-content/uploads/2008/07/pack_of_harvesters.jpg
6. MongoDB Scaling - Single Node write read node_a1
7. Read scaling - add Replicas write read node_b1 node_a1
8. Read scaling - add Replicas write read node_c1 node_b1 node_a1
9. Write scaling - Sharding write read shard1 node_c1 node_b1 node_a1
10. Write scaling - add Shards write read shard1 node_c1 node_b1 node_a1 shard2 node_c2 node_b2 node_a2
11. Write scaling - add Shards write read shard1 node_c1 node_b1 node_a1 shard2 node_c2 node_b2 node_a2 shard3 node_c3 node_b3 node_a3
12. MongoDB Sharding Automatic partitioning and management Range based Convert to sharded system with no downtime Fully consistent • • • •
13. Range Based Partitioning >"db.posts.save("{age:40}") -∞ +∞ -∞ 40 41 +∞ Data in inserted Ranges are split into more “chunks” • •
14. How MongoDB Sharding works >"db.posts.save("{age:40}") >"db.posts.save("{age:50}") -∞ +∞ -∞ 40 41 +∞ 41 50 51 +∞ More Data in inserted Ranges are split into more“chunks” • •
15. How MongoDB Sharding works >"db.posts.save("{age:40}") >"db.posts.save("{age:50}") >"db.posts.save("{age:60}") -∞ +∞ -∞ 40 41 +∞ 41 50 51 +∞ 61 +∞ 51 60
16. -∞ +∞ 41 +∞ 51 +∞ How MongoDB Sharding works >"db.posts.save("{age:40}") >"db.posts.save("{age:50}") >"db.posts.save("{age:60}") -∞ 40 41 50 61 +∞ 51 60 shard1
17. How MongoDB Sharding works >"db.runCommand("{"addshard":""shard2""}"); -∞ 40 41 50 61 +∞ 51 60 shard1 shard2 >"db.runCommand("{"addshard":""shard3""}"); shard3
18. SHARDING ARCHITECTURE
19. Architecture
20. mongos Shard Router Acts just like a MongoD 1 or as many as you want Can run on App Servers Caches meta-data from config servers • • • • •
21. Config Server 3 of them Changes use 2 phase commit If any are down, meta data goes read only System is online as long as 1/3 is up • • • •
22. HOW IT WORKS
23. Keys { name: “Jared”, email: “jsr@abc.com”, } { name: “Scott”, email: “scott@gmail.com”, } { name: “Dan”, email: “dan@yahoo.com”, } > db.runCommand( { shardcollection: “test.users”, key: { email: 1 }} )
24. Chunks -∞ +∞
25. Chunks -∞ +∞ dan@yahoo.com jsr@abc.com scott@gmail.com
26. Chunks -∞ +∞ dan@yahoo.com jsr@abc.com scott@gmail.com Split!
27. Chunks -∞ +∞ dan@yahoo.com jsr@abc.com scott@gmail.com Split! This is a chunk This is a chunk
28. Chunks -∞ +∞ dan@yahoo.com jsr@abc.com scott@gmail.com
29. Chunks -∞ +∞ dan@yahoo.com jsr@abc.com scott@gmail.com
30. Chunks -∞ +∞ dan@yahoo.com jsr@abc.com scott@gmail.com Split!
31. Chunks Min Key Max Key Shard -∞ adam@yahoo.com 1 adam@yahoo.com jared@abc.com 1 jared@abc.com scott@gmail.com 1 scott@gmail.com +∞ 1 Stored in the config servers Cached in MongoS Used to route requests and keep cluster balanced • • •
32. Balancing Shard 1 Shard 2 Shard 3 Shard 4 5 9 1 6 10 2 7 11 3 8 12 4 17 21 13 18 22 14 19 23 15 20 24 16 29 33 25 30 34 26 31 35 27 32 36 28 41 45 37 42 46 38 43 47 39 44 48 40 mongos balancer config config config Chunks!
33. Balancing mongos balancer config config config Shard 1 Shard 2 Shard 3 Shard 4 5 9 1 6 10 2 7 11 3 8 12 4 21 22 23 24 33 34 35 36 45 46 47 48 Imbalance Imbalance
34. Balancing mongos balancer Move chunk 1 to Shard 2 config config config Shard 1 Shard 2 Shard 3 Shard 4 5 9 1 6 10 2 7 11 3 8 12 4 21 22 23 24 33 34 35 36 45 46 47 48
35. Balancing mongos balancer config config config Shard 1 Shard 2 Shard 3 Shard 4 5 9 6 10 2 7 11