Upgrade to Pro — share decks privately, control downloads, hide ads and more …

MongoNYC 2012: Sharding Overview (Introductory ...

mongodb
June 01, 2012
1.1k

MongoNYC 2012: Sharding Overview (Introductory Session)

MongoNYC 2012: Sharding Overview (Introductory Session), Tyler Brock, 10gen.

mongodb

June 01, 2012
Tweet

Transcript

  1. Mongod started with --configsvr option Must have 3 (or 1

    in development) Data is commited using 2 phase commit config Config Server
  2. mongos Acts just like shard router / proxy One or

    as many as you want Light weight -- can run on App servers Caches meta-data from config servers mongos
  3. { name: “Joe”, email: “Joe@fake.com”, }, { name: “Bob”, email:

    “bob@fake.com”, }, { name: “Tyler”, email: “tyler@fake.com”, } Keys test.users
  4. > db.runCommand({ }) { name: “Joe”, email: “Joe@fake.com”, }, {

    name: “Bob”, email: “bob@fake.com”, }, { name: “Tyler”, email: “tyler@fake.com”, } shardcollection: “test.users”, Keys key: { email: 1 } test.users
  5. { name: “Joe”, email: “Joe@fake.com”, }, { name: “Bob”, email:

    “bob@fake.com”, }, { name: “Tyler”, email: “tyler@fake.com”, } shardcollection: “test.users”, Keys key: { email: 1 } test.users
  6. { name: “Joe”, email: “Joe@fake.com”, }, { name: “Bob”, email:

    “bob@fake.com”, }, { name: “Tyler”, email: “tyler@fake.com”, } Keys key: { email: 1 } test.users
  7. Chunks -∞ +∞ Split! This is a chunk This is

    a chunk joe@fake.com moe@fake.com tyler@fake.com
  8. Splitting config config config mongos Shard 1 Shard 2 Shard

    3 Shard 4 Split this big chunk into 2 chunks
  9. Balancing config config config mongos Shard 1 Shard 2 Shard

    3 Shard 4 Shard1, move a chunk to Shard2
  10. Balancing config config config mongos Shard 1 Shard 2 Shard

    3 Shard 4 Shard1, move another chunk to Shard3
  11. Balancing config config config mongos Shard 1 Shard 2 Shard

    3 Shard 4 Shard1, move another chunk to Shard4
  12. Routed Request 1 2 mongos shard shard shard 1. Query

    arrives at Mongos 2. Mongos routes query to a single shard
  13. Routed Request 1 2 3 mongos shard shard shard 1.

    Query arrives at Mongos 2. Mongos routes query to a single shard 3. Shard returns results of query
  14. Routed Request 1 2 3 4 mongos shard shard shard

    1. Query arrives at Mongos 2. Mongos routes query to a single shard 3. Shard returns results of query 4. Results returned to client
  15. Scatter Gather Request 1 1. Query arrives at Mongos 2

    2 2 shard shard shard mongos 2. Mongos broadcasts query to all shards
  16. Scatter Gather Request 1 1. Query arrives at Mongos 2

    2 2 3 3 3 shard shard shard mongos 2. Mongos broadcasts query to all shards 3. Each shard returns results for query
  17. Scatter Gather Request 1 4 1. Query arrives at Mongos

    2 2 2 3 3 3 shard shard shard mongos 2. Mongos broadcasts query to all shards 3. Each shard returns results for query 4. Results combined and returned to client
  18. mongos Distributed Merge Sort Req. 1 2 2 2 shard

    shard shard 1. Query arrives at Mongos 2. Mongos broadcasts query to all shards
  19. mongos Distributed Merge Sort Req. 1 2 2 2 shard

    shard shard 3 3 3 1. Query arrives at Mongos 2. Mongos broadcasts query to all shards 3. Each shard locally sorts results
  20. mongos Distributed Merge Sort Req. 1 2 2 2 4

    4 4 shard shard shard 3 3 3 1. Query arrives at Mongos 2. Mongos broadcasts query to all shards 3. Each shard locally sorts results 4. Results returned to mongos
  21. mongos Distributed Merge Sort Req. 1 5 2 2 2

    4 4 4 shard shard shard 3 3 3 1. Query arrives at Mongos 2. Mongos broadcasts query to all shards 3. Each shard locally sorts results 4. Results returned to mongos 5. Mongos merges sorted results
  22. mongos Distributed Merge Sort Req. 1 6 5 2 2

    2 4 4 4 shard shard shard 3 3 3 1. Query arrives at Mongos 2. Mongos broadcasts query to all shards 3. Each shard locally sorts results 4. Results returned to mongos 5. Mongos merges sorted results 6. Combined results returned to client
  23. Queries By Shard Key Routed db.users.find({email: “bob@10gen.com”}) Sorted by shard

    key Routed in order db.users.find().sort({email:-1}) Find by non shard key Scatter Gather db.users.find({state:”NY”}) Sorted by non shard key Distributed merge sort db.users.find().sort({state:1})
  24. Writes Inserts Requires shard key db.users.insert({ name: “Bob”, email: “Bob@fake.com”})

    Removes Routed db.users.delete({ email: “bob@fake.com”}) Removes Scattered db.users.delete({name: “Bob”}) Updates Routed db.users.update( {email: “bob@10gen.com”}, {$set: { state: “NY”}}) Updates Scattered db.users.update( {state: “CA”}, {$set:{ state: “NY”}} )
  25. { node: "ny153.example.com", application: "apache", time: "2011-01-02T21:21:56Z", level: "ERROR", msg:

    "something is broken" } Bad { time : 1 } Writes should be distributed
  26. { node: "ny153.example.com", application: "apache", time: "2011-01-02T21:21:56Z", level: "ERROR", msg:

    "something is broken" } Bad { time : 1 } Better {node:1, application:1, time:1} Writes should be distributed
  27. Bad {msg: 1, node: 1} { node: "ny153.example.com", application: "apache",

    time: "2011-01-02T21:21:56Z", level: "ERROR", msg: "something is broken” } Queries should be routed to one shard
  28. Better {node: 1, time: 1} Bad {msg: 1, node: 1}

    { node: "ny153.example.com", application: "apache", time: "2011-01-02T21:21:56Z", level: "ERROR", msg: "something is broken” } Queries should be routed to one shard
  29. Bad {node: 1} { node: "ny153.example.com", application: "apache", time: "2011-01-02T21:21:56Z",

    level: "ERROR", msg: "something is broken" } Chunks should be able to split
  30. Better {node:1, time:1} Bad {node: 1} { node: "ny153.example.com", application:

    "apache", time: "2011-01-02T21:21:56Z", level: "ERROR", msg: "something is broken" } Chunks should be able to split
  31. mongod mongod mongod Bring up mongods or Replica Sets mongod

    mongod mongod mongod mongod mongod RS RS RS mongod --shardsvr mongod --replSet --shardsvr
  32. config mongod mongod mongod mongod mongod mongod mongod mongod mongod

    RS RS RS Bring up Config Servers config config mongod --configsvr
  33. config mongod mongod mongod mongod mongod mongod mongod mongod mongod

    RS RS RS Bring up Mongos config config mongos mongos --configdb <list of configdb uris>
  34. > use admin > db.runCommand({"addShard": <shard uri>}) Connect to Mongos

    + Add Shards Enable Sharding > db.runCommand( { enablesharding : "<dbname>" } ); > db.runCommand( { shardcollection : "<namespace>", key : <key> }); Shard a Collection