MongoSP: MongoDB Schema Design

Greg Studer's presentation at Mongo Sao Paulo

Transcript

1. Greg Studer MongoDB Schema Design MongoSP2011
2. greg@10gen.com
3. Topics Introduction • Working with documents • Evolving a schema • Queries and indexes • Rich documents Common patterns • Single table inheritance • One-to-Many & Many-to-Many • Trees • Queues
4. Ways to model data: http://www.flickr.com/photos/42304632@N00/493639870/
5. Normalized
6. Denormalized
7. Terminology RDBMS MongoDB Table Collection Row(s) JSON Document Index Index Join Embedding & Linking
8. Schema-design Criteria How can we manipulate this data? • Dynamic Queries • Secondary Indexes • Atomic Updates • Map Reduce Considerations • No Joins • Document writes are atomic Access Patterns? • Read / Write Ratio • Types of updates • Types of queries • Data life-cycle
9. Here’s an example with a book:
10. A simple start: Map the documents to your application. post = { author: “Hergé”, date: new Date(), text: “Destination Moon”, tags: [“comic”, “adventure”] } > db.post.save(post)
11. > db.posts.find() { _id: ObjectId("4c4ba5c0672c685e5e8aabf3"), author: "Hergé", date: "Sat Jul 24 2010 19:47:11 GMT-0700 (PDT)", text: "Destination Moon", tags: [ "comic", "adventure" ] } Notes: • _id must be unique, but can be anything you’d like • Default BSON ObjectId if one is not supplied Find the document
12. Secondary index on “author” > db.posts.ensureIndex({ author : 1 }) > db.posts.find({ author : 'Hergé' }) { _id : ObjectId("4c4ba5c0672c685e5e8aabf3"), date : "Sat Jul 24 2010 19:47:11 GMT-0700 (PDT)", author: "Hergé", ... } Add an index, find via index
13. Examine the query plan > db.blogs.find({ author : 'Hergé' }).explain() { "cursor" : "BtreeCursor author_1", "nscanned" : 1, "nscannedObjects" : 1, "n" : 1, "millis" : 5, "indexBounds" : { "author" : [ [ "Hergé", "Hergé" ] ] } }
14. Multi-key indexes // Build an index on the ‘tags’ array > db.posts.ensureIndex({ tags : 1 }) // find posts with a specific tag // (This will use an index!) > db.posts.find({ tags : ‘comic’ })
15. Query operators Conditional operators: $ne, $in, $nin, $mod, $all, $size, $exists, $type, .. $lt, $lte, $gt, $gte, $ne Update operators: $set, $inc, $push, $pop, $pull, $pushAll, $pullAll
16. Extending the Schema new_comment = { author : “Kyle”, date : new Date(), text : “great book”, votes : 5 } > db.posts.update( { text : “Destination Moon” }, { ‘$push’ : {comments: new_comment } , ‘$inc’ : {comments_count: 1 } } )
17. { _id : ObjectId("4c4ba5c0672c685e5e8aabf3"), author : "Hergé", date : "Sat Jul 24 2010 19:47:11 GMT-0700 (PDT)", text : "Destination Moon", tags : [ "comic", "adventure" ], comments : [ { author : "Kyle", date : "Sat Jul 24 2010 20:51:03 GMT-0700 (PDT)", text : "great book", votes : 5 } ], comments_count: 1 } Extending the Schema
18. // create index on nested documents: > db.posts.ensureIndex({ "comments.author" : 1 }) > db.posts.find({ "comments.author" : ”Kyle” }) The ‘dot’ operator
19. // create index on nested documents: > db.posts.ensureIndex({ "comments.author" : 1 }) > db.posts.find({ "comments.author" : ”Kyle” }) // create index comment votes: > db.posts.ensureIndex({ "comments.votes" : 1 }) // find all posts with any comments with more than // 50 votes > db.posts.find({ "comments.votes" : { $gt : 50 } }) The ‘dot’ operator
20. Rich Documents
21. Rich Documents https://secure.wikimedia.org/wikipedia/en/wiki/Japanese_rock_garden 663highland, Creative Commons 2.5
23. Rich Documents { _id : ObjectId("4c4ba5c0672c685e5e8aabf3"), line_items : [ { sku: ‘tt-123’, name : ‘Tintin a la Lune’ }, { ski: ‘tt-457’, name : ‘Coke en stock’ } ], address : { name : ‘Banker’, street : ‘111 Main’, zip : 10010 }, payment : { cc : 4567, exp : Date(2011, 7, 7) }, subtotal : 2355 }
24. Inheritance
25. Single Table Inheritance - RDBMS shapes_table id type area radiu s d length width 1 circle 3.14 1 2 square 4 2 3 rect 10 5 2
26. Single Table Inheritance - MongoDB > db.shapes.find() { _id : "1", type : "circle", area : 3.14, radius : 1} { _id : "2", type : "square", area : 4, d : 2} { _id : "3", type : "rect", area : 10, length : 5, width : 2}
27. Single Table Inheritance - MongoDB > db.shapes.find() { _id : "1", type : "circle", area : 3.14, radius : 1 } { _id : "2", type : "square", area : 4, d : 2 } { _id : "3", type : "rect", area : 10, length : 5, width : 2 } // find shapes where radius > 0 > db.shapes.find({ radius : { $gt : 0 } })
28. Single Table Inheritance - MongoDB > db.shapes.find() { _id : "1", type : "circle", area : 3.14, radius : 1 } { _id : "2", type : "square", area : 4, d : 2 } { _id : "3", type : "rect", area : 10, length : 5, width : 2 } // find shapes where radius > 0 > db.shapes.find({ radius : { $gt : 0 } }) // create sparse index > db.shapes.ensureIndex({ radius : 1 }, { sparse : true })
29. One to Many One to Many relationships can specify
30. One to Many - Embedded Array / Array Keys - $slice operator to return subset of array - some queries hard e.g. find latest comments across all documents
31. One to Many - Embedded Array / Array Keys - $slice operator to return subset of array - some queries hard e.g. find latest comments across all documents - Embedded tree - Single document - Natural - Hard to query
32. One to Many - Embedded Array / Array Keys - $slice operator to return subset of array - some queries hard e.g. find latest comments across all documents - Embedded tree - Single document - Natural - Hard to query - Normalized (2 collections) - most flexible - more queries
33. One to Many - patterns - Embedded Array / Array Keys - Embedded Array / Array Keys - Embedded tree - Normalized
34. { _id : ObjectId("4c4ba5c0672c685e5e8aabf3"), author : "Hergé", date : "Sat Jul 24 2010 19:47:11 GMT-0700 (PDT)", text : "Destination Moon", tags : [ "comic", "adventure" ], comments : [ { author : "Kyle", date : "Sat Jul 24 2010 20:51:03 GMT-0700 (PDT)", text : "great book", votes : 5 } ], comments_count: 1 }
35. { _id : ObjectId("4c4ba5c0672c685e5e8aabf3"), author : "Hergé", date : "Sat Jul 24 2010 19:47:11 GMT-0700 (PDT)", text : "Destination Moon", tags : [ "comic", "adventure" ] } { book_id : ObjectId("4c4ba5c0672c685e5e8aabf3"), author : "Kyle", date : "Sat Jul 24 2010 20:51:03 GMT-0700 (PDT)", text : "great book", votes : 5 }
36. Referencing vs. Embedding - Embed when the ‘many’ objects always appear with their parent. - Reference when you need more flexibility.
37. Many - Many Example: - Product can be in many categories - Category can have many products
38. products: { _id : ObjectId("4c4ca23933fb5941681b912e"), name : "Destination Moon", category_ids : [ ObjectId("4c4ca25433fb5941681b912f"), ObjectId("4c4ca25433fb5941681b92af”] } Many - Many
39. products: { _id : ObjectId("4c4ca23933fb5941681b912e"), name : "Destination Moon", category_ids : [ ObjectId("4c4ca25433fb5941681b912f"), ObjectId("4c4ca25433fb5941681b92af”] } categories: { _id : ObjectId("4c4ca25433fb5941681b912f"), name : "adventure", product_ids : [ ObjectId("4c4ca23933fb5941681b912e"), ObjectId("4c4ca30433fb5941681b9130"), ObjectId("4c4ca30433fb5941681b913a"] } Many - Many
40. products: { _id : ObjectId("4c4ca23933fb5941681b912e"), name : "Destination Moon", category_ids : [ ObjectId("4c4ca25433fb5941681b912f"), ObjectId("4c4ca25433fb5941681b92af”] } categories: { _id : ObjectId("4c4ca25433fb5941681b912f"), name : "adventure", product_ids : [ ObjectId("4c4ca23933fb5941681b912e"), ObjectId("4c4ca30433fb5941681b9130"), ObjectId("4c4ca30433fb5941681b913a"] } // Multi-key index on array fields > db.products.ensureIndex({ category_ids : 1 }); > db.categories.ensureIndex({ product_ids : 1 }); Many - Many
41. products: { _id : ObjectId("4c4ca23933fb5941681b912e"), name : "Destination Moon", category_ids : [ ObjectId("4c4ca25433fb5941681b912f"), ObjectId("4c4ca25433fb5941681b92af”] } categories: { _id : ObjectId("4c4ca25433fb5941681b912f"), name : "adventure" } Better alternative
42. products: { _id : ObjectId("4c4ca23933fb5941681b912e"), name : "Destination Moon", category_ids : [ ObjectId("4c4ca25433fb5941681b912f"), ObjectId("4c4ca25433fb5941681b92af”] } categories: { _id : ObjectId("4c4ca25433fb5941681b912f"), name : "adventure" } // All products for a given category > db.products.find({ category_ids : ObjectId("4c4ca25433fb5941681b912f") }) Alternative
43. products: { _id : ObjectId("4c4ca23933fb5941681b912e"), name : "Destination Moon", category_ids : [ ObjectId("4c4ca25433fb5941681b912f"), ObjectId("4c4ca25433fb5941681b92af”] } categories: { _id : ObjectId("4c4ca25433fb5941681b912f"), name : "adventure" } // All products for a given category > db.products.find({ category_ids : ObjectId("4c4ca25433fb5941681b912f") }) // All categories for a given product product = db.products.find({ _id : some_id }) > db.categories.find({ _id : { $in : product.category_ids } }) Alternative
44. Trees Full Tree in Document { comments: [ { author: “Kyle”, text: “...”, replies: [ { author: “Fred”, text: “...”, replies: [ ] } ] } ] } Pros: Single Document, Performance, Intuitive Cons: Hard to search, Partial Results, 4MB limit
45. Trees Parent Links - Each node is stored as a document - Contains the id of the parent Child Links - Each node contains the id’s of the children - Can support graphs (multiple parents / child)
46. Array of Ancestors - Store all Ancestors of a node { _id : "a" } { _id : "b", ancestors : [ "a" ], parent : "a" } { _id : "c", ancestors : [ "a", "b" ], parent : "b" } { _id : "d", ancestors : [ "a", "b" ], parent : "b" } { _id : "e", ancestors : [ "a" ], parent : "a" } { _id : "f", ancestors : [ "a", "e" ], parent : "e" }
47. Array of Ancestors - Store all Ancestors of a node { _id : "a" } { _id : "b", ancestors : [ "a" ], parent : "a" } { _id : "c", ancestors : [ "a", "b" ], parent : "b" } { _id : "d", ancestors : [ "a", "b" ], parent : "b" } { _id : "e", ancestors : [ "a" ], parent : "a" } { _id : "f", ancestors : [ "a", "e" ], parent : "e" } //find all descendants of b: > db.tree2.find({ ancestors : ‘b’ }) //find all direct descendants of b: > db.tree2.find({ parent : ‘b’ })
48. Array of Ancestors - Store all Ancestors of a node { _id : "a" } { _id : "b", ancestors : [ "a" ], parent : "a" } { _id : "c", ancestors : [ "a", "b" ], parent : "b" } { _id : "d", ancestors : [ "a", "b" ], parent : "b" } { _id : "e", ancestors : [ "a" ], parent : "a" } { _id : "f", ancestors : [ "a", "e" ], parent : "e" } //find all descendants of b: > db.tree2.find({ ancestors : ‘b’ }) //find all direct descendants of b: > db.tree2.find({ parent : ‘b’ }) //find all ancestors of f:
49. Queue Requirements - See jobs waiting, jobs in progress - Ensure that each job is started once and only once { inprogress : false, priority : 1, message : “Rich documents FTW!”, ... } // find highest priority job and mark as in-progress job = db.jobs.findAndModify({ query : { inprogress : false }, sort : { priority: -1 }, update : { $set : { inprogress : true, started : new Date() } } })
50. Summary Schema design is different in MongoDB Basic data design principles remain the same Focus on how the app manipulates data Rapidly evolve schema to meet your requirements Enjoy your new freedom, use it wisely :-)
51. @mongodb conferences, appearances, and meetups http://www.10gen.com/events Facebook | Twitter | LinkedIn http://bit.ly/mongofb http://linkd.in/joinmongo download at mongodb.org Looking for translators! http://pt.wiki.mongodb.org/display/DOCS/Home
52. http://manning. com/banker
53. @mongodb conferences, appearances, and meetups http://www.10gen.com/events Facebook | Twitter | LinkedIn http://bit.ly/mongofb http://linkd.in/joinmongo download at mongodb.org We’re Hiring ! info@10gen.com
54. Rich Documents