MongoDB 3.2

What's hot and new in the latest MongoDB version: Document validation, $lookup, BI, storage engines, replicated config servers, and partial indexes.

Transcript

1. MongoDB 3.2 FOR GIANT IDEAS Philipp Krenn @xeraa
2. ViennaDB Papers We Love Vienna
3. Electronic Data Interchange (EDI)
4. Changelog https://docs.mongodb.org/manual/release-notes/3.2/ MongoDB 3.0: 2015/03 MongoDB 3.2: 2015/11
5. +0.2 == Major release ! changes allowed .<odd> development .<even> production
6. Fancy Document Validation Lookup aka "JOIN" Business Intelligence Storage Engines
7. Internal Config Servers as Replica Sets readConcern Partial Indexes
8. Document Validation
9. Schemaless for the win
10. Schema in your app No duplication
11. Schemaless is a lie
12. Indexes
13. Multiple apps Schema duplication
14. Flexibility + Safety = Document Validation
15. Demo $ mkdir test $ mongod --dbpath test/ --port 27001 --logpath test.log $ mongo --port 27001 > db.version() > db.contact.insert({ phone: "1234", email: "philipp@test.com", status: "ok" })
16. Validation > db.createCollection("contact", { validator: { $and: [ { phone: { $type: "string" } }, { email: { $regex: /@test\.com$/ } }, { status: { $in: [ "ok", "incomplete" ] } } ] } } )
17. Change to contacts and insert validation rule > db.getCollectionInfos() > db.contacts.insert({ phone: "1234", email: "philipp@test.com", status: "ok" }) > db.contacts.insert({ phone: "1234" }) > db.contacts.find()
18. Update > db.runCommand({ collMod: "contacts", validator: { $or: [ { phone: { $type: "string" } }, { email: { $regex: /@test\.com$/ } }, { status: { $in: [ "ok", "incomplete" ] } } ] }, validationLevel: "strict", validationAction: "error" })
19. Test > db.getCollectionInfos() > db.contacts.insert({ status: "ok" }) > db.contacts.insert({ status: "foobar" }) > db.runCommand({ insert: "contacts", documents: [ { status: "foobar"} ], bypassDocumentValidation: true })
20. Nested rules > db.runCommand({ collMod: "contacts", validator: { $or: [ { $and: [ { phone: { $type: "string" } }, { email: { $regex: /@test\.com$/ } }, { status: "ok" } ] }, { status: "incomplete" } ] } }) > db.contacts.insert({ name: "philipp", status: "incomplete" })
21. Versioning version is user-defined, type 2 is string > db.runCommand({ collMod: "contacts", validator: { $or: [ { version: { "$exists": false } }, { version: 1, $and: [ { name: { "$exists": true } } ] }, { version: 2, $and: [ { name: { "$exists": true, "$type": 2 } } ] } ] } }) > db.contacts.insert({ test: 1 }) > db.contacts.insert({ name: 1, version: 1 }) > db.contacts.insert({ name: "philipp", version: 2 })
22. Flags: validationLevel: strict | moderate | off If an existing document isn't valid, moderate updates won't validate validationAction: error | warn
23. Limitations Only on insert and update No help why a validation failed $geoNear, $near, $nearSphere, $text, $where
24. Lookup aka "JOIN"
25. We don't need JOINs, we have a rich document structure
26. Ok, we'll have $ref in all drivers
27. Ok, we'll have an aggregation framework
28. Ok, we'll have $lookup in the commercial version
29. Ok, $lookup everywhere Left Outer Equi-Join
30. Some data > db.purchases.insert({ "buyer" : "bill", "item" : "macbook" }) > db.purchases.insert({ "buyer" : "fred", "item" : "macbook" }) > db.purchases.insert({ "buyer" : "john", "item" : "macbook" }) > db.purchases.insert({ "buyer" : "john", "item" : "thinkpad" })
31. Aggregate > db.purchases.aggregate([ { $group: { _id: "$item", total: { $sum: 1 } } } ])
32. More data > db.products.insert({ name: "MacBook Pro", price: 2000, code: "macbook" }) > db.products.insert({ name: "Thinkpad", price: 1800, code: "thinkpad" })
33. Lookup > db.purchases.aggregate([ { $group: { _id: "$item", total: { $sum: 1 } } }, { $lookup: { from: "products", localField: "_id", foreignField: "code", as: "item_details" } } ])
34. Lookup > db.products.aggregate([ { $lookup: { from: "purchases", localField: "code", foreignField: "item", as: "buyers" } } ])
35. Flattened lookup > db.products.aggregate([ { $lookup: { from: "purchases", localField: "code", foreignField: "item", as: "buyers" } }, { $unwind: "$buyers" } ])
36. Flattened and cleaned lookup > db.products.aggregate([ { $lookup: { from: "purchases", localField: "code", foreignField: "item", as: "buyers" } }, { $unwind: "$buyers" }, { $project: { "_id": 0, "name": 1, "code": 1, "price": 1, "buyer": "$buyers.buyer" } } ])
37. More aggregation functions $sample > db.mycoll.aggregate({ $sample: { size: 100 } }) <5% data: Random cursor or index selection >=5% data: Scan entire collection, sort randomly in memory, fetch documents from the top
38. More aggregation functions $stdDevPop, $stdDevSamp, $sqrt, $abs, $log, $log10, $ln, $pow, $exp, $trunc, $ceil, $floor,...
39. Business Intelligence
40. MongoDB Connector for BI
41. Had MongoDB really done the impossible? Had they developed a connector which satisfies all the requirements of NoSQL analytics,...
42. but exposes relational semantics on flat, uniform data, so legacy BI software can handle it? — https://www.linkedin.com/pulse/mongodb-32- now-powered-postgresql-john-de-goes
43. MongoDB had gone from nothing to magic in just a few months [?] — https://www.linkedin.com/pulse/mongodb-32- now-powered-postgresql-john-de-goes
44. It uses a foreign data wrapper with PostgreSQL to provide a relational SQL view into your MongoDB data. — https://docs.mongodb.org/manual/products/bi- connector/
45. "Shit. This is bad news for MongoDB. Really bad." — https://www.linkedin.com/pulse/mongodb-32- now-powered-postgresql-john-de-goes
46. Storage Engines
47. MongoDB is webscale
48. Pluggable storage engine in 3.0 MMAPv1 and WiredTiger
49. Structure B-Tree LSM-Tree Fractal-Tree
50. MMAPv1 Default before 3.2 OS provides memory mapped files B-Tree
51. MMAPv1 Collection-level locking Documents a single block
52. WiredTiger Default in 3.2 Currently B-Tree Maybe LSM-Tree in the future
53. WiredTiger Compression Document-level locking Index prefix compression
54. WiredTiger Experience Much faster inserts Watch out for storage.wiredTiger.engineConfig.cacheSizeGB max(1GB, 60%-1GB)
55. Enterprise Feature Encrypted Storage Engine
56. Enterprise Feature In-Memory Storage Engine Currently in Beta
57. Percona RocksDB and PerconaFT
58. Config Servers as Replica Sets
59. Before 3.2 Exactly 3 independent config servers Read and write with 3 available nodes Read-only with less
60. With 3.2 Can be a replica set With up to 50 members
61. readConcern
63. @aphyr Call Me Maybe
64. writeConcern { w: <value>, j: <boolean>, wtimeout: <number> }
65. https://aphyr.com/posts/322-jepsen-mongodb-stale-reads
66. Dirty read Local state on minority primary modified before confiming with majority of nodes !
67. Stale read Stale data on minority primary
68. readConcern { level: <"majority"|"local"> }
69. Partial Indexes
70. > db.users.insert({ username: "philipp", active: 1 }) > db.users.createIndex( { username: 1 }, { partialFilterExpression: { active: { $eq: 1 } } } ) > db.users.find({ username: "philipp", active: { $eq: 1} })
71. > db.users.find({ username: "philipp", active: { $eq: 1} }).explain() { ... "winningPlan": { "stage": "FETCH", "filter": { "active": { "$eq": 1 } }, "inputStage": { "stage": "IXSCAN", "keyPattern": { "username": 1 }, ... > db.users.find({ username: "philipp" }).explain() { ... "winningPlan": { "stage": "COLLSCAN", "filter": { "username": { "$eq": "philipp" } }, ...
72. Conclusion
73. Some fancy additions Lots of internal cleanup
74. Personal Highlight Backup role can't read system.profile https://jira.mongodb.org/browse/SERVER-21724 in 3.2.1 / 3.0.9
75. PS: Robomongo Indiegogo and 3.0+ support
76. Announcements Datomic Feb 11 N1QL Mar 08
77. Thank you! Questions? @xeraa