Putting the GO in MongoDB

A talk, made by Wisdom Omuya, about the rewrite of the MongoDB CLI tools for version 3.0 using the Go programming language.

Presented by me at the 6th MongoDB Belgium meetup. After Wisdom and I went through it during a 1-on-1 call

Transcript

6. Putting the Go in MongoDB Hannes Van De Vreken @hannesvdvreken Software Engineer hannes@mwl.be
7. MongoDB v3.0.0
8. WHAT TOOLS? tools
9. downloads.mongodb.org
10. Import mongorestore mongoimport mongofiles
11. Export mongoexport mongodump
12. Monitoring mongostat mongotop
13. Others bsondump mongooplog
14. ROUGH EDGES tools
15. Mongorestore Concurrency in 2.6
16. Invalid Documents • Nested CSV, TSV imports { "_id": ObjectId("5542593837096bdf8bbb6d91"), "name.first": "Hannes", "name.last": "Van De Vreken" } name.first,name.last Hannes,Van De Vreken
17. mongo> db.test.find({"name.first": "Wisdom"}) Fetched 0 record(s) in 0ms Unretrievable Data { "_id": ObjectId("5542593837096bdf8bbb6d91"), "name.first": "Hannes", "name.last": "Van De Vreken" }
18. REQUIREMENTS tools
19. Loose Coupling
20. Shorter Release Cycle
21. Portability
22. Concurrent Processing CPU
23. General • Storage engine support • Wiredtiger, RocksDB, etc • Backwards compatibility • Dump and restore BSON • Import and export JSON
24. • Excellent support for concurrency • Runs on all supported platforms • Easier to write and debug • Fun! Why Go?
25. MONGOIMPORT tools
26. Mongoimport Architecture
27. New Mongoimport Flags --numDecodingWorkers (autodetect to max) --numInsertionWorkers (default 1) --batchSize (default 10k) --numThreads (autodetect to max) --maintainInsertionOrder (default false)
28. Input Validation • No more broken imports, illegal field names ”a”, “b.”
29. Dot-nesting • Nested CSV, TSV imports { "_id": ObjectId("5542593837096bdf8bbb6d91"), "name": { "first": "Hannes", "last": "Van De Vreken" } } name.first,name.last Hannes,Van De Vreken
30. Import/Export Type Fidelity
31. Write Concern Specificity • New default w=majority on import/restore/files o safer, matches what our users assume is happening • --writeConcern flag e.g. ‘{w: 3, j: true, fsync: false, wtimeout: 400}’
32. Mongoimport Concurrency in 3.0
33. MONGORESTORE tools
34. Mongorestore Architecture
35. New Mongorestore Flags --numParallelCollections (aka -j default=4) --numInsertionWorkersPerCollection (default=1) --batchSize(default 10k) --numThreads (autodetect to max) --maintainInsertionOrder (default false)
36. Mongorestore Concurrency in 3.0 2X speedup 4X speedup
37. What’s next? • mongorestore and mongodump now support compression and archiving • No need for directories containing BSON files mongodump -d weather -h localhost --archive --gzip | mongorestore -h remotehost --archive --gzip
38. Resources • Repository: • https://github.com/mongodb/mongo-tools • Usage Guide: https://docs.mongodb. org/master/reference/program/ • Issue Tracking: https://jira.mongodb. org/browse/tools/
39. THANKS! tools