Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Putting the GO in MongoDB

Putting the GO in MongoDB

A talk, made by Wisdom Omuya, about the rewrite of the MongoDB CLI tools for version 3.0 using the Go programming language.

Presented by me at the 6th MongoDB Belgium meetup. After Wisdom and I went through it during a 1-on-1 call


Hannes Van De Vreken

June 16, 2015

More Decks by Hannes Van De Vreken

Other Decks in Technology


  1. None
  2. None
  3. None
  4. None
  5. None
  6. Putting the Go in MongoDB Hannes Van De Vreken @hannesvdvreken

    Software Engineer hannes@mwl.be
  7. MongoDB v3.0.0

  8. WHAT TOOLS? tools

  9. downloads.mongodb.org

  10. Import mongorestore mongoimport mongofiles

  11. Export mongoexport mongodump

  12. Monitoring mongostat mongotop

  13. Others bsondump mongooplog

  14. ROUGH EDGES tools

  15. Mongorestore Concurrency in 2.6

  16. Invalid Documents • Nested CSV, TSV imports { "_id": ObjectId("5542593837096bdf8bbb6d91"),

    "name.first": "Hannes", "name.last": "Van De Vreken" } name.first,name.last Hannes,Van De Vreken
  17. mongo> db.test.find({"name.first": "Wisdom"}) Fetched 0 record(s) in 0ms Unretrievable Data

    { "_id": ObjectId("5542593837096bdf8bbb6d91"), "name.first": "Hannes", "name.last": "Van De Vreken" }
  18. REQUIREMENTS tools

  19. Loose Coupling

  20. Shorter Release Cycle

  21. Portability

  22. Concurrent Processing CPU

  23. General • Storage engine support • Wiredtiger, RocksDB, etc •

    Backwards compatibility • Dump and restore BSON • Import and export JSON
  24. • Excellent support for concurrency • Runs on all supported

    platforms • Easier to write and debug • Fun! Why Go?
  25. MONGOIMPORT tools

  26. Mongoimport Architecture

  27. New Mongoimport Flags --numDecodingWorkers (autodetect to max) --numInsertionWorkers (default 1)

    --batchSize (default 10k) --numThreads (autodetect to max) --maintainInsertionOrder (default false)
  28. Input Validation • No more broken imports, illegal field names

    ”a”, “b.”
  29. Dot-nesting • Nested CSV, TSV imports { "_id": ObjectId("5542593837096bdf8bbb6d91"), "name":

    { "first": "Hannes", "last": "Van De Vreken" } } name.first,name.last Hannes,Van De Vreken
  30. Import/Export Type Fidelity

  31. Write Concern Specificity • New default w=majority on import/restore/files o

    safer, matches what our users assume is happening • --writeConcern flag e.g. ‘{w: 3, j: true, fsync: false, wtimeout: 400}’
  32. Mongoimport Concurrency in 3.0

  33. MONGORESTORE tools

  34. Mongorestore Architecture

  35. New Mongorestore Flags --numParallelCollections (aka -j default=4) --numInsertionWorkersPerCollection (default=1) --batchSize(default

    10k) --numThreads (autodetect to max) --maintainInsertionOrder (default false)
  36. Mongorestore Concurrency in 3.0 2X speedup 4X speedup

  37. What’s next? • mongorestore and mongodump now support compression and

    archiving • No need for directories containing BSON files mongodump -d weather -h localhost --archive --gzip | mongorestore -h remotehost --archive --gzip
  38. Resources • Repository: • https://github.com/mongodb/mongo-tools • Usage Guide: https://docs.mongodb. org/master/reference/program/

    • Issue Tracking: https://jira.mongodb. org/browse/tools/
  39. THANKS! tools