Upgrade to Pro — share decks privately, control downloads, hide ads and more …

MongoDB and Meeting Your Storage Needs

MongoDB and Meeting Your Storage Needs

CodeMash 2014 - MongoDB and Meeting Your Storage Needs

Avatar for Brett Berliner

Brett Berliner

January 09, 2014

More Decks by Brett Berliner

Other Decks in Technology

Transcript

  1. Brett is a Lead Developer for Questline. Prior to joining

    Questline, Brett worked as a Programmer Analyst for Grange Insurance as well as an IT consultant for Agile Technologies. The handsome devil is passionate about learning new technologies and working to solve complex problems the right way. He loves music and playing and watching hockey and soccer. Currently, Brett lives in Upper Arlington with his hot, hot girlfriend. About Brett
  2. Questline is a 20 year old company specializing in digital

    communications for the utility industry. We send and manage Email, SMS, Fax and social communications for over 450 clients in the USA & Canada. For the last year, the focus of our entire DevOps team has been on building a next generation messaging and content system. About Questline
  3. “MongoDB is currently the most used NoSQL database. Written in

    C++, Mongo stores schemaless binary JSON (BSON) documents and uses Javascript for primary access. Mongo is primarily lauded for its speed, ease of installation, and most importantly, its maturity.” What is MongoDB?
  4. • Schema-less • Incredibly fast • Indexes • Easy •

    “FREE” - air quotes... • Advanced features Mongo Pros & Cons • Schema-less • Querying difficulties • Document size limit • Indexes need to fit in RAM • ...see presentation Pros: Cons:
  5. Questline’s Situation • Replacing a 10+ year old legacy solution

    (don’t ask) • Owning our data/upgrades through removal of 3rd parties/APIs • Taking advantage of technical know how • Trying something different (vanity) - and why not? It’s someone else’s money...
  6. Questline’s Needs • Data Control • Flexible schema • “Cheap”

    - again, air quotes • Scalability • Availability • Maintainability
  7. Why Mongo for us? • Creator supported and mature C#

    driver • Flexible since there is no schema • Very fast (subjective.. but mostly true) • Javascript for scripting • Scaling (through sharding) • Availability (replica sets) • Maintainability (jury is still out)
  8. • This is not SQL • Indexes (again, this is

    not SQL) • Schema-less doesn’t mean anything goes • Locking (wha? yes.) • Tooling (huh huh) • Workload management • Cost Love is a Battlefield
  9. • Case sensitivity (yes, this matters) • Special characters (need

    escapin’) • Text search (not great - in older versions) • Joins (nope, they don’t exist) • Array support (is awesome) • Co-existence of different data version documents (can be cool) This is not SQL
  10. • Discriminators (defining the object type) • Data types -

    JS data types, interchangeable • UTC is DEFAULT (great! but just be aware) • Counting…sucks. Find an alternative fast. • Split large documents into their component parts (multiple collections if it makes sense) This is not SQL (Deux)
  11. • Queries are easy to chain together...until they are not

    ($in, $elemMatch are weird…) • The driver provides builders for all the query types (query, update, sort) • Cannot update a field based on a field in the same document • Aggregation framework vs. Map-Reduce This is not SQL (Tres) - Queries
  12. • Must fit in memory (as with all DBs..) •

    Order of fields is KEY - query can use either all the fields or a prefix of the fields • Sparse/TTL/Full • Unique index and ‘drop dupes’ - prepare to commit to that change • Can be run in background • Queries with RegEx cannot use indexes (not just in Mongo obviously…) Indexes
  13. • Documents have a size limit (16MB) • Short field

    names - yes, this is important • Collections - shard before 256GB, even though sharding may be possible up to 400GB • Be smart about your docs...don’t over do nesting. Things can get messy very quickly without controls • Use updates/increments where possible • Upserts work best with unique index • 2 properties with same name but different type on single document can’t be deserialized (not Mongo’s fault, but check it if you can’t deserialize a document) Schema-less doesn’t mean a free for all
  14. • Both read and write locks exist • Locks are

    per database after mongo 2.2 (per mongod - server- before 2.2) • Read locks can be reused • Write locks only allow 1 write operation at a time and block reads • Operations can yield the locks Locking
  15. • Javascript + command prompt (so...no intellisense) • Lack of

    official GUI (we use MongoVue…but that’s nothing to be proud of) • MMS is alright but a tad unusable • Mongo’s baked in stats are actually pretty useful, as are running ops • Profiling - meh. Tooling (or lack thereof)
  16. • Bulk inserts - we had better luck using MongoImport

    and MongoExport, but be aware you lose type info • If necessary, trickle writes back - either by message queue or capped collection • Backups - this is on YOU (for a bit longer). Our routine - a python script to cycle primaries and snapshot EBSs (Don’t steal this without paying me. Thanks!) Workload Management
  17. • Free doesn’t mean free • While you can use

    commodity hardware, it can get expensive • Sharding requires factors of 3 • Big machines on AWS = pricey • Obviously, if you have VC.. who cares? Cost
  18. • RabbitMQ or another queue can work wonders (Amazon SQS

    is pretty sweet, but not FIFO) • ElasticSearch hooks in really nicely • We use A LOT of Windows services for background work • Compact - can solve memory issues, if you need to do often, there are other issues at hand • Use Mongo when it makes sense Last Thoughts...