Upgrade to Pro — share decks privately, control downloads, hide ads and more …

MongoNYC 2012: NYT + MongoDB in Production

mongodb
May 30, 2012
1.6k

MongoNYC 2012: NYT + MongoDB in Production

MongoNYC 2012: NYT + MongoDB in Production, Deep Kapadia, NYTimes.
This is a follow up talk to Jake Porway's talk at MongoNYC 2011 where he presented Project Cascade as one of R&D Labs Projects backed by MongoDB. Since then Project Cascade has moved to a scalable production level system. The talk will cover challenges and lessons learnt in moving MongoDB from a research project to a production environment.

mongodb

May 30, 2012
Tweet

Transcript

  1. Why MongoDB? • Quick Prototyping • Flexible schema ◦ Easy

    to dump data from 3rd party data sources ▪ bit.ly ▪ Twitter ◦ Schema may change depending on the need of the day or what metrics we are interested in • Quick scaling if needed
  2. 6 Months ago • Started out with two weeks data

    • Single MongoDB instance ◦ No replication ◦ No backups ◦ No monitoring • Data was stored locally on ephemeral storage on an EC2 instance • Logs were stored locally
  3. Technology stack • Amazon EC2 • MongoDB 2.0.x (started with

    1.8) • Python 2.7.2 • pymongo 2.1.x • Tornado 2.2 load balanced over Nginx • Little bit of Ruby on Rails (going away soon) • Custom WebGL based framework for visualization
  4. Monitoring • Monitoring tools ◦ db.serverStatus() + cron +email ◦

    10gen's Mongo Monitoring Service ◦ M/Monit -very basic ◦ Nagios • At the least monitor ◦ The mongod process ◦ Memory usage ◦ CPU usage ◦ Disk usage • Desirable to monitor EVERYTHING
  5. Replication • REALLY EASY to set up • 4 node

    replica set ◦ Primary ◦ Secondary ◦ Arbiter ◦ Delayed secondary - Priority 0 ▪ Never gets elected as Primary All instances are m1.large except Arbiter which is a t1.micro
  6. Replication • Be aware of how your drivers handle failover

    ◦ pymongo throws AutoReconnectException • Decide up front how you want to handle failover ◦ Lose data ◦ idempotent writes (keep trying until write is successful...up to a certain number of times)
  7. Storage • Do not use local storage on EC2 ◦

    ephemeral - does not persist on reboots • If using EC2 use EBS ◦ Persistent ◦ Easy to snapshot ◦ can be detached and attached to a different server ◦ Can be RAID'ed for reliablity and performance • Note: EBS is known to have inconsistent performance characteristics • Limited by 1GB/s
  8. Storage • We started with RAID 10 on EBS ◦

    difficult to image ◦ slightly steep learning curve if you are not used to tinkering with RAID/LVM • when using RAID, you would need to freeze the filesystem ◦ xfs_freeze • If not on EC2 just use File system snapshots
  9. Logs • Store logs an EBS block ◦ Logs can

    still be viewed in case your server goes down and cannot be restarted • Rotate your logs - please! ◦ db.runCommand("logRotate"); ◦ command line ◦ kill -SIGUSR1 <mongod process id> ◦ killall -SIGUSR1 mongod ◦ logrotate ▪ still requires a post-rotate kill command
  10. Backups and Restore • Snapshot EBS blocks ◦ --journaling is

    your friend ◦ need to use fsync + lock if journaling is disabled • Restoring from snapshots is easy ◦ create a new volume from the snapshot ◦ mount volume to EC2 instance Caveat: When using RAID, you would still need to use fsync + lock even if journaling is enabled.
  11. Backups and Restore • mongodump/mongorestore ◦ can be run while

    the DB is still running ◦ no need to lock the DB ◦ can backup and restore individual collections or even partial collections ◦ rebuilds indexes • Automate your backups ◦ https://github.com/micahwedemeyer/automongobackup
  12. Backups and Restore • use --oplog with mongodump and --

    oplogReplay with mongorestore • Backups and restore can be slow if your data is a few 100 GB. ◦ Plan for it • Use incremental backups ◦ possible with mongdump/mongorestore ▪ mongodump -q
  13. Understand your Data • Know whether your application is write

    heavy or a read heavy • Separate write heavy collections from read heavy collections • Minimize indexes on write heavy collections • Separate operational data from data used for mining/analytics if possible
  14. Querying • db.<collection>.find({x:123}) ◦ returns entire documents matching the criteria

    ◦ will be slow if you have large documents • $exists, $nin & $ne not very efficient ◦ try setting default values for keys instead of using exist ◦ try using $gt and $lt instead of $ne if possible (numerics)
  15. Querying • Limit the data you return only what you

    need ◦ Use range queries ◦ limit the number of results ◦ limit the number of keys returned • Increase or decrease the batch size for a cursor based on your needs ◦ returning a batch is a network operation
  16. Indexes • Use indexes judiciously • Create indexes to match

    your query keys • Understand what you get with a compound index ◦ db.collection.ensureIndex({a:1,b:1}) ▪ gives you an index on a and a&b but not on b ▪ ascending/descending may sometimes matter when using composite indexes
  17. Indexes (continued) • One index per query rule: ◦ Queries

    on multiple keys cannot use multiple indexes. Use a compound index instead ▪ $or is an exception • Make sure that all your indexes fit in the memory ◦ db.<collection>.getTotalIndexSize() • Sometimes indexes may not be helpful ◦ Low selectivity indexes • Use explain()
  18. Other details • Pay attention to the limitations of the

    MongoDB version and the driver you are using ◦ e.g: $exists does not use indexes prior to 2.0 ◦ e.g. $and is not supported in 1.8 • Design for performance ◦ iterate over schema design if it does not perform ◦ sometimes it is better to normalize than store everything in one large document ◦ archive historical data to a warehouse ▪ mining/analytics
  19. Administration tools • We use Rockmongo • But there are

    many other tools available ◦ http://www.mongodb.org/display/DOCS/Admin+UIs Create read only users for developers if needed.