MongoSF 2012: MongoDB Deployment Preparedness

The last bugs are finished, testing is complete, and business is ready. What do you do next? In this talk we will cover the topics to ensure that you are prepared for a successful launch of your MongoDB based product, including:

Machine Sizing : How much CPU, memory, and disk should I use for my MongoDB? Backup / Restore procedures? What are my options and what do I need to do?

Load Testing and Capacity Planning: How much resource is my MongoDB going to use? When do I need to add replicas and shards?

Monitoring: What should I be watching and how do I know if things are running correctly?

Transcript

1. Dan Pasette @pasette Deployment Preparedness
2. This Talk • What is deployment preparedness? • Recommended process and tips • Q & A
3. About me • Engineering Manager in NYC • Server side developer for 15 years • Everyone works with the community
4. Forecasting
5. Disaster Recovery
6. Elements of Success • Prototype • Test • Monitor • Capacity Planning • Ops Playbook
7. A word about process • Iterative seems to work best • Requirements are required
8. Reinventing the Wheel Prototype Test Monitor Capacity   Planning Ops   Playbook
9. Prototype
10. Design Matters • Development spikes should be easy • Schema free does not mean free-form • Index design is crucial • Pre-optimization is the root of all evil!
11. Prototype Test Monitor Capacity   Planning Ops  Playbook
12. Why hard? • PITA • No obvious biz case (not a “feature”) • Time - last task • $$$
13. Why do it? • Reduce the set of unknowables • Prove out theories • Provide input data for planning
14. What should I do? • Part of the process from the start • Isolate (start small) • Get real (holistic and realistic)
15. Prototype Test Monitor Capacity   Planning Ops  Playbook
17. Kinds of Monitoring • Instrumentation and logging • Trending • Alerts
18. Microscope • Fine grained stats to tell you what’s happening now • explain(), db profiling system • mongostat, iostat, mongotop... • MMS -Free, hosted monitoring from 10gen
19. mongostat
20. MMS Microscope
21. Telescope • Trending stats to help you analyze the past and predict the future • munin, cacti, nagios, zabix, graphite... • MMS - http://mms.10gen.com
22. MMS Telescope
23. Prototype Test Monitor Capacity   Planning Ops  Playbook
24. Ways to Scale • Vertical • RAM, SSDs • Horizontal • Sharding and replica sets • Iterate your schema design!
25. Sizing Considerations • Access Pattern • Working Set • Latency Requirements • Data Security
26. Working Set >  db.blogs.stats()   {   "ns"  :  "test.blogs",   "count"  :  1338330,   "size"  :  46915928,   "avgObjSize"  :  35.05557523181876,   "storageSize"  :  86092032,   "numExtents"  :  12,   "nindexes"  :  2,   "lastExtentSize"  :  20872960,   "paddingFactor"  :  1,   "flags"  :  0,   "totalIndexSize"  :  99860480,   "indexSizes"  :  {     "_id_"  :  55877632,     "name_1"  :  43982848   }, Size of data Size on disk (and in memory!) Size of all indexes Average document size Size of each index
27. Access Pattern & Working Set • Random Access • Long Tail, Right Balanced • Event Stream
28. Prototype Test Monitor Capacity   Planning Ops  Playbook
29. Plays in the book • Backups • Restores (backups are not enough) • Upgrades • Replica Set Ops • understand the oplog • Sharding Ops
30. Wrap Up • Eat your veggies • Humans are irrational animals • Just do what you can
31. Download MongoDB http://www.mongodb.org and let us know what you think @pasette @mongodb 10gen is hiring! http://www.10gen.com/jobs