KLMUG4-Mongo-Gone-Wild

Presentation for Kuala Lumpur MongoDB User Group 4 @Laulima - Dec 7, 2011. Attempted to use DTrace to understand the behavior of GridFS under normal vs memory pressure condition. Discussion later on observed behavior.

Transcript

1. Mongo Gone Wild!! (Using DTrace to Observe MongoDB GridFS under Memory Pressure) KLMUG4 2011 December 7, 2011 Michael Leow - Carlist.my
2. Agenda • Introduction • Hypothesis • Basic DTrace • Providers for MongoDB • GridFS Under Normal Condition • GridFS Under Memory Pressure • Discussion • Bonus: DTrace Visualization of MongoPress • Bonus2: More DTrace Analytics/Visualization
3. Introduction • Disclaimer: I am a beginner in SmartOS, DTrace and MongoDB :) • Disclaimer: Content includes mash-up of other DTrace Content from Joyent/Sun/Oracle folks • http://www.reddit.com/r/programming/comments/m2b2b/dont_use_mongodb/ “ But if you're intending to really run a large scale system on Mongo, one that a business might depend on, simply put: Don't. ”
4. Flame Suit On! Image: http://media.photobucket.com/image/flame%20suit/HarleyQuinneWY6/gif%2520stuff/nasty/flame_suit.jpg
5. Hypothesis • Flame Out in Heavy Memory Pressure • FUD until Measured and Tested • Setup: MongoDB-latest in zone setup as per Joyent • Setup: VirtualBox – 2 vCPU, 1GB, Joyent SmartOS • Setup: Default values; no tuning at all • Setup: File sizes tested: 100KB, 1MB, 5MB, 25MB, 50MB, 75MB, 90MB, 100MB, 500MB
6. Hypothesis • Dynamic Tracing: ability to instrument running software • Collect timestamps and other info (workload specifics) • Lets you create ideal metrics that matter and clear the FUD! • Imagine: • Black/white answers: either there is an issue, or there isn’t • 100% reliability • No expert interpretation required • No time consuming analysis required
7. Basic DTrace • An implementation of Dynamic Tracing that provides dynamic (and static) probes • Includes a rich set of actions, including data aggregations • Currently for: • Solaris-based OSes (SmartOS, OpenIndiana, Solaris 10+ etc.) • Mac OS X 10.5+, FreeBSD 7.1+ • < Alpha - Oracle Unbreakable Linux, Linux? http://crtags.blogspot.com • Production safe; in use since 2005 • User- and kernel-land tracing • C & awk inspired language (D-language)
8. Basic DTrace • Probes • A point of instrumentation and data generation • User-defined actions can be executed when probes "fire" • Static: io:::start, proc:::exec-success,perl*:::sub- entry, php*:::error, … • Dynamic: pid$target:mongod:*MMF*:entry, pid816:libc:strlen:entry, fbt:zfs:arc_read:entry, ...
9. Basic DTrace • D Program Structure • Probes have a unique four-tuple name: probe, ... /optional predicate/ { optional action; ... } provider:module:function:name provider module function name
10. Basic DTrace • Command line • Scripts • Consumers – plockstat, intrstat dtrace -n 'syscall:::entry { @sc[execname] = count(); } tick-5sec { trunc(@sc, 10); printa(@sc); }' dtrace -n 'vminfo:::pgpgin { @pg[execname ] = count(); } dtrace -n 'syscall:::entry { @sc[ustack()] = count(); }
11. Providers for MongoDB • Basic one-liners • Compare mongod calls between normal and abnormal condition
12. Providers for MongoDB • VM Subsystem one-liner
13. Providers for MongoDB • Timeline (incl. faults/mongo MMF) - mongo_breakdown_diff_vmm.d
14. Providers for MongoDB • vminfo major faults - mongo_vmm.d • vminfo ALL faults - mongo_all_vmm.d
15. GridFS Under Normal Condition • Observe MongoDB log • Observe timeline pattern • Observe vminfo faults
16. GridFS Under Memory Pressure • Observe MongoDB log • Observe timeline pattern • Observe vminfo faults
17. Discussion • Memory Mapped Files in Solaris (from Solaris Internals)
18. Discussion • Observation: Number of mongod calls similar for both cases • Observation: mmap calls dispersed with large gaps between - why? • Observation: Memory Hungry; suck up all available – how to limit? • Observation: Diskspace Hungry; suck up all available – how to compress/dedupe? • Others?
19. Demo Resources • MongoDB-latest Setup at: http://wiki.joyent.com/display/node/Installing+M ongoDB+on+a+Node.js+SmartMachine • Scripts + Transcript available at: https://plus.google.com/u/0/1121170611184333 37192/posts/WFBhH8uPNDe
20. DTrace Resources
21. DTrace Visualization of MongoPress • Demo
22. NFS Latency Heat Map
23. Cloud Analytics • Joyent Cloud / AniseAsia Cloud @Msia
24. Detecting slow Node.js app
25. Observing Ext3 Behavior in KVM