MongoDB Diagnostics and Performance Tuning

From MongoDB LA, January 19, 2012.

1. Diagnostics and Performance Tuning Dan Crosta, 10gen dcrosta@10gen.com @lazlofruvous
2. Agenda •Tools •Performance Indicators
3. Speed MongoDB is a high-performance database, but how do I know that I’m getting the best performance
4. TOOLS
5. 1. mongostat
6. 2.serverStatus > db.serverStatus(); { ! ! "host" : “MacBook.local", "version" : "2.0.1", "process" : "mongod", "uptime" : 619052, // Lots more stats... }
7. 3.Profiler > db.setProfilingLevel(2); { "was" : 0, "slowms" : 100, "ok" : 1 }
8. 3.Profiler > db.system.profile.find() { "ts" : ISODate("2011-09-30T02:07:11.370Z"), "op" : "query", "ns" : "docs.spreadsheets", "query" : { "username": "dcrosta" }, "nscanned" : 20001, "nreturned" : 1, "responseLength" : 241, "millis" : 1407, "client" : "127.0.0.1", "user" : "" }
9. 4.Monitoring Service • MMS: 10gen.com/try-mms • Nagios • Munin
10. INDICATORS
11. 1.Slow Operations Sun May 22 19:01:47 [conn10] query docs.spreadsheets ntoreturn:100 reslen:510436 nscanned:19976 { username: “dcrosta”} nreturned:100 147ms
12. 2.Replication Lag PRIMARY> rs.status() { "set" : "replSet", "date" : ISODate("2011-09-30T02:28:21Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "MacBook.local:30001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "optime" : { "t" : 1317349400000, "i" : 1 }, "optimeDate" : ISODate("2011-09-30T02:23:20Z"), "self" : true }, { "_id" : 1, "name" : "MacBook.local:30002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 302, "optime" : { "t" : 1317349400000, "i" : 1 }, "optimeDate" : ISODate("2011-09-28T10:17:47Z"), "lastHeartbeat" : ISODate("2011-09-30T02:28:19Z"),
13. 3.Resident Memory > db.serverStatus().mem { "bits" : 64, // Need 64, not 32 "resident" : 7151, // Physical memory "virtual" : 14248, // Files + heap "mapped" : 6942 // Data files
14. 3.Resident Memory > db.stats() { "db" : "docs", "collections" : 3, "objects" : 805543, "avgObjSize" : 5107.312096312674, "dataSize" : 4114159508, // ~4GB "storageSize" : 4282908160, // ~4GB "numExtents" : 33, "indexes" : 3, "indexSize" : 126984192, // ~126MB "fileSize" : 8519680000, // ~8.5GB "ok" : 1 }
15. 3.Resident Memory ! ! indexSize + dataSize <= RAM
16. 4.Page Faults > db.serverStatus().extra_info { ! "note" : "fields vary by platform", ! “heap_usage_bytes” : 210656, ! “page_faults” : 2381 }
17. 5.Write Lock Percentage > db.serverStatus().globalLock { "totalTime" : 2809217799, "lockTime" : 13416655, "ratio" : 0.004775939766854653, }
18. Concurrency • One writer or many readers • Global RW Lock • Yields on long-running ops and if we’re likely to go to disk.
19. High Lock Percentage? You’re Probably Paging!
20. 6.Reader and Writer Queues > db.serverStatus().globalLock { "totalTime" : 2809217799, "lockTime" : 13416655, "ratio" : 0.004775939766854653, "currentQueue" : { "total" : 1, "readers" : 1, "writers" : 0 }, "activeClients" : { "total" : 2, "readers" : 1, "writers" : 1 }
21. 6.Reader and Writer Queues > db.currentOp() { "inprog" : [ { "opid" : 6996, "active" : true, "lockType" : "read", "waitingForLock" : true, "secs_running" : 1, "op" : "query", "ns" : "docs.spreadsheets", "query" : { “username” : “Hackett, Bernie” }, "client" : "10.71.194.111:51015", "desc" : "conn", "threadId" : "0x152693000", "numYields" : 0 },
22. 7.Background Flushing > db.serverStatus().backgroundFlushing { "flushes" : 5634, "total_ms" : 83556, "average_ms" : 14.830670926517572, "last_ms" : 4, "last_finished" : ISODate("2011-09-30T03:30:59.052Z") }
23. Disk Considerations • Raid • SSD • SAN?
24. 8.Connections > db.serverStatus().connections { "current" : 7, "available" : 19993 }
25. 9.Network Speed > db.serverStatus().network { "bytesIn" : 877291, "bytesOut" : 846300, "numRequests" : 9186 }
26. 10.Fragmentation db.spreadsheets.stats() { "ns" : "docs.spreadhseets", "size" : 8200046932, // ~8GB "storageSize" : 11807223808, // ~11GB "paddingFactor" : 1.4302, "totalIndexSize" : 345964544, // ~345MB "indexSizes" : { "_id_" : 66772992, “username_1_filename_1” : 146079744, “username_1_updated_at_1” : 133111808 }, "ok" : 1 }
27. 10.Fragmentation 2 is the Magic Number
28. storageSize / size > 2 • Might not be reclaiming free space fast enough • Padding factor might not be correctly calibrated db.spreadsheets.runCommand(“compact”)
29. paddingFactor > 2 • You might have the wrong data model • You might be growing documents too much • Should review Schema Design
30. download at mongoDB.org
31. We’re Hiring Engineers, Sales, Evangelist, Marketing, Support, Developers @mongodb_jobs http://linkd.in/joinmongo
32. We’re Always Around For Conferences, Appearances and Meetups 10gen.com/events @mongodb h2p://bit.ly/mongo8