10 Key Performance Indicators - Spencer Brody, 10gen

MongoBoston 2011

That's right: you too can learn to read the omens and ensure that your MongoDB deployment stays in tip-top shape. We'll look at memory usage, file... sizes, flushing, journaling, and all the special incantations that reveal MongoDB's true inner self. By the end of the talk, you'll have ten concrete steps you can take to address performance degradation before it happens. You'll also get a few tips on application design and pointers on remote monitoring.

Transcript

1. 10 Key Performance Indicators Wednesday, October 5, 2011
2. Who Am I • MongoDB Kernel Engineer @ 10gen • spencer@10gen.com • @stbrody Wednesday, October 5, 2011
3. Questions about speed • MongoDB is a high-performance database, but how do I know that I’m getting the best performance? Wednesday, October 5, 2011
4. Topics • Tools • Performance indicators • Remedies Wednesday, October 5, 2011
5. Prelude: Tools Wednesday, October 5, 2011
6. 1. mongostat Wednesday, October 5, 2011
7. 2. serverStatus > db.serverStatus(); { "host" : "Spencer-MacBook.local", "version" : "1.9.1-pre-", "process" : "mongod", "uptime" : 619052, // Lots more stats... } Wednesday, October 5, 2011
8. 3. Profiler > db.setProfilingLevel(2); { "was" : 0, "slowms" : 100, "ok" : 1 } Wednesday, October 5, 2011
9. 3. Profiler > db.system.profile.find() { "ts" : ISODate("2011-09-30T02:07:11.370Z"), "op" : "query", "ns" : "docs.spreadsheets", "nscanned" : 20001, "nreturned" : 1, "responseLength" : 241, "millis" : 1407, "client" : "127.0.0.1", "user" : "" } Wednesday, October 5, 2011
10. 4. Monitoring service • Nagios • Munin • MMS (www.10gen.com/try-mms) Wednesday, October 5, 2011
11. Indicators Wednesday, October 5, 2011
12. 1. Slow operations Sun May 22 19:01:47 [conn10] query docs.spreadsheets ntoreturn:100 reslen:510436 nscanned:19976 { username: “Brody, Spencer”} nreturned:100 147ms Wednesday, October 5, 2011
13. 2. Replication lag PRIMARY> rs.status() { "set" : "replSet", "date" : ISODate("2011-09-30T02:28:21Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "Spencer-MacBook.local:30001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "optime" : { "t" : 1317349400000, "i" : 1 }, "optimeDate" : ISODate("2011-09-30T02:23:20Z"), "self" : true }, { "_id" : 1, "name" : "Spencer-MacBook.local:30002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 302, "optime" : { "t" : 1317349400000, "i" : 1 }, "optimeDate" : ISODate("2011-09-28T10:17:47Z"), "lastHeartbeat" : ISODate("2011-09-30T02:28:19Z"), "pingMs" : 0 } ], "ok" : 1 } Wednesday, October 5, 2011
14. 3. Resident Memory > db.serverStatus().mem { "bits" : 64, // Need 64, not 32 "resident" : 7151, // Physical memory "virtual" : 14248, // Files + heap "mapped" : 6942 // Data files } Wednesday, October 5, 2011
15. 3. Resident Memory > db.stats() { "db" : "docs", "collections" : 3, "objects" : 805543, "avgObjSize" : 5107.312096312674, "dataSize" : 4114159508, // ~4GB "storageSize" : 4282908160, // ~4GB "numExtents" : 33, "indexes" : 3, "indexSize" : 126984192, // ~126MB "fileSize" : 8519680000, // ~8.5GB "ok" : 1 } Wednesday, October 5, 2011
16. 3. Resident Memory storageSize + indexSize <= RAM Wednesday, October 5, 2011
17. 4. Page faults > db.serverStatus().extra_info { "note" : "fields vary by platform", “heap_usage_bytes” : 210656, “page_faults” : 2381 } Wednesday, October 5, 2011
18. 5. Write lock percentage > db.serverStatus().globalLock { "totalTime" : 2809217799, "lockTime" : 13416655, "ratio" : 0.004775939766854653, } Wednesday, October 5, 2011
19. Concurrency • One writer OR many readers • Global RWlock • Yields on long-running ops and if we’re likely to go to disk. Wednesday, October 5, 2011
20. 5. Write lock percentage Wednesday, October 5, 2011
21. High lock percentage? You’re probably paging! Wednesday, October 5, 2011
22. 6. Reader and Writer queues > db.serverStatus().globalLock { "totalTime" : 2809217799, "lockTime" : 13416655, "ratio" : 0.004775939766854653, "currentQueue" : { "total" : 1, "readers" : 1, "writers" : 0 }, "activeClients" : { "total" : 2, "readers" : 1, "writers" : 1 } } Wednesday, October 5, 2011
23. 6. Reader and Writer queues > db.currentOp() { "inprog" : [ { "opid" : 6996, "active" : true, "lockType" : "read", "waitingForLock" : true, "secs_running" : 1, "op" : "query", "ns" : "docs.spreadsheets", "query" : { “username” : “Brody, Spencer” }, "client" : "10.71.194.111:51015", "desc" : "conn", "threadId" : "0x152693000", "numYields" : 0 }, ] } Wednesday, October 5, 2011
24. 7. Background Flushing > db.serverStatus().backgroundFlushing { "flushes" : 5634, "total_ms" : 83556, "average_ms" : 14.830670926517572, "last_ms" : 4, "last_finished" : ISODate("2011-09-30T03:30:59.052Z") } Wednesday, October 5, 2011
25. Disk considerations • Raid • SSD • SAN? Wednesday, October 5, 2011
26. 8. Connections > db.serverStatus().connections { "current" : 7, "available" : 19993 } Wednesday, October 5, 2011
27. 9. Network speed > db.serverStatus().network { "bytesIn" : 877291, "bytesOut" : 846300, "numRequests" : 9186 } Wednesday, October 5, 2011
28. 10. Fragmentation db.spreadsheets.stats() { "ns" : "docs.spreadhseets", "size" : 8200046932, // ~8GB "storageSize" : 11807223808, // ~11GB "paddingFactor" : 1.4302, "totalIndexSize" : 345964544, // ~345MB "indexSizes" : { "_id_" : 66772992, “username_1_filename_1” : 146079744, “username_1_updated_at_1” : 133111808 }, "ok" : 1 } Wednesday, October 5, 2011
29. 10. Fragmentation The magic number is 2 Wednesday, October 5, 2011
30. storageSize / size > 2 • Might not be reclaiming free space fast enough • Padding factor might not be correctly calibrated • db.spreadsheets.runCommand(“compact”) Wednesday, October 5, 2011
31. paddingFactor > 2 • You might have the wrong data model • You might be growing documents too much • Should review Schema Design Wednesday, October 5, 2011
32. The End Wednesday, October 5, 2011