10 Key Performance Indicators Spencer Brody, Software Engineer, 10gen

MongoDallas 2011

That's right: you too can learn to read the omens and ensure that your MongoDB deployment stays in tip-top shape. We'll look at memory usage, file sizes, flushing, journaling, and all the special incantations that reveal MongoDB's true inner self. By the end of the talk, you'll have ten concrete steps you can take to address performance degradation before it happens. You'll also get a few tips on application design and pointers on remote monitoring.

Transcript

1. 10 Key Performance Indicators Monday, November 28, 2011
2. Who Am I • MongoDB Kernel Engineer @ 10gen • spencer@10gen.com • @stbrody Monday, November 28, 2011
3. Questions about speed • MongoDB is a high-performance database, but how do I know that I’m getting the best performance? Monday, November 28, 2011
4. Topics • Tools • Performance indicators • Remedies Monday, November 28, 2011
5. Prelude: Tools Monday, November 28, 2011
6. 1. mongostat Monday, November 28, 2011
7. 2. serverStatus > db.serverStatus(); { "host" : "Spencer-MacBook.local", "version" : "1.9.1-pre-", "process" : "mongod", "uptime" : 619052, // Lots more stats... } Monday, November 28, 2011
8. 3. Profiler > db.setProfilingLevel(2); { "was" : 0, "slowms" : 100, "ok" : 1 } Monday, November 28, 2011
9. 3. Profiler > db.system.profile.find() { "ts" : ISODate("2011-09-30T02:07:11.370Z"), "op" : "query", "ns" : "docs.spreadsheets", "nscanned" : 20001, "nreturned" : 1, "responseLength" : 241, "millis" : 1407, "client" : "127.0.0.1", "user" : "" } Monday, November 28, 2011
10. 4. Monitoring service • Nagios • Munin • MMS (www.10gen.com/try-mms) Monday, November 28, 2011
11. Indicators Monday, November 28, 2011
12. 1. Slow operations Sun May 22 19:01:47 [conn10] query docs.spreadsheets ntoreturn:100 reslen:510436 nscanned:19976 { username: “Brody, Spencer”} nreturned:100 147ms Monday, November 28, 2011
13. 2. Resident Memory > db.serverStatus().mem { "bits" : 64, // Need 64, not 32 "resident" : 7151, // Physical memory "virtual" : 14248, // Files + heap "mapped" : 6942 // Data files } Monday, November 28, 2011
14. 2. Resident Memory > db.stats() { "db" : "docs", "collections" : 3, "objects" : 805543, "avgObjSize" : 5107.312096312674, "dataSize" : 4114159508, // ~4GB "storageSize" : 4282908160, // ~4GB "numExtents" : 33, "indexes" : 3, "indexSize" : 126984192, // ~126MB "fileSize" : 8519680000, // ~8.5GB "ok" : 1 } Monday, November 28, 2011
15. 2. Resident Memory storageSize + indexSize <= RAM Monday, November 28, 2011
16. 3. Page faults > db.serverStatus().extra_info { "note" : "fields vary by platform", “heap_usage_bytes” : 210656, “page_faults” : 2381 } Monday, November 28, 2011
17. 4. Write lock percentage > db.serverStatus().globalLock { "totalTime" : 2809217799, "lockTime" : 13416655, "ratio" : 0.004775939766854653, } Monday, November 28, 2011
18. Concurrency • One writer OR many readers • Global RWlock • Yields on long-running ops and if we’re likely to go to disk. Monday, November 28, 2011
19. 4. Write lock percentage Monday, November 28, 2011
20. High lock percentage? You’re probably paging! Monday, November 28, 2011
21. 5. Background Flushing > db.serverStatus().backgroundFlushing { "flushes" : 5634, "total_ms" : 83556, "average_ms" : 14.830670926517572, "last_ms" : 4, "last_finished" : ISODate("2011-09-30T03:30:59.052Z") } Monday, November 28, 2011
22. Disk considerations • Raid • SSD • SAN? Monday, November 28, 2011
23. Disk utilization iostat -x 2 Linux 2.6.38-12-generic (ubuntu) 11/17/2011 _x86_64_ (8 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 10.69 0.62 1.58 0.47 0.00 86.63 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 5.89 0.10 0.36 3.80 25.10 123.85 0.02 48.10 0.84 61.27 0.70 0.03 sdb 0.13 264.95 0.78 10.07 16.31 1103.36 206.52 2.34 215.60 23.01 230.48 3.25 3.52 Monday, November 28, 2011
24. 6. Replication lag PRIMARY> rs.status() { "set" : "replSet", "date" : ISODate("2011-09-30T02:28:21Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "Spencer-MacBook.local:30001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "optime" : { "t" : 1317349400000, "i" : 1 }, "optimeDate" : ISODate("2011-09-30T02:23:20Z"), "self" : true }, { "_id" : 1, "name" : "Spencer-MacBook.local:30002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 302, "optime" : { "t" : 1317349400000, "i" : 1 }, "optimeDate" : ISODate("2011-09-28T10:17:47Z"), "lastHeartbeat" : ISODate("2011-09-30T02:28:19Z"), "pingMs" : 0 } ], "ok" : 1 } Monday, November 28, 2011
25. 7. Reader and Writer queues > db.serverStatus().globalLock { "totalTime" : 2809217799, "lockTime" : 13416655, "ratio" : 0.004775939766854653, "currentQueue" : { "total" : 1, "readers" : 1, "writers" : 0 }, "activeClients" : { "total" : 2, "readers" : 1, "writers" : 1 } } Monday, November 28, 2011
26. 7. Reader and Writer queues > db.currentOp() { "inprog" : [ { "opid" : 6996, "active" : true, "lockType" : "read", "waitingForLock" : true, "secs_running" : 1, "op" : "query", "ns" : "docs.spreadsheets", "query" : { “username” : “Brody, Spencer” }, "client" : "10.71.194.111:51015", "desc" : "conn", "threadId" : "0x152693000", "numYields" : 0 }, ] } Monday, November 28, 2011
27. 8. Connections > db.serverStatus().connections { "current" : 7, "available" : 19993 } Monday, November 28, 2011
28. 9. Network speed > db.serverStatus().network { "bytesIn" : 877291, "bytesOut" : 846300, "numRequests" : 9186 } Monday, November 28, 2011
29. 10. Fragmentation db.spreadsheets.stats() { "ns" : "docs.spreadhseets", "size" : 8200046932, // ~8GB "storageSize" : 11807223808, // ~11GB "paddingFactor" : 1.4302, "totalIndexSize" : 345964544, // ~345MB "indexSizes" : { "_id_" : 66772992, “username_1_filename_1” : 146079744, “username_1_updated_at_1” : 133111808 }, "ok" : 1 } Monday, November 28, 2011
30. 10. Fragmentation The magic number is 2 Monday, November 28, 2011
31. storageSize / size > 2 • Might not be reclaiming free space fast enough • Padding factor might not be correctly calibrated • db.spreadsheets.runCommand(“compact”) Monday, November 28, 2011
32. paddingFactor > 2 • You might have the wrong data model • You might be growing documents too much • Should review Schema Design Monday, November 28, 2011
33. The End Monday, November 28, 2011