10 Key Performance Indicators

Slide 1

Slide 1 text

Key Performance Indicators Christian Kvalheim 10gen Tuesday, April 24, 12

Slide 2

Slide 2 text

Agenda •The Players •Tools •Performance Indicators Tuesday, April 24, 12

Slide 3

Slide 3 text

Speed MongoDB is a high-performance database, but how do I know that I’m getting the best performance Tuesday, April 24, 12

Slide 4

Slide 4 text

Players • Memory – Memory mapped ﬁled, OS memory handling • Locks – Global write lock, going db level in 2.2 • Disk IO – IOPS, Latency • Network – Bandwidth Tuesday, April 24, 12

Slide 5

Slide 5 text

TOOLS Tuesday, April 24, 12

Slide 6

Slide 6 text

mongostat Tuesday, April 24, 12

Slide 7

Slide 7 text

The stats • Flushes • Mapped memory • Virtual memory size (Vsize) • Resident memory • Faults • Locked percentage Tuesday, April 24, 12

Slide 8

Slide 8 text

db.serverStatus > db.serverStatus(); { "host" : “MacBook.local", "version" : "2.0.1", "process" : "mongod", "uptime" : 619052, // Lots more stats... } Tuesday, April 24, 12

Slide 9

Slide 9 text

Proﬁler > db.setProfilingLevel(2); { "was" : 0, "slowms" : 100, "ok" : 1 } Tuesday, April 24, 12

Slide 10

Slide 10 text

Proﬁling • 3 levels – off – slower than x ms – all • Capped collection, 1MB default • Some performance overhead but minimal Tuesday, April 24, 12

Slide 11

Slide 11 text

Proﬁler > db.system.profile.find() { "ts" : ISODate("2011-‐09-‐30T02:07:11.370Z"), "op" : "query", "ns" : "docs.spreadsheets", "nscanned" : 20001, "nreturned" : 1, "responseLength" : 241, "millis" : 1407, "client" : "127.0.0.1", "user" : "" } Tuesday, April 24, 12

Slide 12

Slide 12 text

Monitoring Service • MMS: 10gen.com/try-mms • Nagios • Munin Tuesday, April 24, 12

Slide 13

Slide 13 text

INDICATORS Tuesday, April 24, 12

Slide 14

Slide 14 text

Slow Operations Sun May 22 19:01:47 [conn10] query docs.spreadsheets ntoreturn:100 reslen: 510436 nscanned:19976 { username: “Hackett, Bernie”} nreturned:100 147ms Tuesday, April 24, 12

Slide 15

Slide 15 text

Replication lag • replication lag is difference in time between the primaries last operation and the last operation the secondary committed Tuesday, April 24, 12

Slide 16

Slide 16 text

PRIMARY> rs.status() { "set" : "replSet", "date" : ISODate("2011-‐09-‐30T02:28:21Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "MacBook.local:30001", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "optime" : { "t" : 1317349400000, "i" : 1 }, "optimeDate" : ISODate("2011-‐09-‐30T02:23:20Z"), "self" : true }, { "_id" : 1, "name" : "MacBook.local:30002", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 302, "optime" : { "t" : 1317349400000, "i" : 1 }, "optimeDate" : ISODate("2011-‐09-‐28T10:17:47Z"), "lastHeartbeat" : ISODate("2011-‐09-‐30T02:28:19Z"), Tuesday, April 24, 12

Slide 17

Slide 17 text

Virtual memory Tuesday, April 24, 12

Slide 18

Slide 18 text

Resident Memory > db.serverStatus().mem { "bits" : 64, // Need 64, not 32 "resident" : 7151, // Physical memory "virtual" : 14248, // Files + heap "mapped" : 6942 // Data files } Tuesday, April 24, 12

Slide 19

Slide 19 text

Resident Memory > db.stats() { "db" : "docs", "collections" : 3, "objects" : 805543, "avgObjSize" : 5107.312096312674, "dataSize" : 4114159508, // ~4GB "storageSize" : 4282908160, // ~4GB "numExtents" : 33, "indexes" : 3, "indexSize" : 126984192, // ~126MB "fileSize" : 8519680000, // ~8.5GB "ok" : 1 } Tuesday, April 24, 12

Slide 20

Slide 20 text

Optimal Resident Memory indexSize + storageSize <= RAM Tuesday, April 24, 12

Slide 21

Slide 21 text

Page Faults > db.serverStatus().extra_info { "note" : "fields vary by platform", “heap_usage_bytes” : 210656, “page_faults” : 2381 } Tuesday, April 24, 12

Slide 22

Slide 22 text

Page Faults • The number of times the OS needs to read and write a new page of data into memory • Very high number indicates thrashing – OS spends more time reading/writing data to disk than doing work Tuesday, April 24, 12

Slide 23

Slide 23 text

Write Lock Percentage > db.serverStatus().globalLock { "totalTime" : 2809217799, "lockTime" : 13416655, "ratio" : 0.004775939766854653, } Tuesday, April 24, 12

Slide 24

Slide 24 text

Write lock percentage • the total amount of time the server spent in global write lock during the last sample period (one second) Tuesday, April 24, 12

Slide 25

Slide 25 text

Concurrency • One writer or many readers • Global RW Lock • Yields on long-running ops and if we’re likely to go to disk. Tuesday, April 24, 12

Slide 26

Slide 26 text

High Lock Percentage? You’re Probably Paging! Tuesday, April 24, 12

Slide 27

Slide 27 text

Queues • Tells you how many connections are waiting for reading or writing Tuesday, April 24, 12

Slide 28

Slide 28 text

> db.serverStatus().globalLock { "totalTime" : 2809217799, "lockTime" : 13416655, "ratio" : 0.004775939766854653, "currentQueue" : { "total" : 1, "readers" : 1, "writers" : 0 }, "activeClients" : { "total" : 2, "readers" : 1, "writers" : 1 } Tuesday, April 24, 12

Slide 29

Slide 29 text

Current Op • db.currentOp() let’s you see the current operation executing • db.killOp(id) lets you kill a blocking long running operation Tuesday, April 24, 12

Slide 30

Slide 30 text

> db.currentOp() { "inprog" : [ { "opid" : 6996, "active" : true, "lockType" : "read", "waitingForLock" : true, "secs_running" : 1, "op" : "query", "ns" : "docs.spreadsheets", "query" : { “username” : “Hackett, Bernie” }, "client" : "10.71.194.111:51015", "desc" : "conn", "threadId" : "0x152693000", "numYields" : 0 }, Tuesday, April 24, 12

Slide 31

Slide 31 text

Background Flushing • Tells you how often the data is written to disk • A high value might indicate IO performance issue – Might happen with network attached storage • Lower the time for ﬂushing to disk to write less data more often Tuesday, April 24, 12

Slide 32

Slide 32 text

Background Flushing > db.serverStatus().backgroundFlushing { "flushes" : 5634, "total_ms" : 83556, "average_ms" : 14.830670926517572, "last_ms" : 4, "last_finished" : ISODate("2011-‐09-‐30T03:30:59.052Z") } Tuesday, April 24, 12

Slide 33

Slide 33 text

Disk Considerations • Raid • SSD • SAN? Tuesday, April 24, 12

Slide 34

Slide 34 text

Connections > db.serverStatus().connections { "current" : 7, "available" : 19993 } Tuesday, April 24, 12

Slide 35

Slide 35 text

Connections • Each connection takes up heap space • The more connections the more context switching for the CPU • Clean up your connections after use Tuesday, April 24, 12

Slide 36

Slide 36 text

Network Speed > db.serverStatus().network { "bytesIn" : 877291, "bytesOut" : 846300, "numRequests" : 9186 } Tuesday, April 24, 12

Slide 37

Slide 37 text

Network Speed • Application might saturate connection leaving little replication bandwidth • Slow interconnect between app and db servers might limit your performance – Measure available bandwidth between servers, scp can be used for a sanity check of this. • If a problem bond connections, get 10Gbp cards. • Control the write speed doing getLastError every once in a while in the app Tuesday, April 24, 12

Slide 38

Slide 38 text

Fragmentation db.spreadsheets.stats() { "ns" : "docs.spreadhseets", "size" : 8200046932, // ~8GB "storageSize" : 11807223808, // ~11GB "paddingFactor" : 1.4302, "totalIndexSize" : 345964544, // ~345MB "indexSizes" : { "_id_" : 66772992, “username_1_filename_1” : 146079744, “username_1_updated_at_1” : 133111808 }, "ok" : 1 } Tuesday, April 24, 12

Slide 39

Slide 39 text

Fragmentation • Padding factor is the extra space MongoDB allocates for each document growth when saving documents – doc is 1000 bytes – padding factor 1.5 – total memory allocated 1500 bytes for doc Tuesday, April 24, 12

Slide 40

Slide 40 text

Fragmentation Padding factor >2 is the Horror Number Tuesday, April 24, 12

Slide 41

Slide 41 text

storageSize / size > 2 • Might not be reclaiming free space fast enough • Padding factor might not be correctly calibrated db.spreadsheets.runCommand(“compact”) Tuesday, April 24, 12

Slide 42

Slide 42 text

paddingFactor > 2 • You might have the wrong data model • You might be growing documents too much – embedded documents • Should review Schema Design Tuesday, April 24, 12

Slide 43

Slide 43 text

Summary • Ensuring dataset in memory is important – Avoid page faults • Find slow queries – Minimize time spent in write lock • Make sure you don’t ﬂood Mongo with connections • Ensure you padding factor is < 2 – Check you schema design Tuesday, April 24, 12

Slide 44

Slide 44 text

We’re Hiring Engineers, Sales, Evangelist, Marketing, Support, Developers @mongodb_jobs http://linkd.in/joinmongo Tuesday, April 24, 12

Slide 45

Slide 45 text

We’re Always Around For Conferences, Appearances and Meetups 10gen.com/events @mongodb h2p://bit.ly/mongo8 Tuesday, April 24, 12