MongoDB Intro + Performance Tuning & Monitoring

by James Tan

Slide 1

Slide 1 text

Senior Consultant MongoDB, Inc. London, UK James Tan #MongoDB User Group - Singapore Performance Tuning and Monitoring

Slide 2

Slide 2 text

What is MongoDB?

Slide 3

Slide 3 text

MongoDB is a ___________ database •  Document •  Open source •  High performance •  Horizontally scalable •  Full featured

Slide 4

Slide 4 text

Database Landscape Depth of Functionality Scalability & Performance Memcached MongoDB RDBMS

Slide 5

Slide 5 text

Document Database •  Not for .PDF & .DOC ﬁles •  A document is essentially an associative array •  Document = JSON object •  Document = PHP Array •  Document = Python Dict •  Document = Ruby Hash •  etc

Slide 6

Slide 6 text

Open Source •  MongoDB is an open source project •  On GitHub •  Licensed under the AGPL •  Started & sponsored by MongoDB Inc (formerly 10gen) •  Commercial licenses available •  Contributions welcome

Slide 7

Slide 7 text

High Performance •  Written in C++ •  Extensive use of memory-mapped ﬁles i.e. read-through write-through memory caching. •  Runs nearly everywhere •  Data serialized as BSON (fast parsing) •  Full support for primary & secondary indexes •  Document model = less work

Slide 8

Slide 8 text

Full Featured •  Ad Hoc queries •  Real time aggregation •  Rich query capabilities •  Strongly consistent •  Geospatial features •  Support for most programming languages •  Flexible schema

Slide 9

Slide 9 text

Deployment Models

Slide 10

Slide 10 text

Secondary Secondary Primary Client Application Driver Write Read Replica Set Arbiter

Slide 11

Slide 11 text

Node 1 Secondary Config Server Node 1 Secondary Config Server Node 1 Secondary Config Server Shard Shard Shard Mongos App Server Mongos App Server Mongos App Server Sharded Cluster

Slide 12

Slide 12 text

Working Set Exceeds Physical Memory

Slide 13

Slide 13 text

Read/Write Throughput Exceeds I/O

Slide 14

Slide 14 text

Vertical Scalability (Scale Up)

Slide 15

Slide 15 text

Horizontal Scalability (Scale Out)

Slide 16

Slide 16 text

Performance Tuning and Monitoring

Slide 17

Slide 17 text

What is performance tuning? 1.  Assess the problem and establish acceptable behavior 2.  Measure the current performance 3.  Find the bottleneck* 4.  Remove the bottleneck 5.  Re-test to conﬁrm 6.  Lather, rinse, repeat * - (This is often the hard part) (Adapted from http://en.wikipedia.org/wiki/Performance_tuning )

Slide 18

Slide 18 text

Pro-Tip: Know Thyself You have to recognize normal to know when it isn’t Source: http://www.ﬂickr.com/photos/skippy/6853920/

Slide 19

Slide 19 text

Metrics to Watch •  Memory usage •  Opcounters •  Lock % •  Queues •  Background ﬂush average •  Replication stats

Slide 20

Slide 20 text

mongotop & mongostats

Slide 21

Slide 21 text

What is MMS? The MongoDB Management Service: a free service (or software) for monitoring and backup

Slide 22

Slide 22 text

Setting it up http://mms.mongodb.com/help/monitoring/tutorial/ •  Setup an account •  Install the agent •  Add your hosts •  Optional: hardware stats through munin-node •  Optional: enable logging and proﬁling •  More: mms.mongodb.com/help/monitoring/install/

Slide 23

Slide 23 text

Notes •  Agent written in Go (was Python) •  Failover: run multiple agents (1 primary) •  Hosts: use CNAMEs, especially on AWS! •  You can use a group per env (each needs an agent) •  Connections are over SSL •  On-Premise solution for those that don’t want to use the hosted service

Slide 24

Slide 24 text

Example: Replication Lag •  Secondaries under-specced vs primaries •  Access patterns between primary/ secondaries •  Insufﬁcient bandwidth •  Foreground index builds on secondaries “…when you have eliminated the impossible, whatever remains, however improbable, must be the truth…” -- Sherlock Holmes Sir Arthur Conan Doyle, The Sign of the Four

Slide 25

Slide 25 text

Remember to use alerts! Don’t wait until your secondaries fall off your oplog!

Slide 26

Slide 26 text

Analyzing mongod.log

Slide 27

Slide 27 text

Example: Slow Performance Log analysis The best tools for analyzing MongoDB logs are included in mtools*: mlogﬁlter (ﬁlter logs for slow queries, table scans, etc…) mplotqueries (graph query response times and volumes) * https://github.com/rueckstiess/mtools

Slide 28

Slide 28 text

Example: Slow Performance Log analysis (example syntax) Show me queries that took more than 1000 ms from 6 am to 6 pm: mlogfilter mongodb.log -‐-‐from 06:00 -‐-‐to 18:00 -‐-‐slow 1000 > mongodb-‐filtered.log Now, graph those queries: mplotqueries -‐-‐logscale mongodb-‐ filtered.log

Slide 29

Slide 29 text

Example: Slow Performance

Slide 30

Slide 30 text

Example: Slow query Wed Jul 17 13:40:14 [conn28600] query x.y [snip] ntoreturn:16 ntoskip:0 nscanned:16779 scanAndOrder:1 keyUpdates:0 numYields: 906 locks(micros) r:46877422 nreturned:16 reslen: 6948 38172ms 38 seconds! Scanned 17k documents, returned 16.

Slide 31

Slide 31 text

Example: Slow update Wed Jul 17 14:11:59 [conn56541] update x.y query: { e: ”[id1]" } update: { $addToSet: { fr: ”[id2]" } } nscanned:1 nmoved:1 nupdated:1 keyUpdates:0 locks(micros) w:85145 11768ms Almost 12 seconds! This time, there’s “nmoved:1”, too. This means a document was moved on disk – it outgrew the space allocated for it.

Slide 32

Slide 32 text

Monitoring: watch for warnings MMS warns you if your systems have startup warnings or if they are running outdated versions. Don’t ignore these!