Why we recommend PMM to our clients

As service providers, one of our responsibilities is helping clients understand what causes contributed to a production downtime incident, and how to avoid (as much as possible) them from happening again. We do this with Incident Reports, and one common recommendation we make is to have a historical monitoring system in place. All our clients have point-in-time monitoring solutions in place, solutions that can alert them when a system is down or behaving in unacceptable ways. But historical monitoring is still not common, and we believe a lot of companies can benefit from deploying one of them. In most cases, we have recommended Percona Monitoring and Management (PMM), as a good and Open Source solution for this problem. In this session, we will talk about the reasons why we recommend PMM as a way to prevent incidents, and also to investigate their possible causes when one has happened.

Transcript

1. 2 © The Pythian Group Inc., 2018 November 6, 2018 Open Source Monitoring Conference Nuremberg, Germany Why we recommend PMM to our clients Matthias Crauwels
2. © The Pythian Group Inc., 2018 3 Who am I ?
3. © The Pythian Group Inc., 2018 4 4 © The Pythian Group Inc., 2017 Matthias Crauwels • Living in Ghent, Belgium • Bachelor Computer Science • ~20 years Linux user / admin • 10+ years PHP developer • 5+ years MySQL DBA • 1+ year at Pythian as MySQL Database Consultant • Recently promoted to Lead Database Consultant • Father of Leander
4. © The Pythian Group Inc., 2018 5 © The Pythian Group Inc., 2018 5 PYTHIAN Pythian excels at helping businesses around the world use data and the cloud to transform how they compete and win in the data economy. From cloud automation to machine learning, Pythian leads the industry with proven innovative technologies and deep data expertise. For more than 20 years Pythian has built its reputation by delivering solutions to the toughest data challenges faster and better than anyone else.
5. 6 © 2018 Pythian. Confidential AI / ML / BLOCKCHAIN Intelligent analytics and decision making Software autonomy Disruptive data technologies CLOUD MIGRATION & OPERATIONS Plan, Migrate, Manage, Optimize, Innovate Multi-cloud, Hybrid-Cloud, Cloud Native ANALYTIC DATA SYSTEMS Kick AaaS cloud-native, pre-packaged analytics platform Custom analytics platform design, implementation and support services–for on-premises and cloud Data science consulting and implementation services OPERATIONAL DATA SYSTEMS Database services–architecture to ongoing management On prem and in the cloud Oracle, MS SQL, MySQL, Cassandra, MongoDB, Hadoop, AWS/Azure/Google DBaaS
6. 7 © The Pythian Group Inc., 2018 AGENDA 7 • Introduction • What is PMM? • Components & Architecture • Installing & Tuning PMM • Why PMM?
7. 8 © The Pythian Group Inc., 2018 Let’s get started!
8. 9 © The Pythian Group Inc., 2018 © The Pythian Group Inc., 2018 9 PMM is acronym for Percona Monitoring and Management "Percona Monitoring and Management (PMM) is an open-source platform for managing and monitoring MySQL and MongoDB performance. It is developed by Percona in collaboration with experts in the field of managed database services, support and consulting." Current version v.1.15.0 What is PMM?
9. 10 © The Pythian Group Inc., 2018 © The Pythian Group Inc., 2018 10 • Prometheus (v2 as of PMM v1.13.0) Primary data store for time series data • Grafana (v5.1 as of PMM v1.12.0) Visualisation, primary web interface • QAN (Query Analytics) Collect query metrics to analyse query performance over time • Consul Service discovery • Orchestrator Topology management and visualization (not enabled by default since 1.3.0) Components and Architecture (server side)
10. 11 © The Pythian Group Inc., 2018 © The Pythian Group Inc., 2018 11 • Pull based (PMM server pulls from the PMM client) • node_exporter Linux system metrics • mysqld_exporter MySQL and InnoDB metrics • mongodb_exporter MongoDB metrics • proxysql_exporter ProxySQL metrics • Push based (PMM client pushes to the PMM server) • pmm-mysql-queries Pushes MySQL query statistics to QAN • pmm-mongo-queries Pushes MongoDB query statistics to QAN Components and Architecture (client side)
11. 12 © The Pythian Group Inc., 2018 © The Pythian Group Inc., 2018 12
12. 13 © The Pythian Group Inc., 2018 © The Pythian Group Inc., 2018 13 pmm-admin tool for managing the PMM client installation Tool written in Go to configure the connection between PMM client and the PMM server. This is the tool that will update the consul service discovery data. Note: Even with the consul web-endpoint (/consul/) available on the server it is not recommended to make changes to that interface without using the pmm-admin tool. Components and Architecture (client side)
13. 14 © The Pythian Group Inc., 2018 © The Pythian Group Inc., 2018 14 # pmm-admin list ... PMM Server | 192.168.100.1 Client Name | ubuntu-amd64 Client Address | 192.168.200.1 Service manager | linux-systemd ---------------- ----------- ----------- -------- ---------------- -------- SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS ---------------- ----------- ----------- -------- ---------------- -------- linux:metrics mongo-main 42000 YES - mongodb:metrics mongo-main 42003 YES localhost:27017 pmm-admin samples
14. 15 © The Pythian Group Inc., 2018 © The Pythian Group Inc., 2018 15 # pmm-admin check-network PMM Network Status Server Address | 192.168.100.1 Client Address | 192.168.200.1 * System Time NTP Server (0.pool.ntp.org) | 2017-05-03 12:05:38 -0400 EDT PMM Server | 2017-05-03 16:05:38 +0000 GMT PMM Client | 2017-05-03 12:05:38 -0400 EDT PMM Server Time Drift | OK PMM Client Time Drift | OK PMM Client to PMM Server Time Drift | OK ... pmm-admin samples
15. 16 © The Pythian Group Inc., 2018 © The Pythian Group Inc., 2018 16 # pmm-admin check-network PMM Network Status Server Address | 192.168.100.1 Client Address | 192.168.200.1 ... * Connection: Client --> Server -------------------- ------------- SERVER SERVICE STATUS -------------------- ------------- Consul API OK Prometheus API OK Query Analytics API OK Connection duration | 166.689µs Request duration | 364.527µs Full round trip | 531.216µs ... pmm-admin samples
16. 17 © The Pythian Group Inc., 2018 © The Pythian Group Inc., 2018 17 # pmm-admin check-network PMM Network Status Server Address | 192.168.100.1 Client Address | 192.168.200.1 ... * Connection: Client <-- Server ---------------- ----------- -------------------- -------- ---------- --------- SERVICE TYPE NAME REMOTE ENDPOINT STATUS HTTPS/TLS PASSWORD ---------------- ----------- -------------------- -------- ---------- --------- linux:metrics mongo-main 192.168.200.1:42000 OK YES - mongodb:metrics mongo-main 192.168.200.1:42003 PROBLEM YES - pmm-admin samples
17. 18 © The Pythian Group Inc., 2018 © The Pythian Group Inc., 2018 18 3 distribution methods are available • Docker image • OVA package (virtual appliance) • AWS marketplace Most popular in my experience is the Docker approach. Installing PMM server (1)
18. 19 © The Pythian Group Inc., 2018 © The Pythian Group Inc., 2018 19 Recommended to have 4 data volumes • Prometheus: /opt/prometheus/data • Consul: /opt/consul-data • MySQL: /var/lib/mysql • Grafana: /var/lib/grafana At first run PMM will initialize the data directories. It will also set the privileges right at this time. You'll need to run without the volume mappings at this time, then stop the container/appliance and move the data to the appropriate volumes. Installing PMM server (2)
19. 20 © The Pythian Group Inc., 2018 © The Pythian Group Inc., 2018 20 By default, a docker container could access as much memory as the host’s kernel scheduler allows. In an environment where the host is shared with other containers or processes, you may want to limit the amount of memory PMM container can allocate. To do so in a standalone docker environment: $ sudo docker run --memory=4G .. Also by default, prometheus will allocate all memory available to the container for storing the most recently used data chunks. If the memory available to the container is limited, restrict prometheus memory usage for data chunks so it leaves enough memory for MySQL, consul and grafana. In the example below we are setting a limit of 2Gb. $ sudo docker run -e METRICS_MEMORY=2097152 .. Tuning PMM server (1)
20. 21 © The Pythian Group Inc., 2018 © The Pythian Group Inc., 2018 21 By default, prometheus stores time-series data for 30 days, and QAN stores query data for 8 days. To change this defaults use the following variables (in hours): $ sudo docker run -e METRICS_RETENTION=4400h -e QUERIES_RETENTION=4400h .. Of course, total space usage depends on the amount of monitored hosts, the retention period and even the metrics resolution. The default sample rate is 1 second. You could consider increasing METRICS_RESOLUTION (in seconds) to reduce the storage footprint: $ sudo docker run -e METRICS_RESOLUTION=5s Tuning PMM server (2)
21. 22 © The Pythian Group Inc., 2018 Why PMM?
22. 23 © The Pythian Group Inc., 2018 23 How Metrics Monitor is helpful
23. 24 © The Pythian Group Inc., 2018 © The Pythian Group Inc., 2018 24 Historical Monitoring
24. 25 © The Pythian Group Inc., 2018 © The Pythian Group Inc., 2018 25 There are a number of commercial tools available that can offer similar kind of monitoring solution. Usually Software-as-a-Service (SaaS) solutions. • Expensive (mostly pay per server solutions) • Data shipped off-site • Dependent on your internet uplink from your backend servers • Not so customizable • ... Open Source
25. 26 © The Pythian Group Inc., 2018 26 Security • Inside our network • SSL communication • Docker / Appliance • HTTPs Authentication • Custom setup
26. 27 © The Pythian Group Inc., 2018 © The Pythian Group Inc., 2018 27 • Multiple options for installation • PMM Client packages availability • Open Source • Custom dashboards • Possibility to monitor other services too Flexibility
27. 28 © The Pythian Group Inc., 2018 28 Easy to set up
28. 29 © The Pythian Group Inc., 2018 © The Pythian Group Inc., 2018 29 Since Grafana v4.0 there is the option configure alerting in Grafana. Grafana is a core component for PMM PMM has all the metrics let's configure thresholds and push out alerts Alerting
29. 30 © The Pythian Group Inc., 2018 Questions?
30. 31 © The Pythian Group Inc., 2018 Trivia time
31. 32 © The Pythian Group Inc., 2018 32
32. 33 © The Pythian Group Inc., 2018 33
33. 34 © The Pythian Group Inc., 2018 34
34. 35 © The Pythian Group Inc., 2018 35
35. 36 © The Pythian Group Inc., 2018 Thank you!
36. Contact Matthias Crauwels crauwels@pythian.com +1 (613) 565-8696 ext. 1215 We are hiring! http://pythian.com/careers