Monitoring Applications with Prometheus and Grafana

Conference: https://www.womentech.net/women-tech-conference
Talk : https://www.youtube.com/watch?v=ygk7yBnljsg&feature=youtu.be

Session: Monitoring Applications with Prometheus and Grafana
"If you can't measure it, you can't improve it."
Monitoring an application’s health and metrics makes it possible to manage it in a better way and notice unoptimized behavior.
With this session, we will aim to learn about how to start monitoring your technology infrastructure with Prometheus and Grafana and learn about how to derive metrics that matter to reliable and performant operations of your infrastructure.

Transcript

1. Nancy Chauhan Software Developer Monitoring Applications with Prometheus and Grafana Women Tech Global Conference 2020
2. You can find me at : • Github : Nancy-Chauhan • Twitter : _nancychauhan • Website : nancychauhan.in Software Developer at Grofers, India Women Tech Global Conference 2020 About me
3. Agenda • Need for services and systems monitoring • Different approaches to monitoring and observability. • Prometheus - landscape and basic architecture. • Monitoring your application – What to monitor – How to monitor – Visualizing – Alerting • Prometheus in Real World Women Tech Global Conference 2020
4. What is broken ? and Why ? Women Tech Global Conference 2020
5. Why monitor ? • Know when things go wrong • Be able to debug and gain insight • Trending to see changes over time, and drive technical/business decisions • To feed into other systems/processes Reality : systems are complicated and break a lot ! Women Tech Global Conference 2020
6. Potential Problems • Software Bug -> Request/ Application errors • Low memory utilization -> Money wasted • High CPU utilization • Disk Full -> No new data stored , Slow performance • Network Outage -> Services cannot communicate Women Tech Global Conference 2020
7. Types of Monitoring • Check Based monitoring ( health checks) ◦ Nagios, Icinga • Logs/ Events ◦ Loki, Splunk, Elasticsearch, • Metric & Time Series ◦ Prometheus, StatsD, InfluxDB • Request Tracing ◦ Zipkin, Jaegar Women Tech Global Conference 2020
8. What is Prometheus ? Women Tech Global Conference 2020
9. Architecture Diagram Women Tech Global Conference 2020
10. Women Tech Global Conference 2020 https://prometheus.io/docs/introduction/overview/
11. Prometheus Architecture Women Tech Global Conference 2020
12. Prometheus Architecture Women Tech Global Conference 2020
13. Prometheus Architecture Women Tech Global Conference 2020
14. Prometheus Architecture Women Tech Global Conference 2020
15. Prometheus Architecture Women Tech Global Conference 2020
16. Favourite Features • Dimensional data model • Powerful query language (PromQL) • Simple & efficient server • Support for Service discovery Integration Women Tech Global Conference 2020
17. What is Prometheus ? Women Tech Global Conference 2020 Monitoring your Application IIlustration by Asif Jamal
18. What to monitor ? • Request and response statuses – success or failure • Timings of transactions – external calls, data processing • Request and response timings • Application runtime metrics – memory, cpu, gc, etc. Women Tech Global Conference 2020
19. What to monitor ? (contd.) • Environment metrics – container metrics, load factor of container / VM, network IO bytes, disk IO etc. • Event counts – no of time something happened like cache miss, exceptions • Queue sizes, Cache sizes Women Tech Global Conference 2020
20. How to monitor • Using your framework’s support for exporting metrics into prometheus • Using prometheus client libraries directly • Using node exporter to export metrics for your VM • Prometheus operator for Kubernetes Women Tech Global Conference 2020
21. How to monitor (contd.) • Identify what metrics you are interested in • Categorize the data type for the metrics – whether its gauge or counter or histogram • Write code to export the metric • Setup prometheus to scrape metrics from your application Women Tech Global Conference 2020
22. Visualizing Although Prometheus provides basic visualisation, Grafana provides a full framework for : • sharing dashboards, • creating advanced queries and graphs, • and allowing for sharing and reuse of those dashboards. Women Tech Global Conference 2020
23. Women Tech Global Conference 2020 So that you can go from this:
24. Women Tech Global Conference 2020 To this :
25. Alerting • To truly protect applications, alerting teams is necessary when something goes wrong. • High level Tasks : • Setup an instance of Alert manager • Configure Prometheus instance to use Alert manager • Create alert rules in alert manager configuration Women Tech Global Conference 2020
26. Prometheus in Real World Women Tech Global Conference 2020
27. Monitoring Microservices • Request and response times of the service • Success and failure counters • Business metrics like order size average, A/B testing metrics • Database transactions and external call timings • Kubernetes pod metrics Women Tech Global Conference 2020
28. Prometheus as a scraper • Prometheus is simple and leaves features like analysis and long term storage to extensions • Many organizations use Prometheus to collect metrics for processing and storing metrics in a different system such as Cortex, Uber M3, Thanos, CloudWatch etc. Women Tech Global Conference 2020
29. Prometheus exporters • Sometimes a service you want to monitor isn’t prometheus ready • Define exporters to “export” metrics from a non-prometheus aware monitoring system into Prometheus • Prometheus project provides JMX exporter, StatsD exporter and push gateway • Jenkins exporter exports metrics from Jenkins builds to prometheus Women Tech Global Conference 2020
30. References • https://www.youtube.com/watch?v=5O1djJ13gRU • https://www.youtube.com/watch?v=D09x0eR4vu4&t=176s Women Tech Global Conference 2020
31. Thank you Women Tech Global Conference 2020