Introduction to Prometheus

My talk at 1st Singapore Prometheus Meetup Nov 17

Transcript

1. An Introduction to Prometheus Binh Le
2. About Me • Systems Reliability Engineer at Cloudflare • Monitoring a lot of things for ~5 years
3. About Cloudflare
4. About Monitoring System • Prometheus, Nagios, Zabbix, CloudWatch, etc. • Why monitor? ◦ Know if things are broken ◦ Incident response or ad-hoc debuging ◦ Comparing over time ◦ Analyzing trends ◦ Capacity planning
5. What IS a monitoring system? Alert Query & Display Collect & Store
6. Traditional Monitoring System • Server / host as the main unit • Black-box monitoring ◦ check_<thing> • Simple data model and query
7. What have changed? • The Cloud: EC2, GCE, auto-scaling, ... • Containers: Docker, Kubernetes, … • Complex and multi-layered stack • Microservices
8. Prometheus • Embracing white-box monitoring • Built for containers and orchestration • Work with service discovery • Multi-dimensional data model • Powerful query language • Scalable
9. Services Services Prometheus Server Prometheus Components Jobs Alertmanager Services Exporter Pagerduty Slack ... Grafana Web UI Push alerts Notify Scrape metrics Exporter API Storage PromQL
10. Collecting Metrics • Service expose metrics over HTTP ◦ White-box monitoring • Standard metrics exposition format ◦ http_requests_total{path="/", code="200"} 10 • Pull mode ◦ Prometheus scrapes the HTTP endpoints
11. Exporters • Application preferably expose their metrics ◦ cadvisor, kubernetes, etcd, ... • Otherwise, use an exporter ◦ Available for: node (for machine monitoring), haproxy, mysql, postgres, and many more ... ◦ Fairly easy to write your own ◦ Textfile Collector if you don’t want an exporter
12. Querying Metrics - Data Model • Multi-dimensional timeseries
13. Querying Metrics - PromQL • Flexible and powerful query language ◦ http_requests_total{instance="localhost:8000"} ◦ rate(http_requests_total{code=~"5.*"}[5m]) ◦ sum(rate(http_requests_total[5m])) by (path) ◦ max(1 - (predict_linear(node_filesystem_avail[1h], 60 * 60 * 4) / node_filesystem_size)) without(mountpoint) > 0.95
14. Built-in Web UI for query & display metrics
15. Grafana Integration
16. Alert • Alertmanager ◦ Separated process ◦ Routing of alerts ◦ Alert deduplication from HA Prometheus ◦ Managing silences and inhibition rules
17. Alertmanager Web UI
18. Unsee
19. Simple Alert ALERT High_Error_Rate IF error_rate_1m > 10 FOR 10s ANNOTATIONS { summary = "High error rate on {{ $labels.instance }}" }
20. Not So Simple Alert ALERT Disk_Likely_To_Fill_Next_4_Hours IF max(1 - ( predict_linear(node_filesystem_avail[1h], 60 * 60 * 4) / node_filesystem_size )) without(mountpoint) > 0.95 LABELS { notify="hipchat" } ANNOTATIONS { summary = `Device {{ $labels.device }} on {{ $labels.instance }} is likely to be full in the next 4 days`, }
21. Even More Not So Simple Alert ALERT HDFS_Cluster_Is_Unbalanced IF count( abs( (max(hadoop_namenode_FSNamesystemState_CapacityUsed) / min(hadoop_namenode_FSNamesystemState_CapacityTotal)) - ON() GROUP_RIGHT() (hadoop_datanode_fs_DfsUsed / hadoop_datanode_fs_Capacity) ) * 100 > 10 ) LABELS { notify="jira" } ANNOTATIONS { summary = `{{ $value }} HDFS nodes are unbalanced, cluster need rebalancing`, link = "https://<redacted>/ALERT+HDFS_Cluster_Is_Unbalanced", dashboard = "https://grafana.<redacted>/dashboard/db/hadoop-overview", }
22. Demo https://github.com/lebinh/prometheus-meetup-demo
23. Prometheus at Cloudflare
24. If you are interested • Prometheus: A Next-Generation Monitoring System • Monitoring Cloudflare's Planet-Scale Edge Network
25. Thank You @lebinh or binh@cloudflare.com