Lightning Talk Prometheus

Lightning talk I gave internally at Skyscrapers about Prometheus.

Transcript

1. Prometheus Mattias Gees
2. Components • Prometheus • Alertmanager • Grafana • (Pushgateway)
3. Architecture
4. Time series database!
5. Data Model Notation <metric name>{<label name>=<label value>, ...} Example api_http_requests_total{method="POST", handler="/messages"}
6. Metric Types • Counter • Gauge • Histogram • Summary
7. Metrics gathering /metrics
8. /metrics # HELP go_gc_duration_seconds A summary of the GC invocation durations. # TYPE go_gc_duration_seconds summary go_gc_duration_seconds{quantile="0"} 6.4062e-05 go_gc_duration_seconds{quantile="0.25"} 0.000108055 go_gc_duration_seconds{quantile="0.5"} 0.000136816 go_gc_duration_seconds{quantile="0.75"} 0.000248504 go_gc_duration_seconds{quantile="1"} 0.006248475 go_gc_duration_seconds_sum 0.175922918 go_gc_duration_seconds_count 779
9. Push & Pull • Client library • Exporters • Software directly • Push based
10. Client library • Implemented in application • Standard metrics (some) • Custom metrics
11. Client library Official • Go • Java or Scala • Python • Ruby
12. Client library Unofficial • Bash • Node.JS • PHP • ...
13. Exporters • Converts application metrics to prometheus formatted Metrics • 3rd party applications • Databases, middleware, messaging systems, APIs, logging, ...
14. Exporters Examples • MySQL • Mongo • Elasticsearch • Ubiquiti UniFi exporter • RabbitMQ exporter • Apache exporter
15. Exporters Examples • PHP-FPM exporter • Nginx exporter • AWS ECS exporter • AWS SQS exporter • Fluentd exporter • Grok exporter
16. Software directly • Etcd • Kubernetes • Neo4j • ...
17. Push • Short lived processes • crons • batch • Pushgateway
18. Config
19. Service discovery • Kubernetes • EC2 • DNS • File • ...
20. Example - job_name: infrastructure/k8s-monitor-exporter-kube-controller-manager/0 scrape_interval: 15s scrape_timeout: 10s metrics_path: /metrics scheme: http kubernetes_sd_configs: - api_server: null role: endpoints namespaces: names: - kube-system bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true relabel_configs: - source_labels: [__meta_kubernetes_service_label_app] separator: ; regex: exporter-kube-controller-manager replacement: $1 action: keep
21. query
22. query
23. Alerts alert: K8SApiserverDown expr: absent(up{job="kubernetes"} == 1) for: 20m labels: severity: critical annotations: description: No API servers are reachable or all have disappeared from service discovery
24. Alertmanager
25. Alertmanager • Global config • Route • Receivers • Templates
26. Alertmanager Route route: receiver: opsgenie group_by: - job routes: - receiver: slack match: severity: warning - receiver: opsgenieproxy match: alertname: DeadMansSwitch group_wait: 1s group_interval: 1s repeat_interval: 1s group_wait: 30s group_interval: 5m repeat_interval: 12h
27. Alertmanager Receivers receivers: - name: opsgenie opsgenie_configs: - send_resolved: true api_key: <secret> api_host: https://api.opsgenie.com/ message: '{{ template "opsgenie.default.message" . }}' description: '{{ template "opsgenie.default.description" . }}' source: '{{ template "opsgenie.default.source" . }}' tags: sla_test,kubernetes,client_skyscrapers - name: opsgenieproxy webhook_configs: - send_resolved: false url: http://k8s-monitor-opsgenie-heartbeat-proxy/proxy - name: slack slack_configs: - send_resolved: true api_url: <secret> username: skyscrapers-test color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}' title: '{{ template "slack.default.title" . }}' title_link: '{{ template "slack.default.titlelink" . }}' pretext: '{{ template "slack.default.pretext" . }}' text: '{{ template "slack.default.text" . }}' fallback: '{{ template "slack.default.fallback" . }}' icon_emoji: '{{ template "slack.default.iconemoji" . }}' icon_url: '{{ template "slack.default.iconurl" . }}'
28. Grafana
29. Grafana
30. Others • Federation • Storage • Recording rules • Kubernetes
31. Questions?
32. More information https://prometheus.io/ Docs are good ;)