Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Lightning Talk Prometheus
Search
Mattias Gees
March 01, 2018
Programming
0
63
Lightning Talk Prometheus
Lightning talk I gave internally at Skyscrapers about Prometheus.
Mattias Gees
March 01, 2018
Tweet
Share
More Decks by Mattias Gees
See All by Mattias Gees
A Cloud Native Journey At Scale (Belgium Kubernetes Meetup)
mattiasgees
0
34
Tarmak, why do we need another Kubernetes provisioner?
mattiasgees
1
260
Pod autoscaling with custom-metrics
mattiasgees
0
74
Terraform workspaces
mattiasgees
0
100
Introduction to Ansible
mattiasgees
0
110
Other Decks in Programming
See All in Programming
Zendeskのチケットを Amazon Bedrockで 解析した
ryokosuge
3
320
詳解!defer panic recover のしくみ / Understanding defer, panic, and recover
convto
0
250
アプリの "かわいい" を支えるアニメーションツールRiveについて
uetyo
0
280
Processing Gem ベースの、2D レトロゲームエンジンの開発
tokujiros
2
130
Android 16 × Jetpack Composeで縦書きテキストエディタを作ろう / Vertical Text Editor with Compose on Android 16
cc4966
2
270
「待たせ上手」なスケルトンスクリーン、 そのUXの裏側
teamlab
PRO
0
570
Swift Updates - Learn Languages 2025
koher
2
520
JSONataを使ってみよう Step Functionsが楽しくなる実践テクニック #devio2025
dafujii
1
650
意外と簡単!?フロントエンドでパスキー認証を実現する WebAuthn
teamlab
PRO
2
780
アセットのコンパイルについて
ojun9
0
130
go test -json そして testing.T.Attr / Kyoto.go #63
utgwkk
3
320
私の後悔をAWS DMSで解決した話
hiramax
4
210
Featured
See All Featured
How STYLIGHT went responsive
nonsquared
100
5.8k
We Have a Design System, Now What?
morganepeng
53
7.8k
Building a Modern Day E-commerce SEO Strategy
aleyda
43
7.6k
Docker and Python
trallard
46
3.6k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
Code Review Best Practice
trishagee
71
19k
The Art of Programming - Codeland 2020
erikaheidi
56
13k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
36
2.5k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.9k
VelocityConf: Rendering Performance Case Studies
addyosmani
332
24k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Large-scale JavaScript Application Architecture
addyosmani
513
110k
Transcript
Prometheus Mattias Gees
Components • Prometheus • Alertmanager • Grafana • (Pushgateway)
Architecture
Time series database!
Data Model Notation <metric name>{<label name>=<label value>, ...} Example api_http_requests_total{method="POST",
handler="/messages"}
Metric Types • Counter • Gauge • Histogram • Summary
Metrics gathering /metrics
/metrics # HELP go_gc_duration_seconds A summary of the GC invocation
durations. # TYPE go_gc_duration_seconds summary go_gc_duration_seconds{quantile="0"} 6.4062e-05 go_gc_duration_seconds{quantile="0.25"} 0.000108055 go_gc_duration_seconds{quantile="0.5"} 0.000136816 go_gc_duration_seconds{quantile="0.75"} 0.000248504 go_gc_duration_seconds{quantile="1"} 0.006248475 go_gc_duration_seconds_sum 0.175922918 go_gc_duration_seconds_count 779
Push & Pull • Client library • Exporters • Software
directly • Push based
Client library • Implemented in application • Standard metrics (some)
• Custom metrics
Client library Official • Go • Java or Scala •
Python • Ruby
Client library Unofficial • Bash • Node.JS • PHP •
...
Exporters • Converts application metrics to prometheus formatted Metrics •
3rd party applications • Databases, middleware, messaging systems, APIs, logging, ...
Exporters Examples • MySQL • Mongo • Elasticsearch • Ubiquiti
UniFi exporter • RabbitMQ exporter • Apache exporter
Exporters Examples • PHP-FPM exporter • Nginx exporter • AWS
ECS exporter • AWS SQS exporter • Fluentd exporter • Grok exporter
Software directly • Etcd • Kubernetes • Neo4j • ...
Push • Short lived processes • crons • batch •
Pushgateway
Config
Service discovery • Kubernetes • EC2 • DNS • File
• ...
Example - job_name: infrastructure/k8s-monitor-exporter-kube-controller-manager/0 scrape_interval: 15s scrape_timeout: 10s metrics_path: /metrics
scheme: http kubernetes_sd_configs: - api_server: null role: endpoints namespaces: names: - kube-system bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true relabel_configs: - source_labels: [__meta_kubernetes_service_label_app] separator: ; regex: exporter-kube-controller-manager replacement: $1 action: keep
query
query
Alerts alert: K8SApiserverDown expr: absent(up{job="kubernetes"} == 1) for: 20m labels:
severity: critical annotations: description: No API servers are reachable or all have disappeared from service discovery
Alertmanager
Alertmanager • Global config • Route • Receivers • Templates
Alertmanager Route route: receiver: opsgenie group_by: - job routes: -
receiver: slack match: severity: warning - receiver: opsgenieproxy match: alertname: DeadMansSwitch group_wait: 1s group_interval: 1s repeat_interval: 1s group_wait: 30s group_interval: 5m repeat_interval: 12h
Alertmanager Receivers receivers: - name: opsgenie opsgenie_configs: - send_resolved: true
api_key: <secret> api_host: https://api.opsgenie.com/ message: '{{ template "opsgenie.default.message" . }}' description: '{{ template "opsgenie.default.description" . }}' source: '{{ template "opsgenie.default.source" . }}' tags: sla_test,kubernetes,client_skyscrapers - name: opsgenieproxy webhook_configs: - send_resolved: false url: http://k8s-monitor-opsgenie-heartbeat-proxy/proxy - name: slack slack_configs: - send_resolved: true api_url: <secret> username: skyscrapers-test color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}' title: '{{ template "slack.default.title" . }}' title_link: '{{ template "slack.default.titlelink" . }}' pretext: '{{ template "slack.default.pretext" . }}' text: '{{ template "slack.default.text" . }}' fallback: '{{ template "slack.default.fallback" . }}' icon_emoji: '{{ template "slack.default.iconemoji" . }}' icon_url: '{{ template "slack.default.iconurl" . }}'
Grafana
Grafana
Others • Federation • Storage • Recording rules • Kubernetes
Questions?
More information https://prometheus.io/ Docs are good ;)