Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Lightning Talk Prometheus
Search
Mattias Gees
March 01, 2018
Programming
0
62
Lightning Talk Prometheus
Lightning talk I gave internally at Skyscrapers about Prometheus.
Mattias Gees
March 01, 2018
Tweet
Share
More Decks by Mattias Gees
See All by Mattias Gees
A Cloud Native Journey At Scale (Belgium Kubernetes Meetup)
mattiasgees
0
33
Tarmak, why do we need another Kubernetes provisioner?
mattiasgees
1
260
Pod autoscaling with custom-metrics
mattiasgees
0
73
Terraform workspaces
mattiasgees
0
100
Introduction to Ansible
mattiasgees
0
110
Other Decks in Programming
See All in Programming
Composerが「依存解決」のためにどんな工夫をしているか #phpcon
o0h
PRO
1
250
アンドパッドの Go 勉強会「 gopher 会」とその内容の紹介
andpad
0
290
Benchmark
sysong
0
280
『自分のデータだけ見せたい!』を叶える──Laravel × Casbin で複雑権限をスッキリ解きほぐす 25 分
akitotsukahara
1
600
PostgreSQLのRow Level SecurityをPHPのORMで扱う Eloquent vs Doctrine #phpcon #track2
77web
2
420
Railsアプリケーションと パフォーマンスチューニング ー 秒間5万リクエストの モバイルオーダーシステムを支える事例 ー Rubyセミナー 大阪
falcon8823
4
1k
Go1.25からのGOMAXPROCS
kuro_kurorrr
1
850
すべてのコンテキストを、 ユーザー価値に変える
applism118
2
1.1k
設計やレビューに悩んでいるPHPerに贈る、クリーンなオブジェクト設計の指針たち
panda_program
6
1.8k
PHPでWebSocketサーバーを実装しよう2025
kubotak
0
250
“いい感じ“な定量評価を求めて - Four Keysとアウトカムの間の探求 -
nealle
0
570
Google Agent Development Kit でLINE Botを作ってみた
ymd65536
2
220
Featured
See All Featured
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.8k
What’s in a name? Adding method to the madness
productmarketing
PRO
23
3.5k
Docker and Python
trallard
44
3.5k
What's in a price? How to price your products and services
michaelherold
246
12k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
GraphQLとの向き合い方2022年版
quramy
49
14k
Balancing Empowerment & Direction
lara
1
390
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
48
5.4k
StorybookのUI Testing Handbookを読んだ
zakiyama
30
5.9k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
Why Our Code Smells
bkeepers
PRO
337
57k
Transcript
Prometheus Mattias Gees
Components • Prometheus • Alertmanager • Grafana • (Pushgateway)
Architecture
Time series database!
Data Model Notation <metric name>{<label name>=<label value>, ...} Example api_http_requests_total{method="POST",
handler="/messages"}
Metric Types • Counter • Gauge • Histogram • Summary
Metrics gathering /metrics
/metrics # HELP go_gc_duration_seconds A summary of the GC invocation
durations. # TYPE go_gc_duration_seconds summary go_gc_duration_seconds{quantile="0"} 6.4062e-05 go_gc_duration_seconds{quantile="0.25"} 0.000108055 go_gc_duration_seconds{quantile="0.5"} 0.000136816 go_gc_duration_seconds{quantile="0.75"} 0.000248504 go_gc_duration_seconds{quantile="1"} 0.006248475 go_gc_duration_seconds_sum 0.175922918 go_gc_duration_seconds_count 779
Push & Pull • Client library • Exporters • Software
directly • Push based
Client library • Implemented in application • Standard metrics (some)
• Custom metrics
Client library Official • Go • Java or Scala •
Python • Ruby
Client library Unofficial • Bash • Node.JS • PHP •
...
Exporters • Converts application metrics to prometheus formatted Metrics •
3rd party applications • Databases, middleware, messaging systems, APIs, logging, ...
Exporters Examples • MySQL • Mongo • Elasticsearch • Ubiquiti
UniFi exporter • RabbitMQ exporter • Apache exporter
Exporters Examples • PHP-FPM exporter • Nginx exporter • AWS
ECS exporter • AWS SQS exporter • Fluentd exporter • Grok exporter
Software directly • Etcd • Kubernetes • Neo4j • ...
Push • Short lived processes • crons • batch •
Pushgateway
Config
Service discovery • Kubernetes • EC2 • DNS • File
• ...
Example - job_name: infrastructure/k8s-monitor-exporter-kube-controller-manager/0 scrape_interval: 15s scrape_timeout: 10s metrics_path: /metrics
scheme: http kubernetes_sd_configs: - api_server: null role: endpoints namespaces: names: - kube-system bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true relabel_configs: - source_labels: [__meta_kubernetes_service_label_app] separator: ; regex: exporter-kube-controller-manager replacement: $1 action: keep
query
query
Alerts alert: K8SApiserverDown expr: absent(up{job="kubernetes"} == 1) for: 20m labels:
severity: critical annotations: description: No API servers are reachable or all have disappeared from service discovery
Alertmanager
Alertmanager • Global config • Route • Receivers • Templates
Alertmanager Route route: receiver: opsgenie group_by: - job routes: -
receiver: slack match: severity: warning - receiver: opsgenieproxy match: alertname: DeadMansSwitch group_wait: 1s group_interval: 1s repeat_interval: 1s group_wait: 30s group_interval: 5m repeat_interval: 12h
Alertmanager Receivers receivers: - name: opsgenie opsgenie_configs: - send_resolved: true
api_key: <secret> api_host: https://api.opsgenie.com/ message: '{{ template "opsgenie.default.message" . }}' description: '{{ template "opsgenie.default.description" . }}' source: '{{ template "opsgenie.default.source" . }}' tags: sla_test,kubernetes,client_skyscrapers - name: opsgenieproxy webhook_configs: - send_resolved: false url: http://k8s-monitor-opsgenie-heartbeat-proxy/proxy - name: slack slack_configs: - send_resolved: true api_url: <secret> username: skyscrapers-test color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}' title: '{{ template "slack.default.title" . }}' title_link: '{{ template "slack.default.titlelink" . }}' pretext: '{{ template "slack.default.pretext" . }}' text: '{{ template "slack.default.text" . }}' fallback: '{{ template "slack.default.fallback" . }}' icon_emoji: '{{ template "slack.default.iconemoji" . }}' icon_url: '{{ template "slack.default.iconurl" . }}'
Grafana
Grafana
Others • Federation • Storage • Recording rules • Kubernetes
Questions?
More information https://prometheus.io/ Docs are good ;)