Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Lightning Talk Prometheus
Search
Mattias Gees
March 01, 2018
Programming
0
59
Lightning Talk Prometheus
Lightning talk I gave internally at Skyscrapers about Prometheus.
Mattias Gees
March 01, 2018
Tweet
Share
More Decks by Mattias Gees
See All by Mattias Gees
A Cloud Native Journey At Scale (Belgium Kubernetes Meetup)
mattiasgees
0
27
Tarmak, why do we need another Kubernetes provisioner?
mattiasgees
1
230
Pod autoscaling with custom-metrics
mattiasgees
0
68
Terraform workspaces
mattiasgees
0
93
Introduction to Ansible
mattiasgees
0
89
Other Decks in Programming
See All in Programming
オブジェクト指向は必要なのか / Is object-oriented needed?
kishida
27
19k
LLMチャットボットのアプリケーション設計Tips
os1ma
4
660
上手な探索的テストとその上達方法について
matsu802
4
650
AppDeveloperCon 2024 EU: Building polyglot developer experiences in 2024
salaboy
0
380
Prepare for Jakarta EE 11 - Performance and Developer Productivity
ivargrimstad
0
500
Deep Dive 大規模システムアーキテクチャ/開発組織エンジニアリング / Deep Dive Large-Scale System Architecture, Development Organization Engineering
nrslib
15
2.9k
Creating Retro-Style Photos Using Swift
ski
1
340
とにかくHTTP3をライトニングに話す / Anyway, I'll talk to Lightning about HTTP3.
seike460
PRO
0
120
品質が高いコードって何?Rev2.1
ickx
1
490
object-oriented-conference-2024
fuwasegu
6
1.9k
TCA魔法学入門🪄
dazy
0
280
PHP 8.3で追加されたjson_validate()を徹底的に深掘りしてみよう
mashirou1234
1
720
Featured
See All Featured
Code Review Best Practice
trishagee
54
15k
Mobile First: as difficult as doing things right
swwweet
215
8.5k
Docker and Python
trallard
33
2.6k
The Power of CSS Pseudo Elements
geoffreycrofte
58
4.9k
Learning to Love Humans: Emotional Interface Design
aarron
266
39k
VelocityConf: Rendering Performance Case Studies
addyosmani
319
23k
Building Adaptive Systems
keathley
29
1.8k
Building Effective Engineering Teams - LeadDev
addyosmani
25
1.8k
10 Git Anti Patterns You Should be Aware of
lemiorhan
644
57k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
12
1.4k
The Illustrated Children's Guide to Kubernetes
chrisshort
28
46k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
657
120k
Transcript
Prometheus Mattias Gees
Components • Prometheus • Alertmanager • Grafana • (Pushgateway)
Architecture
Time series database!
Data Model Notation <metric name>{<label name>=<label value>, ...} Example api_http_requests_total{method="POST",
handler="/messages"}
Metric Types • Counter • Gauge • Histogram • Summary
Metrics gathering /metrics
/metrics # HELP go_gc_duration_seconds A summary of the GC invocation
durations. # TYPE go_gc_duration_seconds summary go_gc_duration_seconds{quantile="0"} 6.4062e-05 go_gc_duration_seconds{quantile="0.25"} 0.000108055 go_gc_duration_seconds{quantile="0.5"} 0.000136816 go_gc_duration_seconds{quantile="0.75"} 0.000248504 go_gc_duration_seconds{quantile="1"} 0.006248475 go_gc_duration_seconds_sum 0.175922918 go_gc_duration_seconds_count 779
Push & Pull • Client library • Exporters • Software
directly • Push based
Client library • Implemented in application • Standard metrics (some)
• Custom metrics
Client library Official • Go • Java or Scala •
Python • Ruby
Client library Unofficial • Bash • Node.JS • PHP •
...
Exporters • Converts application metrics to prometheus formatted Metrics •
3rd party applications • Databases, middleware, messaging systems, APIs, logging, ...
Exporters Examples • MySQL • Mongo • Elasticsearch • Ubiquiti
UniFi exporter • RabbitMQ exporter • Apache exporter
Exporters Examples • PHP-FPM exporter • Nginx exporter • AWS
ECS exporter • AWS SQS exporter • Fluentd exporter • Grok exporter
Software directly • Etcd • Kubernetes • Neo4j • ...
Push • Short lived processes • crons • batch •
Pushgateway
Config
Service discovery • Kubernetes • EC2 • DNS • File
• ...
Example - job_name: infrastructure/k8s-monitor-exporter-kube-controller-manager/0 scrape_interval: 15s scrape_timeout: 10s metrics_path: /metrics
scheme: http kubernetes_sd_configs: - api_server: null role: endpoints namespaces: names: - kube-system bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true relabel_configs: - source_labels: [__meta_kubernetes_service_label_app] separator: ; regex: exporter-kube-controller-manager replacement: $1 action: keep
query
query
Alerts alert: K8SApiserverDown expr: absent(up{job="kubernetes"} == 1) for: 20m labels:
severity: critical annotations: description: No API servers are reachable or all have disappeared from service discovery
Alertmanager
Alertmanager • Global config • Route • Receivers • Templates
Alertmanager Route route: receiver: opsgenie group_by: - job routes: -
receiver: slack match: severity: warning - receiver: opsgenieproxy match: alertname: DeadMansSwitch group_wait: 1s group_interval: 1s repeat_interval: 1s group_wait: 30s group_interval: 5m repeat_interval: 12h
Alertmanager Receivers receivers: - name: opsgenie opsgenie_configs: - send_resolved: true
api_key: <secret> api_host: https://api.opsgenie.com/ message: '{{ template "opsgenie.default.message" . }}' description: '{{ template "opsgenie.default.description" . }}' source: '{{ template "opsgenie.default.source" . }}' tags: sla_test,kubernetes,client_skyscrapers - name: opsgenieproxy webhook_configs: - send_resolved: false url: http://k8s-monitor-opsgenie-heartbeat-proxy/proxy - name: slack slack_configs: - send_resolved: true api_url: <secret> username: skyscrapers-test color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}' title: '{{ template "slack.default.title" . }}' title_link: '{{ template "slack.default.titlelink" . }}' pretext: '{{ template "slack.default.pretext" . }}' text: '{{ template "slack.default.text" . }}' fallback: '{{ template "slack.default.fallback" . }}' icon_emoji: '{{ template "slack.default.iconemoji" . }}' icon_url: '{{ template "slack.default.iconurl" . }}'
Grafana
Grafana
Others • Federation • Storage • Recording rules • Kubernetes
Questions?
More information https://prometheus.io/ Docs are good ;)