Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Lightning Talk Prometheus
Search
Mattias Gees
March 01, 2018
Programming
0
63
Lightning Talk Prometheus
Lightning talk I gave internally at Skyscrapers about Prometheus.
Mattias Gees
March 01, 2018
Tweet
Share
More Decks by Mattias Gees
See All by Mattias Gees
A Cloud Native Journey At Scale (Belgium Kubernetes Meetup)
mattiasgees
0
34
Tarmak, why do we need another Kubernetes provisioner?
mattiasgees
1
260
Pod autoscaling with custom-metrics
mattiasgees
0
74
Terraform workspaces
mattiasgees
0
100
Introduction to Ansible
mattiasgees
0
110
Other Decks in Programming
See All in Programming
AI時代のUIはどこへ行く?
yusukebe
16
8.6k
OSS開発者という働き方
andpad
5
1.7k
Deep Dive into Kotlin Flow
jmatsu
1
190
為你自己學 Python - 冷知識篇
eddie
1
350
ProxyによるWindow間RPC機構の構築
syumai
3
1.1k
RDoc meets YARD
okuramasafumi
4
160
はじめてのMaterial3 Expressive
ym223
2
180
Namespace and Its Future
tagomoris
6
700
「手軽で便利」に潜む罠。 Popover API を WCAG 2.2の視点で安全に使うには
taitotnk
0
820
ぬるぬる動かせ! Riveでアニメーション実装🐾
kno3a87
1
200
テストコードはもう書かない:JetBrains AI Assistantに委ねる非同期処理のテスト自動設計・生成
makun
0
160
go test -json そして testing.T.Attr / Kyoto.go #63
utgwkk
3
280
Featured
See All Featured
Balancing Empowerment & Direction
lara
3
610
Designing for Performance
lara
610
69k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
187
55k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.6k
A Modern Web Designer's Workflow
chriscoyier
696
190k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
50k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Building a Modern Day E-commerce SEO Strategy
aleyda
43
7.6k
The World Runs on Bad Software
bkeepers
PRO
70
11k
Build your cross-platform service in a week with App Engine
jlugia
231
18k
How STYLIGHT went responsive
nonsquared
100
5.8k
Transcript
Prometheus Mattias Gees
Components • Prometheus • Alertmanager • Grafana • (Pushgateway)
Architecture
Time series database!
Data Model Notation <metric name>{<label name>=<label value>, ...} Example api_http_requests_total{method="POST",
handler="/messages"}
Metric Types • Counter • Gauge • Histogram • Summary
Metrics gathering /metrics
/metrics # HELP go_gc_duration_seconds A summary of the GC invocation
durations. # TYPE go_gc_duration_seconds summary go_gc_duration_seconds{quantile="0"} 6.4062e-05 go_gc_duration_seconds{quantile="0.25"} 0.000108055 go_gc_duration_seconds{quantile="0.5"} 0.000136816 go_gc_duration_seconds{quantile="0.75"} 0.000248504 go_gc_duration_seconds{quantile="1"} 0.006248475 go_gc_duration_seconds_sum 0.175922918 go_gc_duration_seconds_count 779
Push & Pull • Client library • Exporters • Software
directly • Push based
Client library • Implemented in application • Standard metrics (some)
• Custom metrics
Client library Official • Go • Java or Scala •
Python • Ruby
Client library Unofficial • Bash • Node.JS • PHP •
...
Exporters • Converts application metrics to prometheus formatted Metrics •
3rd party applications • Databases, middleware, messaging systems, APIs, logging, ...
Exporters Examples • MySQL • Mongo • Elasticsearch • Ubiquiti
UniFi exporter • RabbitMQ exporter • Apache exporter
Exporters Examples • PHP-FPM exporter • Nginx exporter • AWS
ECS exporter • AWS SQS exporter • Fluentd exporter • Grok exporter
Software directly • Etcd • Kubernetes • Neo4j • ...
Push • Short lived processes • crons • batch •
Pushgateway
Config
Service discovery • Kubernetes • EC2 • DNS • File
• ...
Example - job_name: infrastructure/k8s-monitor-exporter-kube-controller-manager/0 scrape_interval: 15s scrape_timeout: 10s metrics_path: /metrics
scheme: http kubernetes_sd_configs: - api_server: null role: endpoints namespaces: names: - kube-system bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true relabel_configs: - source_labels: [__meta_kubernetes_service_label_app] separator: ; regex: exporter-kube-controller-manager replacement: $1 action: keep
query
query
Alerts alert: K8SApiserverDown expr: absent(up{job="kubernetes"} == 1) for: 20m labels:
severity: critical annotations: description: No API servers are reachable or all have disappeared from service discovery
Alertmanager
Alertmanager • Global config • Route • Receivers • Templates
Alertmanager Route route: receiver: opsgenie group_by: - job routes: -
receiver: slack match: severity: warning - receiver: opsgenieproxy match: alertname: DeadMansSwitch group_wait: 1s group_interval: 1s repeat_interval: 1s group_wait: 30s group_interval: 5m repeat_interval: 12h
Alertmanager Receivers receivers: - name: opsgenie opsgenie_configs: - send_resolved: true
api_key: <secret> api_host: https://api.opsgenie.com/ message: '{{ template "opsgenie.default.message" . }}' description: '{{ template "opsgenie.default.description" . }}' source: '{{ template "opsgenie.default.source" . }}' tags: sla_test,kubernetes,client_skyscrapers - name: opsgenieproxy webhook_configs: - send_resolved: false url: http://k8s-monitor-opsgenie-heartbeat-proxy/proxy - name: slack slack_configs: - send_resolved: true api_url: <secret> username: skyscrapers-test color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}' title: '{{ template "slack.default.title" . }}' title_link: '{{ template "slack.default.titlelink" . }}' pretext: '{{ template "slack.default.pretext" . }}' text: '{{ template "slack.default.text" . }}' fallback: '{{ template "slack.default.fallback" . }}' icon_emoji: '{{ template "slack.default.iconemoji" . }}' icon_url: '{{ template "slack.default.iconurl" . }}'
Grafana
Grafana
Others • Federation • Storage • Recording rules • Kubernetes
Questions?
More information https://prometheus.io/ Docs are good ;)