Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ユーザ目線でのPrometheus #mackerel_ug /monitoring-prom...
Search
Manabu Matsuzaki
October 27, 2017
Technology
1
3.6k
ユーザ目線でのPrometheus #mackerel_ug /monitoring-prometheus
モニタリング勉強会の発表資料です
Manabu Matsuzaki
October 27, 2017
Tweet
Share
More Decks by Manabu Matsuzaki
See All by Manabu Matsuzaki
Spring BootユーザのためのArmeria入門 #jsug / Introduce to Armeria for Spring users
matsumana
0
2.8k
Canary Release with Argo Rollouts #ふくばねてす / canary-release-with-argo-rollouts
matsumana
1
1.2k
Getting started Central Dogma with Golang #fukuokago #umedago / getting-started-central-dogma-with-golang
matsumana
0
890
Micrometer入門 #javaq / introduce-to-micrometer
matsumana
1
2.9k
ArmeriaとCentral Dogmaから学ぶ、マイクロサービスに必要な機能 #edayfuk / lean-from-armeria-and-central-dogma
matsumana
0
4.3k
SREcon19 Americas 参加レポート #srefukuoka / srecon19-americas-report
matsumana
0
880
SRE入門 & チームで取り組んでいるSRE #srefukuoka / introduce-to-sre
matsumana
0
1.3k
Introduce to Armeria and Central Dogma #GWD_Nulab / introduce-to-armeria-and-central-dogma
matsumana
0
560
Connector/JでMaster/Slave Replication構成のMySQLに接続する #mysql_casual_fukuoka /connector-j-master-slave-replication
matsumana
0
1.5k
Other Decks in Technology
See All in Technology
「良さそう」と「とても良い」の間には 「良さそうだがホンマか」がたくさんある / 2025.07.01 LLM品質Night
smiyawaki0820
1
500
Core Audio tapを使ったリアルタイム音声処理のお話
yuta0306
0
180
PO初心者が考えた ”POらしさ”
nb_rady
0
190
Delegating the chores of authenticating users to Keycloak
ahus1
0
140
Claude Code に プロジェクト管理やらせたみた
unson
5
2.4k
KiCadでPad on Viaの基板作ってみた
iotengineer22
0
290
Tech-Verse 2025 Keynote
lycorptech_jp
PRO
0
1.8k
【5分でわかる】セーフィー エンジニア向け会社紹介
safie_recruit
0
27k
Should Our Project Join the CNCF? (Japanese Recap)
whywaita
PRO
0
330
KubeCon + CloudNativeCon Japan 2025 Recap
ren510dev
1
370
無意味な開発生産性の議論から抜け出すための予兆検知とお金とAI
i35_267
3
12k
Lazy application authentication with Tailscale
bluehatbrit
0
170
Featured
See All Featured
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
48
5.4k
Stop Working from a Prison Cell
hatefulcrawdad
270
21k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
44
2.4k
Writing Fast Ruby
sferik
628
62k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
A better future with KSS
kneath
239
17k
Practical Orchestrator
shlominoach
188
11k
How to Ace a Technical Interview
jacobian
277
23k
KATA
mclloyd
30
14k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
GraphQLの誤解/rethinking-graphql
sonatard
71
11k
Transcript
ϢʔβઢͰͷPrometheus ϞχλϦϯάษڧձ 2017/10/27 @matsumana
ࣗݾհ • ໊લɿ দ࡚ ֶ • ॴଐɿ LINE Fukuokaגࣜձࣾ ։ൃࣨ
• Twitterɿ @matsumana
ΞδΣϯμ • ࣮ࡍʹͲ͏͍ͬͨϝτϦΫεΛऔ͍ͬͯΔͷ͔ • exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε • Ξϥʔτϧʔϧͷॻ͖ํͷྫ • ·ͱΊ ※࣌ؒͷ߹্ɺPrometheusೖతͳࣄ͠·ͤΜ
લఏ • ݱࡏ༻͍ͯ͠Δόʔδϣϯ • Prometheus 1.8.0
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • node_exporter • loading • CPU༻ • ϝϞϦ༻
• εϫοϓ • slab • context switch
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • node_exporter • σΟεΫI/O • σΟεΫ༻ • inode
• TCPଓ • ESTABLISHED • TIME_WAIT • ωοτϫʔΫ τϥϑΟοΫ
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • apache_exporter • mod_statusϞδϡʔϧͰऔΕΔϝτϦΫε • nginx_exporter • ngx_http_stub_status_moduleϞδϡʔϧͰऔΕΔϝτϦΫε
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • plack_exporter • Plack::Middleware::ServerStatus::LiteͰऔΕΔϝτϦΫε
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • jmx_exporter • JVMͷڞ௨తͳϝτϦΫε • Heap • Metaspace
• GC • Thread • Code cache
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • jmx_exporter • ϛυϧΣΞ/ϥΠϒϥϦݻ༗ͷϝτϦΫε • Tomcat • Jetty
• HikariCP • Caffeine • etc…
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • memcached_exporter • ΩϟογϡαΠζ • Ωϟογϡ • Ωϟογϡώοτ
• operation(get/set) • Eviction
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • mysqld_exporter • QPS • Slow query •
Replicaton delay • binlog • connection • InnoDBؔ࿈
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • redis_exporter • ༻ϝϞϦ • Key • Eviction
• Slow query
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • elasticsearch_exporter • Cluster health • Document(primary shardͷΈ)
• DocumentαΠζ(primary shardͷΈ) • Thread pool queue • Thread pool rejected • Circuit Breaker tripped
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • elasticsearch_exporter • Fielddata cache size • Query
cache size • Query cache hit,miss,total • Request cache size
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • fluent-plugin-prometheus • ֤BufferedOutput pluginͷϝτϦΫε • buffer_queue_length •
buffer_total_bytes • retry_count
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • fluentd_exporter • fluentdϓϩηε୯ҐͷCPU༻ • fluentdϓϩηε୯ҐͷϝϞϦ༻ • td-agent_exporter
• fluentdϓϩηε୯ҐͷCPU༻ • fluentdϓϩηε୯ҐͷϝϞϦ༻ • fluent-agent-lite_exporter • fluentdϓϩηε୯ҐͷCPU༻ • fluentdϓϩηε୯ҐͷϝϞϦ༻
exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε • https://prometheus.io/docs/instrumenting/exporters/ • https://github.com/prometheus/prometheus/wiki/Default- port-allocations • άάΔ • ͕ࣗඞཁͳϝτϦΫε͕export͞Ε͍ͯͳΕPR͢Δ
·ͣɺଞͷਓ͕࡞ͬͨͷ͕ແ͍͔୳ͯ͠ΈΔ
• ΦϑΟγϟϧͷΨΠυϥΠϯ https://prometheus.io/docs/instrumenting/writing_exporters/ exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε ࣗͰ࡞Δ߹
• ΦϑΟγϟϧͷclient library https://prometheus.io/docs/instrumenting/clientlibs/ • ΦϑΟγϟϧgolang, Java, Python, RubyͷΈ •
Third partyʹPHPͳͲ͋Δ exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε ࣗͰ࡞Δ߹
• exportergoͰॻ͘ͷ͕ྑͦ͞͏͚ͩͲҙ͋Δ • Linux Kernel 2.6.23ະຬgolangࣗମ͕αϙʔτͯ͠ͳ͍ CentOS5ܥͩͱgolangͷexporterਖ਼͘͠ಈ͔ͳ͍͔ https://github.com/golang/go/wiki/MinimumRequirements exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε ࣗͰ࡞Δ߹
exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε ΦϑΟγϟϧΨΠυϥΠϯʹهࡌ͞Ε͍ͯΔͷΛҰ෦͝հ • ଞͷexporterͰΘΕͯͳ͍portΛ͏ɻ࡞ͬͨΒهࡌ͠Α͏ https://github.com/prometheus/prometheus/wiki/Default-port- allocations • ϝτϦΫε໊ɺϥϕϧ໊ͷϕετϓϥΫςΟεʹै͏ https://prometheus.io/docs/practices/naming/ •
exporterͰϝτϦΫεΛܭࢉ͠ͳ͍ɻPromQLͰܭࢉ͢Δ • ྫʣΩϟογϡώοτͳͲ
exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε • exporterͷόʔδϣϯ൪߸ΛϝτϦΫεͱͯ͠export͢Δ • ࢹରͷαʔόexporterͷ͕૿͑ΔͱɺͲͷόʔδϣ ϯ͕σϓϩΠ͞Ε͍ͯΔ͔Θ͔Βͳ͘ͳΔ • ͪͳΈʹɺPrometheusຊମҎԼͷΑ͏ͳϝτϦΫεΛ export͍ͯ͠Δ prometheus_build_info{branch=“master",goversion="go1.8.3",revision="3afb3fffa3a29
c3de865e1172fb740442e9d0133",version="1.7.1"} 1 ΦϑΟγϟϧΨΠυϥΠϯʹهࡌ͞Ε͍ͯΔͷΛҰ෦͝հ
exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε ΦϑΟγϟϧΨΠυϥΠϯʹهࡌ͞Ε͍ͯΔͷΛҰ෦͝հ • ̋̋̋_up ϝτϦΫεΛexport͢Δ • ྫ͑ɺMySQLͷ߹mysql_upͱ͍͏ϝτϦΫε • ࢹର͔ΒϝτϦΫεऔಘޭͳΒ1ɺࣦഊͳΒ0Λexport •
ཧ༝ޙड़
Ξϥʔτϧʔϧͷॻ͖ํͷྫ • ྫ1) exporter͕མ͍ͪͯͳ͍͔Ͳ͏͔ࢹ͢Δ • ྫ2) exporertͷࢹର͕མ͍ͪͯͳ͍͔Ͳ͏͔ࢹ͢Δ
• ྫ3) dailyόον͕ਖ਼ৗʹಈ͍͍ͯΔ͔Ͳ͏͔ࢹ͢Δ
Ξϥʔτϧʔϧͷॻ͖ํͷྫ • Prometheusαʔό͕ࣗɺશͯͷexporterʹ͍ͭͯ upϝτϦΫεΛه͍ͯ͠Δ https://prometheus.io/docs/concepts/jobs_instances/ ྫʣnode_exporterͷ߹ up{instance="host00:9100", job="node"} 1
ྫ1ʣexporter͕མ͍ͪͯͳ͍͔Ͳ͏͔ࢹ͢Δ
Ξϥʔτϧʔϧͷॻ͖ํͷྫ ALERT ExporterDown IF up == 0 FOR 1m LABELS
{severity="major"} ANNOTATIONS { description="{{ $labels.instance }} of job {{ $labels.job }} has been down” } ྫ1ʣexporter͕མ͍ͪͯͳ͍͔Ͳ͏͔ࢹ͢Δ • ͳͷͰɺҎԼͷ༷ͳΞϥʔτϧʔϧΛ1ͭొ͢Ε શͯͷexporterʹରԠͰ͖Δ
Ξϥʔτϧʔϧͷॻ͖ํͷྫ ྫ̎ʣexporertͷࢹର͕མ͍ͪͯͳ͍͔Ͳ͏͔ࢹ͢Δ • ҎԼͷ༷ͳΞϥʔτϧʔϧΛ1ͭొ͢Εɺ ̋̋̋_upϝτϦΫεΛ࣋ͭશͯͷexporterʹରԠͰ͖Δ ALERT ExporterTargetDown IF {__name__=~".*_up"} ==
0 FOR 1m LABELS {severity="major"} ANNOTATIONS { summary="Exporter target ( {{ $labels.job }} ) down" }
Ξϥʔτϧʔϧͷॻ͖ํͷྫ • node_exporterͷTextfile CollectorػೳΛͬͯϝτϦΫεΛexport͢Δ (pushgatewayΛͬͯϝτϦΫεexportग़དྷΔ) • ͜Μͳײ͡ͷshellΛjobεέδϡʔϥͰ࣮ߦ͢ΔΑ͏ʹͯ͠ɺ ྫ̏ʣdailyόον͕ਖ਼ৗʹಈ͍͍ͯΔ͔Ͳ͏͔ࢹ͢Δ #!/bin/bash #
్தͰΤϥʔ͕ൃੜͨ͠Βऴྃ͢Δ set -e UNIXTIME=$(date +%s) RESULT=/path/to/node_exporter_textfile_directory/my_batch_job.prom # execute job /path/to/my_batch_job.sh # write metrics echo my_batch_job_completion_time{cron="my_batch_job", period="daily"} $UNIXTIME > $RESULT.tmp mv $RESULT.tmp $RESULT
Ξϥʔτϧʔϧͷॻ͖ํͷྫ • ҎԼͷ༷ͳΞϥʔτϧʔϧΛొ͢ΕݕՄೳ ྫ̏ʣdailyόον͕ਖ਼ৗʹಈ͍͍ͯΔ͔Ͳ͏͔ࢹ͢Δ ALERT DailyCronLate IF time() - {period="daily"}
> 60 * 60 * 24 FOR 5m LABELS {severity="warning"} ANNOTATIONS { summary="Daily cronjob for {{ $labels.cron }} has not run in 24 hours” }
·ͱΊ • ͍ͬͯΔexporterͱɺऔಘग़དྷΔϝτϦΫεͷҰ෦Λ͝հ ͠·ͨ͠ • exporterΛ࡞Δ߹ͷҙ(golang)ʹ͍ͭͯ͝հ͠·ͨ͠ • exporterΛ࡞Δ߹ͷϕετϓϥΫςΟεΛ͍͔ͭ͘ൈਮͯ͠ ͝հ͠·ͨ͠ •
ΞϥʔτϧʔϧͷྫΛ͍͔ͭ͘͝հ͠·ͨ͠
Thank you :)