Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ユーザ目線でのPrometheus #mackerel_ug /monitoring-prom...
Search
Manabu Matsuzaki
October 27, 2017
Technology
1
3.6k
ユーザ目線でのPrometheus #mackerel_ug /monitoring-prometheus
モニタリング勉強会の発表資料です
Manabu Matsuzaki
October 27, 2017
Tweet
Share
More Decks by Manabu Matsuzaki
See All by Manabu Matsuzaki
Spring BootユーザのためのArmeria入門 #jsug / Introduce to Armeria for Spring users
matsumana
0
2.8k
Canary Release with Argo Rollouts #ふくばねてす / canary-release-with-argo-rollouts
matsumana
1
1.2k
Getting started Central Dogma with Golang #fukuokago #umedago / getting-started-central-dogma-with-golang
matsumana
0
890
Micrometer入門 #javaq / introduce-to-micrometer
matsumana
1
2.9k
ArmeriaとCentral Dogmaから学ぶ、マイクロサービスに必要な機能 #edayfuk / lean-from-armeria-and-central-dogma
matsumana
0
4.3k
SREcon19 Americas 参加レポート #srefukuoka / srecon19-americas-report
matsumana
0
880
SRE入門 & チームで取り組んでいるSRE #srefukuoka / introduce-to-sre
matsumana
0
1.3k
Introduce to Armeria and Central Dogma #GWD_Nulab / introduce-to-armeria-and-central-dogma
matsumana
0
560
Connector/JでMaster/Slave Replication構成のMySQLに接続する #mysql_casual_fukuoka /connector-j-master-slave-replication
matsumana
0
1.5k
Other Decks in Technology
See All in Technology
React開発にStorybookとCopilotを導入して、爆速でUIを編集・確認する方法
yu_kod
1
280
20250707-AI活用の個人差を埋めるチームづくり
shnjtk
6
3.9k
How Do I Contact HP Printer Support? [Full 2025 Guide for U.S. Businesses]
harrry1211
0
120
OSSのSNSツール「Misskey」をさわってみよう(右下ワイプで私のOSCの20年を振り返ります) / 20250705-osc2025-do
akkiesoft
0
170
Lufthansa ®️ USA Contact Numbers: Complete 2025 Support Guide
lufthanahelpsupport
0
200
開発生産性を組織全体の「生産性」へ! 部門間連携の壁を越える実践的ステップ
sudo5in5k
3
7.3k
使いたいMCPサーバーはWeb APIをラップして自分で作る #QiitaBash
bengo4com
0
2k
インフラ寄りSREの生存戦略
sansantech
PRO
2
510
データグループにおけるフロントエンド開発
lycorptech_jp
PRO
1
110
Zero Data Loss Autonomous Recovery Service サービス概要
oracle4engineer
PRO
2
7.8k
OpenTelemetryセマンティック規約の恩恵とMackerel APMにおける活用例 / SRE NEXT 2025
mackerelio
1
110
AIの全社活用を推進するための安全なレールを敷いた話
shoheimitani
2
540
Featured
See All Featured
Fireside Chat
paigeccino
37
3.5k
Being A Developer After 40
akosma
90
590k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
Building Applications with DynamoDB
mza
95
6.5k
Automating Front-end Workflow
addyosmani
1370
200k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
281
13k
Music & Morning Musume
bryan
46
6.6k
Building Adaptive Systems
keathley
43
2.7k
Rails Girls Zürich Keynote
gr2m
95
14k
Facilitating Awesome Meetings
lara
54
6.4k
What's in a price? How to price your products and services
michaelherold
246
12k
Visualization
eitanlees
146
16k
Transcript
ϢʔβઢͰͷPrometheus ϞχλϦϯάษڧձ 2017/10/27 @matsumana
ࣗݾհ • ໊લɿ দ࡚ ֶ • ॴଐɿ LINE Fukuokaגࣜձࣾ ։ൃࣨ
• Twitterɿ @matsumana
ΞδΣϯμ • ࣮ࡍʹͲ͏͍ͬͨϝτϦΫεΛऔ͍ͬͯΔͷ͔ • exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε • Ξϥʔτϧʔϧͷॻ͖ํͷྫ • ·ͱΊ ※࣌ؒͷ߹্ɺPrometheusೖతͳࣄ͠·ͤΜ
લఏ • ݱࡏ༻͍ͯ͠Δόʔδϣϯ • Prometheus 1.8.0
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • node_exporter • loading • CPU༻ • ϝϞϦ༻
• εϫοϓ • slab • context switch
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • node_exporter • σΟεΫI/O • σΟεΫ༻ • inode
• TCPଓ • ESTABLISHED • TIME_WAIT • ωοτϫʔΫ τϥϑΟοΫ
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • apache_exporter • mod_statusϞδϡʔϧͰऔΕΔϝτϦΫε • nginx_exporter • ngx_http_stub_status_moduleϞδϡʔϧͰऔΕΔϝτϦΫε
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • plack_exporter • Plack::Middleware::ServerStatus::LiteͰऔΕΔϝτϦΫε
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • jmx_exporter • JVMͷڞ௨తͳϝτϦΫε • Heap • Metaspace
• GC • Thread • Code cache
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • jmx_exporter • ϛυϧΣΞ/ϥΠϒϥϦݻ༗ͷϝτϦΫε • Tomcat • Jetty
• HikariCP • Caffeine • etc…
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • memcached_exporter • ΩϟογϡαΠζ • Ωϟογϡ • Ωϟογϡώοτ
• operation(get/set) • Eviction
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • mysqld_exporter • QPS • Slow query •
Replicaton delay • binlog • connection • InnoDBؔ࿈
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • redis_exporter • ༻ϝϞϦ • Key • Eviction
• Slow query
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • elasticsearch_exporter • Cluster health • Document(primary shardͷΈ)
• DocumentαΠζ(primary shardͷΈ) • Thread pool queue • Thread pool rejected • Circuit Breaker tripped
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • elasticsearch_exporter • Fielddata cache size • Query
cache size • Query cache hit,miss,total • Request cache size
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • fluent-plugin-prometheus • ֤BufferedOutput pluginͷϝτϦΫε • buffer_queue_length •
buffer_total_bytes • retry_count
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • fluentd_exporter • fluentdϓϩηε୯ҐͷCPU༻ • fluentdϓϩηε୯ҐͷϝϞϦ༻ • td-agent_exporter
• fluentdϓϩηε୯ҐͷCPU༻ • fluentdϓϩηε୯ҐͷϝϞϦ༻ • fluent-agent-lite_exporter • fluentdϓϩηε୯ҐͷCPU༻ • fluentdϓϩηε୯ҐͷϝϞϦ༻
exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε • https://prometheus.io/docs/instrumenting/exporters/ • https://github.com/prometheus/prometheus/wiki/Default- port-allocations • άάΔ • ͕ࣗඞཁͳϝτϦΫε͕export͞Ε͍ͯͳΕPR͢Δ
·ͣɺଞͷਓ͕࡞ͬͨͷ͕ແ͍͔୳ͯ͠ΈΔ
• ΦϑΟγϟϧͷΨΠυϥΠϯ https://prometheus.io/docs/instrumenting/writing_exporters/ exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε ࣗͰ࡞Δ߹
• ΦϑΟγϟϧͷclient library https://prometheus.io/docs/instrumenting/clientlibs/ • ΦϑΟγϟϧgolang, Java, Python, RubyͷΈ •
Third partyʹPHPͳͲ͋Δ exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε ࣗͰ࡞Δ߹
• exportergoͰॻ͘ͷ͕ྑͦ͞͏͚ͩͲҙ͋Δ • Linux Kernel 2.6.23ະຬgolangࣗମ͕αϙʔτͯ͠ͳ͍ CentOS5ܥͩͱgolangͷexporterਖ਼͘͠ಈ͔ͳ͍͔ https://github.com/golang/go/wiki/MinimumRequirements exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε ࣗͰ࡞Δ߹
exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε ΦϑΟγϟϧΨΠυϥΠϯʹهࡌ͞Ε͍ͯΔͷΛҰ෦͝հ • ଞͷexporterͰΘΕͯͳ͍portΛ͏ɻ࡞ͬͨΒهࡌ͠Α͏ https://github.com/prometheus/prometheus/wiki/Default-port- allocations • ϝτϦΫε໊ɺϥϕϧ໊ͷϕετϓϥΫςΟεʹै͏ https://prometheus.io/docs/practices/naming/ •
exporterͰϝτϦΫεΛܭࢉ͠ͳ͍ɻPromQLͰܭࢉ͢Δ • ྫʣΩϟογϡώοτͳͲ
exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε • exporterͷόʔδϣϯ൪߸ΛϝτϦΫεͱͯ͠export͢Δ • ࢹରͷαʔόexporterͷ͕૿͑ΔͱɺͲͷόʔδϣ ϯ͕σϓϩΠ͞Ε͍ͯΔ͔Θ͔Βͳ͘ͳΔ • ͪͳΈʹɺPrometheusຊମҎԼͷΑ͏ͳϝτϦΫεΛ export͍ͯ͠Δ prometheus_build_info{branch=“master",goversion="go1.8.3",revision="3afb3fffa3a29
c3de865e1172fb740442e9d0133",version="1.7.1"} 1 ΦϑΟγϟϧΨΠυϥΠϯʹهࡌ͞Ε͍ͯΔͷΛҰ෦͝հ
exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε ΦϑΟγϟϧΨΠυϥΠϯʹهࡌ͞Ε͍ͯΔͷΛҰ෦͝հ • ̋̋̋_up ϝτϦΫεΛexport͢Δ • ྫ͑ɺMySQLͷ߹mysql_upͱ͍͏ϝτϦΫε • ࢹର͔ΒϝτϦΫεऔಘޭͳΒ1ɺࣦഊͳΒ0Λexport •
ཧ༝ޙड़
Ξϥʔτϧʔϧͷॻ͖ํͷྫ • ྫ1) exporter͕མ͍ͪͯͳ͍͔Ͳ͏͔ࢹ͢Δ • ྫ2) exporertͷࢹର͕མ͍ͪͯͳ͍͔Ͳ͏͔ࢹ͢Δ
• ྫ3) dailyόον͕ਖ਼ৗʹಈ͍͍ͯΔ͔Ͳ͏͔ࢹ͢Δ
Ξϥʔτϧʔϧͷॻ͖ํͷྫ • Prometheusαʔό͕ࣗɺશͯͷexporterʹ͍ͭͯ upϝτϦΫεΛه͍ͯ͠Δ https://prometheus.io/docs/concepts/jobs_instances/ ྫʣnode_exporterͷ߹ up{instance="host00:9100", job="node"} 1
ྫ1ʣexporter͕མ͍ͪͯͳ͍͔Ͳ͏͔ࢹ͢Δ
Ξϥʔτϧʔϧͷॻ͖ํͷྫ ALERT ExporterDown IF up == 0 FOR 1m LABELS
{severity="major"} ANNOTATIONS { description="{{ $labels.instance }} of job {{ $labels.job }} has been down” } ྫ1ʣexporter͕མ͍ͪͯͳ͍͔Ͳ͏͔ࢹ͢Δ • ͳͷͰɺҎԼͷ༷ͳΞϥʔτϧʔϧΛ1ͭొ͢Ε શͯͷexporterʹରԠͰ͖Δ
Ξϥʔτϧʔϧͷॻ͖ํͷྫ ྫ̎ʣexporertͷࢹର͕མ͍ͪͯͳ͍͔Ͳ͏͔ࢹ͢Δ • ҎԼͷ༷ͳΞϥʔτϧʔϧΛ1ͭొ͢Εɺ ̋̋̋_upϝτϦΫεΛ࣋ͭશͯͷexporterʹରԠͰ͖Δ ALERT ExporterTargetDown IF {__name__=~".*_up"} ==
0 FOR 1m LABELS {severity="major"} ANNOTATIONS { summary="Exporter target ( {{ $labels.job }} ) down" }
Ξϥʔτϧʔϧͷॻ͖ํͷྫ • node_exporterͷTextfile CollectorػೳΛͬͯϝτϦΫεΛexport͢Δ (pushgatewayΛͬͯϝτϦΫεexportग़དྷΔ) • ͜Μͳײ͡ͷshellΛjobεέδϡʔϥͰ࣮ߦ͢ΔΑ͏ʹͯ͠ɺ ྫ̏ʣdailyόον͕ਖ਼ৗʹಈ͍͍ͯΔ͔Ͳ͏͔ࢹ͢Δ #!/bin/bash #
్தͰΤϥʔ͕ൃੜͨ͠Βऴྃ͢Δ set -e UNIXTIME=$(date +%s) RESULT=/path/to/node_exporter_textfile_directory/my_batch_job.prom # execute job /path/to/my_batch_job.sh # write metrics echo my_batch_job_completion_time{cron="my_batch_job", period="daily"} $UNIXTIME > $RESULT.tmp mv $RESULT.tmp $RESULT
Ξϥʔτϧʔϧͷॻ͖ํͷྫ • ҎԼͷ༷ͳΞϥʔτϧʔϧΛొ͢ΕݕՄೳ ྫ̏ʣdailyόον͕ਖ਼ৗʹಈ͍͍ͯΔ͔Ͳ͏͔ࢹ͢Δ ALERT DailyCronLate IF time() - {period="daily"}
> 60 * 60 * 24 FOR 5m LABELS {severity="warning"} ANNOTATIONS { summary="Daily cronjob for {{ $labels.cron }} has not run in 24 hours” }
·ͱΊ • ͍ͬͯΔexporterͱɺऔಘग़དྷΔϝτϦΫεͷҰ෦Λ͝հ ͠·ͨ͠ • exporterΛ࡞Δ߹ͷҙ(golang)ʹ͍ͭͯ͝հ͠·ͨ͠ • exporterΛ࡞Δ߹ͷϕετϓϥΫςΟεΛ͍͔ͭ͘ൈਮͯ͠ ͝հ͠·ͨ͠ •
ΞϥʔτϧʔϧͷྫΛ͍͔ͭ͘͝հ͠·ͨ͠
Thank you :)