Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
クラウドネイティブな監視をMackerelで / Mackerel Day#2
Search
FUJIWARA Shunichiro
December 23, 2019
Technology
4
5.8k
クラウドネイティブな監視をMackerelで / Mackerel Day#2
FUJIWARA Shunichiro
December 23, 2019
Tweet
Share
More Decks by FUJIWARA Shunichiro
See All by FUJIWARA Shunichiro
alecthomas/kong はいいぞ
fujiwara3
6
1.8k
ecspressoの設計思想に至る道 / sekkeinight2025
fujiwara3
12
2.9k
さくらのIaaS基盤のモニタリングとOpenTelemetry/OSC Hokkaido 2025
fujiwara3
3
1.3k
監視のこれまでとこれから/sakura monitoring seminar 2025
fujiwara3
11
5.3k
k6による負荷試験 入門から日常的な実践まで/Re:TechTalk #01
fujiwara3
2
160
困難を「一般解」で解く
fujiwara3
10
3.8k
「隙間家具OSS」に至る道/Fujiwara Tech Conference 2025
fujiwara3
7
13k
alecthomas/kong はいいぞ / kamakura.go#7
fujiwara3
1
1.2k
ISUCONに強くなるかもしれない日々の過ごしかた/Findy ISUCON 2024-11-14
fujiwara3
11
1.4k
Other Decks in Technology
See All in Technology
CoRL 2025 Survey
harukiabe
1
210
研究開発部メンバーの働き⽅ / Sansan R&D Profile
sansan33
PRO
3
20k
今この時代に技術とどう向き合うべきか
gree_tech
PRO
2
2k
カンファレンスに託児サポートがあるということ / Having Childcare Support at Conferences
nobu09
1
610
衛星画像超解像化によって実現する2D, 3D空間情報の即時生成と“AI as a Service”/ Real-time generation spatial data enabled_by satellite image super-resolution
lehupa
0
180
難しいセキュリティ用語をわかりやすくしてみた
yuta3110
0
290
AI Agent Dojo #2 watsonx Orchestrateフローの作成
oniak3ibm
PRO
0
130
20251014_Pythonを実務で徹底的に使いこなした話
ippei0923
0
210
Codexとも仲良く。CodeRabbit CLIの紹介
moongift
PRO
1
240
LLMプロダクトの信頼性を上げるには?LLM Observabilityによる、対話型音声AIアプリケーションの安定運用
ivry_presentationmaterials
0
390
Claude Codeを駆使した初めてのiOSアプリ開発 ~ゼロから3週間でグローバルハッカソンで入賞するまで~
oikon48
10
4.9k
プレーリーカードを活用しよう❗❗デジタル名刺交換からはじまるイベント会場交流のススメ
tsukaman
0
180
Featured
See All Featured
Music & Morning Musume
bryan
46
6.8k
[RailsConf 2023] Rails as a piece of cake
palkan
57
5.9k
Documentation Writing (for coders)
carmenintech
75
5.1k
Principles of Awesome APIs and How to Build Them.
keavy
127
17k
The World Runs on Bad Software
bkeepers
PRO
72
11k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
35
3.2k
Art, The Web, and Tiny UX
lynnandtonic
303
21k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
33
2.5k
GitHub's CSS Performance
jonrohan
1032
470k
Gamification - CAS2011
davidbonilla
81
5.5k
Transcript
ΫϥυωΠςΟϒͳࢹΛ Mackerel Ͱ 2019.12.23 Mackerel Day #2 @fujiwara
@fujiwara .BDLFSFMΞϯόαμʔ ʙ HJUIVCDPNLBZBDFDTQSFTTP "NB[PO&$4σϓϩΠπʔϧ HJUIVCDPNGVKJXBSBMBNCSPMM "84-BNCEBσϓϩΠπʔ ϧ
Game & Community
None
ΫϥυωΠςΟϒʁ $/$'$MPVE/BUJWF%FpOJUJPOW ΫϥυωΠςΟϒٕज़ɺύϒϦοΫΫϥυɺϓϥΠϕʔτΫϥυɺϋΠϒ ϦουΫϥυͳͲͷۙతͰμΠφϛοΫͳڥʹ͓͍ͯɺεέʔϥϒϧͳΞϓ ϦέʔγϣϯΛߏங͓Αͼ࣮ߦ͢ΔͨΊͷೳྗΛ৫ʹͨΒ͠·͢ɻ ͜ͷΞϓϩʔνͷදྫʹɺίϯςφɺαʔϏεϝογϡɺϚΠΫϩαʔϏεɺΠ ϛϡʔλϒϧΠϯϑϥετϥΫνϟɺ͓ΑͼએݴܕAPI͕͋Γ·͢ɻ IUUQTHJUIVCDPNDODGUPDCMPCNBTUFS%&'*/*5*0/NE
ίϯςφɺαʔϏεϝογϡɺϚΠΫϩαʔϏε… ͦΕΒͷٕज़Λ͍ͬͯΔ㱠ΫϥυωΠςΟϒ "Design for Failure" ͱͦΕΛ࣮ݱ͢ΔΈίϯϙʔωϯτɺͦΕΛ͍ ͜ͳ͢͜ͱ͕ͦ͜ΫϥυωΠςΟϒ োͷൃੜʹରͯࣗ͠ಈ෮چͰ͖ΔΑ͏ʹσβΠϯ͢ΔɻোͷൃੜʹΑΔϢʔ βʔӨڹ͕ͦͦͳ͍Α͏ʹΞʔΩςΫνϟΛσβΠϯ͢Δ2 IUUQTTQFBLFSEFDLDPNUPSJDMTEFTJHOGPSGBJMVSFJTUIFUSVFDMPVEOBUJWF
"Design for Failure" োΛఆͯ͠γεςϜΛσβΠϯ͢Δ w Πϯελϯεࢮ͵ w ϚωʔδυαʔϏεࢮ͵ ͚ͲେGBJMPWFS͢Δ w
σʔληϯλʔ͝ͱࢮ͵ كʹ͋Δ Ͳ͜·ͰΛఆ͠ɺͲ͔͜ΒఘΊΔ͔ ࢹͦΕΒΛંΓࠐΜͰઃܭ͢ΔˠΫϥυωΠςΟϒͳࢹ
ࢹରಈతʹ૿͑ͨΓݮͬͨΓ͢Δ ίϯςφσϓϩΠ͝ͱʹੜ·ΕมΘΔ ίϯςφͰͳͯ͘ w &$ͷΦʔτεέʔϦϯάͰ4QPUΠϯελϯεΛ͏ w ΘΓͱ͙͢མͪΔɺଞͷ্͕͕Δ ϚωʔδυαʔϏεෛՙʹԠͯ͡૿ݮͰ͖ΔΑ͏ʹͳ͖ͬͯͨ "VSPSB"VUP4DBMJOHͳͲ
Mackerel ͰΫϥυωΠςΟϒͷୈҰา ʮDPOOFDUJWJUZݕΛΊΔʯ
connec%vity ݕΛΊΔ ͜ͷΞϥʔτ ! $SJUJDBM͔͠ͳ͍ w ։ൃڥ͕ਂʹམͪͯඈͼى͖Δඞཁͳ͍ w ຊ൪ڥͰམͪͯαʔϏεʹӨڹ͢Δϗετͳ͍ Α͏ʹ࡞Δ
ϗετ͕ࢮ͵͜ͱΛલఏʹσβΠϯ͢ΔʹΫϥυωΠςΟϒ $SJUJDBMͳΞϥʔτʮαʔϏεͷܧଓੑʹӨڹ͢ΔͷʯͷΈ
Mackerel ͷΫϥυωΠςΟϒ͞ ! ϗετ͕૿͑ͯݮͬͯࣗಈͰैͰ͖Δ NBDLFSFMDPOUBJOFSBHFOU "84"[VSFΠϯςάϨʔγϣϯ ! ୀ͢ΔͱϗετϝτϦοΫ͕ݟ͑ͳ͘ͳΔ $16ͳͲҰ෦ͷϝτϦοΫͷΈΔɺΧελϜϝτϦοΫ ߹ܭฏۉͳͲظͰ͍͍͕ͨɺফ͑ͯ͠·͏
! ϗετ୯Ґ՝ۚ ϚΠΫϩϗετ ԁ݄ Ͱ424-BNCEBͷΑ͏ʹ ͍҆ɺϝτϦοΫ͕গͳ͍ରͷࢹʹͪΐͬͱߴ͍ʜ ૯ϝτϦοΫ՝ۚϓϥϯ͕΄͍͠
ϚωʔδυαʔϏεͷਐԽʹظͭͭ͠ ΫϥυωΠςΟϒͳࢹΛਐΊΔͨΊʹ ʮܺؒՈ۩044ʯ ϚωʔδυαʔϏεͷػೳαʔϏεؒ࿈ܞ͕ࣗͨͪͷӡ༻ʹ͓͍ͯෆेͳ ߹ʹɺͦͷ伱ؒΛຒΊͯΑΓΑ͍ӡ༻Λ࣮ݱ͢ΔͨΊʹ։ൃ͞ΕͨιϑτΣΞɻ ಛʹOSSͷͷΛࢦ͢ɻ3 IUUQTTQFBLFSEFDLDPNGVKJXBSBYJKJBOKJBKVPTTGBMTFTVTVNF IUUQTTQFBLFSEFDLDPNGVKJXBSBBXTEFWEBZUPLZP
ࠓհ͢Δ伱ؒՈ۩ OSS w NBQSPCF w NBDLFSFMQMVHJOQSPNFUIFVTRVFSZ IUUQTHJUIVCDPNGVKJXBSBNBDLFSFMQMVHJOQSPNFUIFVTRVFSZ IUUQTHJUIVCDPNGVKJXBSBNBQSPCF
ʲ՝ʳAWSΠϯςάϨʔγϣϯͰొ͞ΕͨϗετͰ mackerel-plugin ͰͷϝτϦοΫऔΓ͍ͨ
ྫɿAmazon RDS(MySQL)ʹରͯ͠ mackerel-plugin-mysql Λ࣮ߦ
ʲղ๏ʳͲ͔͜ͷϗετͷ mackerel-agent Ͱ plugin ࣮ߦʁ [plugin.metrics.rds01] command = "mackerel-plugin-mysql -host='rds01.***.ap-northeast-1.rds.amazonaws.com'
(ུ)" custom_identifier = "rds01.***.ap-northeast-1.rds.amazonaws.com" [plugin.metrics.rds02] command = "mackerel-plugin-mysql -host='rds02.***.ap-northeast-1.rds.amazonaws.com' (ུ)" custom_identifier = "rds02.***.ap-northeast-1.rds.amazonaws.com" ! ͜ͷϗετ͕མͪͨΒϝτϦοΫऩू͕ࢭ·Δ " ͋ͱ͔Β૿͑ͨϗετΛࢹ͢Δͷʹઃఆมߋ͕໘ # ࠷ۙNBDLFSFMBHFOU Λಈ͔͢ϗετ ͕ͳ͍͜ͱʜ ࢹରͷ૿ݮʹࣗಈै͍ͨ͠ʂ
maprobe w .BDLFSFMʹొ͞Εͨϗετʹରͯ͠ w ֎ܗࢹQJOHUDQIUUQ w NBDLFSFMQMVHJOΛ࣮ߦ ϗετϝτϦοΫͱͯ͠ߘ w ొࡁΈͷϗετϝτϦοΫΛू͠
αʔϏεϝτϦοΫͱͯ͠ߘ Λߦ͏ͨΊͷΤʔδΣϯτ
ʲղ๏ʳ maprobe Ͱ plugin ࣮ߦ probes: - service: production role:
RDS command: command: - 'mackerel-plugin-mysql' - '-host={{.Host.CustomIdentifier}}' - '-username=root' - '-password={{env "RDS_PASSWORD"}}' αʔϏεQSPEVDUJPO ϩʔϧ3%4ͷϗετશͯʹରͯ͠ NBDLFSFMQMVHJONZTRMΛ࣮ߦ ݁ՌΛݸʑͷϗετϝτϦοΫͱͯ͠.BDLFSFMૹ৴͢Δ
maprobe ରϗετͷ૿ݮʹࣗಈै ຖ.BDLFSFM"1*Λୟ͍ͯϗετΛݕࡧ ! ϗετͷ૿ݮʹࣗಈͰै %PDLFSίϯςφΞϦ㽂 docker pull fujiwara/maprobe 4ʹஔ͍ͨઃఆϑΝΠϧΛࣗಈͰ࠶ಡΈࠐΈ
! 4Λߋ৽͢ΕίϯςφϏϧυɾσϓϩΠෆ༻Ͱઃఆө maprobe agent --config s3://example.com/config.yaml IUUQTIVCEPDLFSDPNSGVKJXBSBNBQSPCF
ʲ՝ʳconnec%vity ΛΊͨΒ ϗετͷࢮ׆ࢹͲ͏͢Δʁ
ʲղ๏ʳmaprobe ͷϔϧενΣοΫػೳ ΈࠐΈͷϔϧενΣοΫػೳ QJOH 5$1 )551͕͋Δ probes: - service: production
role: EC2 ping: address: "{{ .Host.IPAddresses.eth0 }}" - service: production role: ElastiCacheRedis tcp: host: "{{ .Host.CustomIdentifier }}" port: 6379 send: "PING\n" expect_pattern: "PONG"
maprobe ͰͷϔϧενΣοΫ NBQSPCFͷϔϧενΣοΫ݁ՌϗετϝτϦοΫʹͳΔ DIFDLࢹͰͳ͍
check ࢹ͕Α͘ͳ͍ͱ͜Ζ(ࢲݟ) ઃఆมߋ͕ϑΝΠϧͷमਖ਼ σϓϩΠ ʮͪΐͬͱ͍·͚ͩࢹP⒎ᮢมߋʯ͕͍͠ ᮢͷධՁํ๏͕ϓϥάΠϯ͝ͱʹ·ͪ·ͪ --critical-underʮҎԼʯ͔ʮະຬʯ͔ʜ ҰʹଟͷϗετͰൃใ͕ͪ͠ ेϗετ͔ΒDIFDLࢹࣦഊ͕དྷͯݪҼݸͩͬͨΓ OUQͷ࣌ࠁͣΕɺEBFNPOͷઃఆ
EFQMPZ ϛεʜ
check ࢹ = metric ࢹͷಛघͳύλʔϯ ϝτϦοΫΛอଘͯͦ͠ΕΛධՁ͢Εಉ͜͡ͱ͕Ͱ͖Δ ϝτϦοΫࢹɺࣜࢹΛ׆༻͢Δ
ྫɿ ping ʹΑΔࢮ׆ࢹ sum(role(production:EC2, ping.count.failure)) QSPDVUDJPO&$ͷ͍ͣΕ͔ͷϗετʹQJOH͕ࣦഊͨ͠ΒXBSO Կ͔མͪͯαʔϏε͕ఏڙͰ͖͍ͯΕ$SJUJDBMͰͳ͍
ྫɿ job queue ͷཹ job ΛΞϥʔτ sum(role(production:job-queue, custom.gearmand.queue.*.total)) ෳͷϗετʹKPCRVFVF͕͋Δ ཹKPCΛϝτϦοΫʹ͍ͯ͠Δ
ཷ·Δͱ͖શͯͷRVFVF͕ཷ·Δ͜ͱ͕ଟ͍ DIFDLࢹͰݸผʹΞϥʔτ͢Δͱશ෦ͷϗετͰൃใ͕ͪ͠ ߹ܭΛݟΔ͜ͱͰશମͷॲཧঢ়گΛѲ͢Δ
ʲ՝ʳୀ͢Δͱফ͑ͯ͠·͏ ϗετϝτϦοΫΛ͍͍ͨ
ΧελϜϝτϦοΫϗετ͕ୀ͢Δͱফ͑ͯ͠·͏
ʲղ๏ʳmaprobe ͰϗετϝτϦοΫΛूอଘ aggregates: - service: production role: push-server metrics: -
name: custom.push.messages.sent outputs: - func: sum name: custom.push.messages.total_sent αʔϏεQSPEVDUJPO ϩʔϧQVTITFSWFSʹରͯ͠ ϗετϝτϦοΫͷQVTINFTTBHFTTFOUΛશऔಘ ˠԋࢉͨ݁͠ՌΛαʔϏεϝτϦοΫͱͯ͠อଘ͢Δ
maprobe aggregate func/ons ݱࡏTVN NJO NBY BWFSBHF DPVOUΛαϙʔτ QFSDFOUJMF͋ͬͨ΄͏͕Αͦ͞͏͚ͩͲະ࣮ ΧελϜϝτϦοΫ͕ফ͑ͳ͚ΕࣜάϥϑͰ݁ͳͷͰԿଔ
ʲ՝ʳͬͱΫϥυωΠςΟϒͳ Ϧιʔεͷࢹ
ϚΠΫϩαʔϏεʂ αʔϏεϝογϡʂ &OWPZΛ͍࢝Ί͍ͯΔͷͰɺϝτϦΫεΛऔΓ͍ͨ ͱ͋Δ&OWPZͷ/statsΛୟ͘ͱʜ $ curl -s x.x.x.x:9901/stats ... cluster.web.default.total_match_count:
1 cluster.web.external.upstream_rq_200: 988 cluster.web.external.upstream_rq_2xx: 988 cluster.web.external.upstream_rq_302: 13 cluster.web.external.upstream_rq_3xx: 13 cluster.web.external.upstream_rq_400: 3 cluster.web.external.upstream_rq_403: 26 cluster.web.external.upstream_rq_404: 5 cluster.web.external.upstream_rq_4xx: 34 .... IUUQTFOWPZQSPYZJP
͜ΕΛશ෦ Mackerel ʹૹΕΑ͍ʁ cluster.web.default.total_match_count: 1 cluster.web.external.upstream_rq_200: 988 cluster.web.external.upstream_rq_2xx: 988 cluster.web.external.upstream_rq_302:
13 cluster.web.external.upstream_rq_3xx: 13 cluster.web.external.upstream_rq_400: 3 cluster.web.external.upstream_rq_403: 26 cluster.web.external.upstream_rq_404: 5 cluster.web.external.upstream_rq_4xx: 34 ...
͍߹Θͤ ϑΟʔυόοΫϑΥʔϜ͔Βؾܰʹฉ͍ͯΈͨ ۙʑenvoy(https://www.envoyproxy.io/)Λಋೖ༧ఆͳͷͰ͕͢ɺMackerel ެࣜͱͯ͠envoy statsऔಘϓϥάΠϯΛެ։͞ΕΔ༧ఆ͋Γ·͢Ͱ͠ΐ͏͔ʁ envoyͷϓϥάΠϯͰ͕͢ɺݱ࣌Ͱެࣜͱͯ͠ެ։͢Δ༧ఆ͍͟͝ ·ͤΜɻ ! ࡞Δ͔ʜʜ
͔͠͠ Envoy େྔͷϝτϦοΫΛు͖ग़͢ $ curl -s x.x.x.x:9901/stats | wc -l
337 ͜ΕΛશ෦.BDLFSFMʹ͍࣋ͬͯ͘ͱʜ ϚΠΫϩϗετϝτϦοΫϗετˠϗετ૬ &OWPZ͕͍Δϗετ͝ͱʹYԁ ੫ผ
࡞ઓมߋ શ෦Λ͍࣋ͬͯ͘ͷίετ͕ݫ͍͠ ͱ͍͑ͲΜͳ͕΄͍͔͠Α͔͍ͬͯ͘ͳ͍ &OWPZӡ༻ܦݧ͕ઙ͍ͷͰɺӡ༻͠ͳ͕ΒݟΔΛܾΊ͍ͨ QMVHJOΛ࡞ͬͯ༗༻ͳϝτϦοΫΛऔΔʹӡ༻ܦݧ͕ඞཁ ͭ·ΓͱΓ͋͑ͣશ෦औΓ͍ͨ
Prometheus Ͳ͏͔ʁ 1VMMܕϝτϦοΫऩूɾࢹπʔϧ อଘͨ͠Λ1SPN2-ΫΤϦͰॊೈʹՃͯ͠औಘͰ͖Δ ظؒͷϝτϦοΫอଘ͋·Γߟྀ͞Ε͍ͯͳ͍ (SBGBOBͱ͔ͰՄࢹԽ͕ී௨ʁ
! Envoy → Prometheus → Mackerel զʑ.BDLFSFMͰΞϥʔτ͍ͨ͠͠άϥϑݟ͍ͨ ͲͷΛऔΔ͖͔͕ݟ͑ͳ͍ͷͰɺۙશ෦औ͓͖͍ͬͯͨ &OWPZˠ1SPNFUIFVT
͜Ε͋Δ 1SPNFUIFVTˠΫΤϦ݁ՌΛNFUSJDQMVHJOܗࣜͰग़ྗ NFUSJDQMVHJOܗࣜͷग़ྗΛ.BDLFSFMʹอଘ ͜Ε͋Δ ͜ͷ͚ͩ࡞ΕΑ͍ͷͰʂ Γͳ͍1SPNFUIFVTʹΫΤϦͯ͠औΕΑ͍
mackrel-plugin-prometheus-query ࡞Γ·ͨ͠ HJUIVCDPNGVKJXBSBNBDLFSFMQMVHJOQSPNFUIFVTRVFSZ $ mackerel-plugin-prometheus-query \ -query "up" \ -metric-key-format
"promq.{job}.{instance}" promq.web.10_1_129_175_9901 1 1575941187 promq.web.10_1_130_170_9901 1 1575941187 promq.web.10_1_131_53_9901 1 1575941187 promq.prometheus.localhost_9090 1 1575941187
Prometheus ͷΫΤϦྫ ྫɿؒͷVQTUSFBNͷϦτϥΠճΛٻΊΔΫΤϦ sum( delta( envoy_cluster_upstream_rq_retry{envoy_cluster_name="web"}[1m] ) ) .BDLFSFMʹϗετ୯ҐͰͳ͘શମͷΛૹΔ ݸʑͷ1SPNFUIFVTΛݟΕ͋ΔͷͰ
plugin ͷग़ྗΛ mkr throw Ͱ͛Δ $ mackerel-plugin-prometheus-query \ -query 'sum(delta(envoy_cluster_upstream_rq_retry_success
{envoy_cluster_name="web"}[1m]))' \ -metric-key-format "envoy.web.upstream.retry.success" \ | mkr throw --service production
·ͱΊ ΫϥυωΠςΟϒͳࢹͱ োΛڐ༰͢Δઃܭશମͷ݈શੑΛݟΔ ͦͷಓ۩ͱͯ͠ͷܺؒՈ۩044 w NBQSPCF ࣗಈैɺ֎ܗࢹɺQMVHJOࢹɺϝτϦοΫू w NBDLFSFMQMVHJOQSPNFUIFVTRVFSZ TIPSUUFSN1SPNFUIFVTͰ
MPOHUFSN BMFSUJOH.BDLFSFMͰ