Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
クラウドネイティブな監視をMackerelで / Mackerel Day#2
Search
FUJIWARA Shunichiro
December 23, 2019
Technology
4
5.6k
クラウドネイティブな監視をMackerelで / Mackerel Day#2
FUJIWARA Shunichiro
December 23, 2019
Tweet
Share
More Decks by FUJIWARA Shunichiro
See All by FUJIWARA Shunichiro
さくらのIaaS基盤のモニタリングとOpenTelemetry/OSC Hokkaido 2025
fujiwara3
3
510
監視のこれまでとこれから/sakura monitoring seminar 2025
fujiwara3
11
4.6k
k6による負荷試験 入門から日常的な実践まで/Re:TechTalk #01
fujiwara3
2
88
困難を「一般解」で解く
fujiwara3
10
3.7k
「隙間家具OSS」に至る道/Fujiwara Tech Conference 2025
fujiwara3
7
12k
alecthomas/kong はいいぞ / kamakura.go#7
fujiwara3
1
1k
ISUCONに強くなるかもしれない日々の過ごしかた/Findy ISUCON 2024-11-14
fujiwara3
11
1.4k
「最高のチューニング」をしないために / hack@delta 24.10
fujiwara3
21
4.4k
AWS Lambdaで実現するスケーラブルで低コストなWebサービス構築/YAPC::Hakodate2024
fujiwara3
10
6.5k
Other Decks in Technology
See All in Technology
[SRE NEXT] ARR150億円_エンジニア140名_27チーム_17プロダクトから始めるSLO.pdf
satos
5
3k
〜『世界中の家族のこころのインフラ』を目指して”次の10年”へ〜 SREが導いたグローバルサービスの信頼性向上戦略とその舞台裏 / Towards the Next Decade: Enhancing Global Service Reliability
kohbis
3
1.5k
ClaudeCode_vs_GeminiCLI_Terraformで比較してみた
tkikuchi
1
1.1k
第64回コンピュータビジョン勉強会「The PanAf-FGBG Dataset: Understanding the Impact of Backgrounds in Wildlife Behaviour Recognition」
x_ttyszk
0
240
「現場で活躍するAIエージェント」を実現するチームと開発プロセス
tkikuchi1002
3
350
推し書籍📚 / Books and a QA Engineer
ak1210
0
140
QuickSight SPICE の効果的な運用戦略~S3 + Athena 構成での実践ノウハウ~/quicksight-spice-s3-athena-best-practices
emiki
0
290
Autify Company Deck
autifyhq
2
44k
AI エージェントと考え直すデータ基盤
na0
20
7.9k
Deep Security Conference 2025:生成AI時代のセキュリティ監視 /dsc2025-genai-secmon
mizutani
4
2.9k
アクセスピークを制するオートスケール再設計: 障害を乗り越えKEDAで実現したリソース管理の最適化
myamashii
1
670
CDK Vibe Coding Fes
tomoki10
1
630
Featured
See All Featured
Product Roadmaps are Hard
iamctodd
PRO
54
11k
How to Ace a Technical Interview
jacobian
278
23k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
Building Adaptive Systems
keathley
43
2.7k
Statistics for Hackers
jakevdp
799
220k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
44
2.4k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.9k
Build The Right Thing And Hit Your Dates
maggiecrowley
37
2.8k
Stop Working from a Prison Cell
hatefulcrawdad
271
21k
The Art of Programming - Codeland 2020
erikaheidi
54
13k
The Cult of Friendly URLs
andyhume
79
6.5k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.6k
Transcript
ΫϥυωΠςΟϒͳࢹΛ Mackerel Ͱ 2019.12.23 Mackerel Day #2 @fujiwara
@fujiwara .BDLFSFMΞϯόαμʔ ʙ HJUIVCDPNLBZBDFDTQSFTTP "NB[PO&$4σϓϩΠπʔϧ HJUIVCDPNGVKJXBSBMBNCSPMM "84-BNCEBσϓϩΠπʔ ϧ
Game & Community
None
ΫϥυωΠςΟϒʁ $/$'$MPVE/BUJWF%FpOJUJPOW ΫϥυωΠςΟϒٕज़ɺύϒϦοΫΫϥυɺϓϥΠϕʔτΫϥυɺϋΠϒ ϦουΫϥυͳͲͷۙతͰμΠφϛοΫͳڥʹ͓͍ͯɺεέʔϥϒϧͳΞϓ ϦέʔγϣϯΛߏங͓Αͼ࣮ߦ͢ΔͨΊͷೳྗΛ৫ʹͨΒ͠·͢ɻ ͜ͷΞϓϩʔνͷදྫʹɺίϯςφɺαʔϏεϝογϡɺϚΠΫϩαʔϏεɺΠ ϛϡʔλϒϧΠϯϑϥετϥΫνϟɺ͓ΑͼએݴܕAPI͕͋Γ·͢ɻ IUUQTHJUIVCDPNDODGUPDCMPCNBTUFS%&'*/*5*0/NE
ίϯςφɺαʔϏεϝογϡɺϚΠΫϩαʔϏε… ͦΕΒͷٕज़Λ͍ͬͯΔ㱠ΫϥυωΠςΟϒ "Design for Failure" ͱͦΕΛ࣮ݱ͢ΔΈίϯϙʔωϯτɺͦΕΛ͍ ͜ͳ͢͜ͱ͕ͦ͜ΫϥυωΠςΟϒ োͷൃੜʹରͯࣗ͠ಈ෮چͰ͖ΔΑ͏ʹσβΠϯ͢ΔɻোͷൃੜʹΑΔϢʔ βʔӨڹ͕ͦͦͳ͍Α͏ʹΞʔΩςΫνϟΛσβΠϯ͢Δ2 IUUQTTQFBLFSEFDLDPNUPSJDMTEFTJHOGPSGBJMVSFJTUIFUSVFDMPVEOBUJWF
"Design for Failure" োΛఆͯ͠γεςϜΛσβΠϯ͢Δ w Πϯελϯεࢮ͵ w ϚωʔδυαʔϏεࢮ͵ ͚ͲେGBJMPWFS͢Δ w
σʔληϯλʔ͝ͱࢮ͵ كʹ͋Δ Ͳ͜·ͰΛఆ͠ɺͲ͔͜ΒఘΊΔ͔ ࢹͦΕΒΛંΓࠐΜͰઃܭ͢ΔˠΫϥυωΠςΟϒͳࢹ
ࢹରಈతʹ૿͑ͨΓݮͬͨΓ͢Δ ίϯςφσϓϩΠ͝ͱʹੜ·ΕมΘΔ ίϯςφͰͳͯ͘ w &$ͷΦʔτεέʔϦϯάͰ4QPUΠϯελϯεΛ͏ w ΘΓͱ͙͢མͪΔɺଞͷ্͕͕Δ ϚωʔδυαʔϏεෛՙʹԠͯ͡૿ݮͰ͖ΔΑ͏ʹͳ͖ͬͯͨ "VSPSB"VUP4DBMJOHͳͲ
Mackerel ͰΫϥυωΠςΟϒͷୈҰา ʮDPOOFDUJWJUZݕΛΊΔʯ
connec%vity ݕΛΊΔ ͜ͷΞϥʔτ ! $SJUJDBM͔͠ͳ͍ w ։ൃڥ͕ਂʹམͪͯඈͼى͖Δඞཁͳ͍ w ຊ൪ڥͰམͪͯαʔϏεʹӨڹ͢Δϗετͳ͍ Α͏ʹ࡞Δ
ϗετ͕ࢮ͵͜ͱΛલఏʹσβΠϯ͢ΔʹΫϥυωΠςΟϒ $SJUJDBMͳΞϥʔτʮαʔϏεͷܧଓੑʹӨڹ͢ΔͷʯͷΈ
Mackerel ͷΫϥυωΠςΟϒ͞ ! ϗετ͕૿͑ͯݮͬͯࣗಈͰैͰ͖Δ NBDLFSFMDPOUBJOFSBHFOU "84"[VSFΠϯςάϨʔγϣϯ ! ୀ͢ΔͱϗετϝτϦοΫ͕ݟ͑ͳ͘ͳΔ $16ͳͲҰ෦ͷϝτϦοΫͷΈΔɺΧελϜϝτϦοΫ ߹ܭฏۉͳͲظͰ͍͍͕ͨɺফ͑ͯ͠·͏
! ϗετ୯Ґ՝ۚ ϚΠΫϩϗετ ԁ݄ Ͱ424-BNCEBͷΑ͏ʹ ͍҆ɺϝτϦοΫ͕গͳ͍ରͷࢹʹͪΐͬͱߴ͍ʜ ૯ϝτϦοΫ՝ۚϓϥϯ͕΄͍͠
ϚωʔδυαʔϏεͷਐԽʹظͭͭ͠ ΫϥυωΠςΟϒͳࢹΛਐΊΔͨΊʹ ʮܺؒՈ۩044ʯ ϚωʔδυαʔϏεͷػೳαʔϏεؒ࿈ܞ͕ࣗͨͪͷӡ༻ʹ͓͍ͯෆेͳ ߹ʹɺͦͷ伱ؒΛຒΊͯΑΓΑ͍ӡ༻Λ࣮ݱ͢ΔͨΊʹ։ൃ͞ΕͨιϑτΣΞɻ ಛʹOSSͷͷΛࢦ͢ɻ3 IUUQTTQFBLFSEFDLDPNGVKJXBSBYJKJBOKJBKVPTTGBMTFTVTVNF IUUQTTQFBLFSEFDLDPNGVKJXBSBBXTEFWEBZUPLZP
ࠓհ͢Δ伱ؒՈ۩ OSS w NBQSPCF w NBDLFSFMQMVHJOQSPNFUIFVTRVFSZ IUUQTHJUIVCDPNGVKJXBSBNBDLFSFMQMVHJOQSPNFUIFVTRVFSZ IUUQTHJUIVCDPNGVKJXBSBNBQSPCF
ʲ՝ʳAWSΠϯςάϨʔγϣϯͰొ͞ΕͨϗετͰ mackerel-plugin ͰͷϝτϦοΫऔΓ͍ͨ
ྫɿAmazon RDS(MySQL)ʹରͯ͠ mackerel-plugin-mysql Λ࣮ߦ
ʲղ๏ʳͲ͔͜ͷϗετͷ mackerel-agent Ͱ plugin ࣮ߦʁ [plugin.metrics.rds01] command = "mackerel-plugin-mysql -host='rds01.***.ap-northeast-1.rds.amazonaws.com'
(ུ)" custom_identifier = "rds01.***.ap-northeast-1.rds.amazonaws.com" [plugin.metrics.rds02] command = "mackerel-plugin-mysql -host='rds02.***.ap-northeast-1.rds.amazonaws.com' (ུ)" custom_identifier = "rds02.***.ap-northeast-1.rds.amazonaws.com" ! ͜ͷϗετ͕མͪͨΒϝτϦοΫऩू͕ࢭ·Δ " ͋ͱ͔Β૿͑ͨϗετΛࢹ͢Δͷʹઃఆมߋ͕໘ # ࠷ۙNBDLFSFMBHFOU Λಈ͔͢ϗετ ͕ͳ͍͜ͱʜ ࢹରͷ૿ݮʹࣗಈै͍ͨ͠ʂ
maprobe w .BDLFSFMʹొ͞Εͨϗετʹରͯ͠ w ֎ܗࢹQJOHUDQIUUQ w NBDLFSFMQMVHJOΛ࣮ߦ ϗετϝτϦοΫͱͯ͠ߘ w ొࡁΈͷϗετϝτϦοΫΛू͠
αʔϏεϝτϦοΫͱͯ͠ߘ Λߦ͏ͨΊͷΤʔδΣϯτ
ʲղ๏ʳ maprobe Ͱ plugin ࣮ߦ probes: - service: production role:
RDS command: command: - 'mackerel-plugin-mysql' - '-host={{.Host.CustomIdentifier}}' - '-username=root' - '-password={{env "RDS_PASSWORD"}}' αʔϏεQSPEVDUJPO ϩʔϧ3%4ͷϗετશͯʹରͯ͠ NBDLFSFMQMVHJONZTRMΛ࣮ߦ ݁ՌΛݸʑͷϗετϝτϦοΫͱͯ͠.BDLFSFMૹ৴͢Δ
maprobe ରϗετͷ૿ݮʹࣗಈै ຖ.BDLFSFM"1*Λୟ͍ͯϗετΛݕࡧ ! ϗετͷ૿ݮʹࣗಈͰै %PDLFSίϯςφΞϦ㽂 docker pull fujiwara/maprobe 4ʹஔ͍ͨઃఆϑΝΠϧΛࣗಈͰ࠶ಡΈࠐΈ
! 4Λߋ৽͢ΕίϯςφϏϧυɾσϓϩΠෆ༻Ͱઃఆө maprobe agent --config s3://example.com/config.yaml IUUQTIVCEPDLFSDPNSGVKJXBSBNBQSPCF
ʲ՝ʳconnec%vity ΛΊͨΒ ϗετͷࢮ׆ࢹͲ͏͢Δʁ
ʲղ๏ʳmaprobe ͷϔϧενΣοΫػೳ ΈࠐΈͷϔϧενΣοΫػೳ QJOH 5$1 )551͕͋Δ probes: - service: production
role: EC2 ping: address: "{{ .Host.IPAddresses.eth0 }}" - service: production role: ElastiCacheRedis tcp: host: "{{ .Host.CustomIdentifier }}" port: 6379 send: "PING\n" expect_pattern: "PONG"
maprobe ͰͷϔϧενΣοΫ NBQSPCFͷϔϧενΣοΫ݁ՌϗετϝτϦοΫʹͳΔ DIFDLࢹͰͳ͍
check ࢹ͕Α͘ͳ͍ͱ͜Ζ(ࢲݟ) ઃఆมߋ͕ϑΝΠϧͷमਖ਼ σϓϩΠ ʮͪΐͬͱ͍·͚ͩࢹP⒎ᮢมߋʯ͕͍͠ ᮢͷධՁํ๏͕ϓϥάΠϯ͝ͱʹ·ͪ·ͪ --critical-underʮҎԼʯ͔ʮະຬʯ͔ʜ ҰʹଟͷϗετͰൃใ͕ͪ͠ ेϗετ͔ΒDIFDLࢹࣦഊ͕དྷͯݪҼݸͩͬͨΓ OUQͷ࣌ࠁͣΕɺEBFNPOͷઃఆ
EFQMPZ ϛεʜ
check ࢹ = metric ࢹͷಛघͳύλʔϯ ϝτϦοΫΛอଘͯͦ͠ΕΛධՁ͢Εಉ͜͡ͱ͕Ͱ͖Δ ϝτϦοΫࢹɺࣜࢹΛ׆༻͢Δ
ྫɿ ping ʹΑΔࢮ׆ࢹ sum(role(production:EC2, ping.count.failure)) QSPDVUDJPO&$ͷ͍ͣΕ͔ͷϗετʹQJOH͕ࣦഊͨ͠ΒXBSO Կ͔མͪͯαʔϏε͕ఏڙͰ͖͍ͯΕ$SJUJDBMͰͳ͍
ྫɿ job queue ͷཹ job ΛΞϥʔτ sum(role(production:job-queue, custom.gearmand.queue.*.total)) ෳͷϗετʹKPCRVFVF͕͋Δ ཹKPCΛϝτϦοΫʹ͍ͯ͠Δ
ཷ·Δͱ͖શͯͷRVFVF͕ཷ·Δ͜ͱ͕ଟ͍ DIFDLࢹͰݸผʹΞϥʔτ͢Δͱશ෦ͷϗετͰൃใ͕ͪ͠ ߹ܭΛݟΔ͜ͱͰશମͷॲཧঢ়گΛѲ͢Δ
ʲ՝ʳୀ͢Δͱফ͑ͯ͠·͏ ϗετϝτϦοΫΛ͍͍ͨ
ΧελϜϝτϦοΫϗετ͕ୀ͢Δͱফ͑ͯ͠·͏
ʲղ๏ʳmaprobe ͰϗετϝτϦοΫΛूอଘ aggregates: - service: production role: push-server metrics: -
name: custom.push.messages.sent outputs: - func: sum name: custom.push.messages.total_sent αʔϏεQSPEVDUJPO ϩʔϧQVTITFSWFSʹରͯ͠ ϗετϝτϦοΫͷQVTINFTTBHFTTFOUΛશऔಘ ˠԋࢉͨ݁͠ՌΛαʔϏεϝτϦοΫͱͯ͠อଘ͢Δ
maprobe aggregate func/ons ݱࡏTVN NJO NBY BWFSBHF DPVOUΛαϙʔτ QFSDFOUJMF͋ͬͨ΄͏͕Αͦ͞͏͚ͩͲະ࣮ ΧελϜϝτϦοΫ͕ফ͑ͳ͚ΕࣜάϥϑͰ݁ͳͷͰԿଔ
ʲ՝ʳͬͱΫϥυωΠςΟϒͳ Ϧιʔεͷࢹ
ϚΠΫϩαʔϏεʂ αʔϏεϝογϡʂ &OWPZΛ͍࢝Ί͍ͯΔͷͰɺϝτϦΫεΛऔΓ͍ͨ ͱ͋Δ&OWPZͷ/statsΛୟ͘ͱʜ $ curl -s x.x.x.x:9901/stats ... cluster.web.default.total_match_count:
1 cluster.web.external.upstream_rq_200: 988 cluster.web.external.upstream_rq_2xx: 988 cluster.web.external.upstream_rq_302: 13 cluster.web.external.upstream_rq_3xx: 13 cluster.web.external.upstream_rq_400: 3 cluster.web.external.upstream_rq_403: 26 cluster.web.external.upstream_rq_404: 5 cluster.web.external.upstream_rq_4xx: 34 .... IUUQTFOWPZQSPYZJP
͜ΕΛશ෦ Mackerel ʹૹΕΑ͍ʁ cluster.web.default.total_match_count: 1 cluster.web.external.upstream_rq_200: 988 cluster.web.external.upstream_rq_2xx: 988 cluster.web.external.upstream_rq_302:
13 cluster.web.external.upstream_rq_3xx: 13 cluster.web.external.upstream_rq_400: 3 cluster.web.external.upstream_rq_403: 26 cluster.web.external.upstream_rq_404: 5 cluster.web.external.upstream_rq_4xx: 34 ...
͍߹Θͤ ϑΟʔυόοΫϑΥʔϜ͔Βؾܰʹฉ͍ͯΈͨ ۙʑenvoy(https://www.envoyproxy.io/)Λಋೖ༧ఆͳͷͰ͕͢ɺMackerel ެࣜͱͯ͠envoy statsऔಘϓϥάΠϯΛެ։͞ΕΔ༧ఆ͋Γ·͢Ͱ͠ΐ͏͔ʁ envoyͷϓϥάΠϯͰ͕͢ɺݱ࣌Ͱެࣜͱͯ͠ެ։͢Δ༧ఆ͍͟͝ ·ͤΜɻ ! ࡞Δ͔ʜʜ
͔͠͠ Envoy େྔͷϝτϦοΫΛు͖ग़͢ $ curl -s x.x.x.x:9901/stats | wc -l
337 ͜ΕΛશ෦.BDLFSFMʹ͍࣋ͬͯ͘ͱʜ ϚΠΫϩϗετϝτϦοΫϗετˠϗετ૬ &OWPZ͕͍Δϗετ͝ͱʹYԁ ੫ผ
࡞ઓมߋ શ෦Λ͍࣋ͬͯ͘ͷίετ͕ݫ͍͠ ͱ͍͑ͲΜͳ͕΄͍͔͠Α͔͍ͬͯ͘ͳ͍ &OWPZӡ༻ܦݧ͕ઙ͍ͷͰɺӡ༻͠ͳ͕ΒݟΔΛܾΊ͍ͨ QMVHJOΛ࡞ͬͯ༗༻ͳϝτϦοΫΛऔΔʹӡ༻ܦݧ͕ඞཁ ͭ·ΓͱΓ͋͑ͣશ෦औΓ͍ͨ
Prometheus Ͳ͏͔ʁ 1VMMܕϝτϦοΫऩूɾࢹπʔϧ อଘͨ͠Λ1SPN2-ΫΤϦͰॊೈʹՃͯ͠औಘͰ͖Δ ظؒͷϝτϦοΫอଘ͋·Γߟྀ͞Ε͍ͯͳ͍ (SBGBOBͱ͔ͰՄࢹԽ͕ී௨ʁ
! Envoy → Prometheus → Mackerel զʑ.BDLFSFMͰΞϥʔτ͍ͨ͠͠άϥϑݟ͍ͨ ͲͷΛऔΔ͖͔͕ݟ͑ͳ͍ͷͰɺۙશ෦औ͓͖͍ͬͯͨ &OWPZˠ1SPNFUIFVT
͜Ε͋Δ 1SPNFUIFVTˠΫΤϦ݁ՌΛNFUSJDQMVHJOܗࣜͰग़ྗ NFUSJDQMVHJOܗࣜͷग़ྗΛ.BDLFSFMʹอଘ ͜Ε͋Δ ͜ͷ͚ͩ࡞ΕΑ͍ͷͰʂ Γͳ͍1SPNFUIFVTʹΫΤϦͯ͠औΕΑ͍
mackrel-plugin-prometheus-query ࡞Γ·ͨ͠ HJUIVCDPNGVKJXBSBNBDLFSFMQMVHJOQSPNFUIFVTRVFSZ $ mackerel-plugin-prometheus-query \ -query "up" \ -metric-key-format
"promq.{job}.{instance}" promq.web.10_1_129_175_9901 1 1575941187 promq.web.10_1_130_170_9901 1 1575941187 promq.web.10_1_131_53_9901 1 1575941187 promq.prometheus.localhost_9090 1 1575941187
Prometheus ͷΫΤϦྫ ྫɿؒͷVQTUSFBNͷϦτϥΠճΛٻΊΔΫΤϦ sum( delta( envoy_cluster_upstream_rq_retry{envoy_cluster_name="web"}[1m] ) ) .BDLFSFMʹϗετ୯ҐͰͳ͘શମͷΛૹΔ ݸʑͷ1SPNFUIFVTΛݟΕ͋ΔͷͰ
plugin ͷग़ྗΛ mkr throw Ͱ͛Δ $ mackerel-plugin-prometheus-query \ -query 'sum(delta(envoy_cluster_upstream_rq_retry_success
{envoy_cluster_name="web"}[1m]))' \ -metric-key-format "envoy.web.upstream.retry.success" \ | mkr throw --service production
·ͱΊ ΫϥυωΠςΟϒͳࢹͱ োΛڐ༰͢Δઃܭશମͷ݈શੑΛݟΔ ͦͷಓ۩ͱͯ͠ͷܺؒՈ۩044 w NBQSPCF ࣗಈैɺ֎ܗࢹɺQMVHJOࢹɺϝτϦοΫू w NBDLFSFMQMVHJOQSPNFUIFVTRVFSZ TIPSUUFSN1SPNFUIFVTͰ
MPOHUFSN BMFSUJOH.BDLFSFMͰ