Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
クラウドネイティブな監視をMackerelで / Mackerel Day#2
Search
FUJIWARA Shunichiro
December 23, 2019
Technology
4
5.5k
クラウドネイティブな監視をMackerelで / Mackerel Day#2
FUJIWARA Shunichiro
December 23, 2019
Tweet
Share
More Decks by FUJIWARA Shunichiro
See All by FUJIWARA Shunichiro
「隙間家具OSS」に至る道/Fujiwara Tech Conference 2025
fujiwara3
7
9.9k
alecthomas/kong はいいぞ / kamakura.go#7
fujiwara3
1
790
ISUCONに強くなるかもしれない日々の過ごしかた/Findy ISUCON 2024-11-14
fujiwara3
10
1.2k
「最高のチューニング」をしないために / hack@delta 24.10
fujiwara3
21
4.2k
AWS Lambdaで実現するスケーラブルで低コストなWebサービス構築/YAPC::Hakodate2024
fujiwara3
10
5.6k
CEL(Common Expression Language)で書いた条件にマッチしたIAM Policyを見つける / iam-policy-finder
fujiwara3
2
1.6k
awslim - Goで実装された高速なAWS CLIの代替品を作った/layerx.go#1
fujiwara3
6
810
AWS CLIの起動が重くてつらいので aws-sdk-client-go を書いた / kamakura.go#6
fujiwara3
7
11k
コードを書く隙間を見つけて生きていく技術/Findy 思考の現在地
fujiwara3
31
7.2k
Other Decks in Technology
See All in Technology
分解して理解する Aspire
nenonaninu
2
510
依存パッケージの更新はコツコツが勝つコツ! / phpcon_nagoya2025
blue_goheimochi
3
180
2.5Dモデルのすべて
yu4u
2
960
速くて安いWebサイトを作る
nishiharatsubasa
15
15k
Culture Deck
optfit
0
500
TAMとre:Capセキュリティ編 〜拡張脅威検出デモを添えて〜
fujiihda
2
380
Perlの生きのこり - エンジニアがこの先生きのこるためのカンファレンス2025
kfly8
1
230
なぜ私は自分が使わないサービスを作るのか? / Why would I create a service that I would not use?
aiandrox
0
890
OpenID Connect for Identity Assurance の概要と翻訳版のご紹介 / 20250219-BizDay17-OIDC4IDA-Intro
oidfj
0
430
NFV基盤のOpenStack更新 ~9世代バージョンアップへの挑戦~
vtj
0
300
Share my, our lessons from the road to re:Invent
naospon
0
110
システム・ML活用を広げるdbtのデータモデリング / Expanding System & ML Use with dbt Modeling
i125
1
290
Featured
See All Featured
Art, The Web, and Tiny UX
lynnandtonic
298
20k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
47
5.2k
Build The Right Thing And Hit Your Dates
maggiecrowley
34
2.5k
Facilitating Awesome Meetings
lara
52
6.2k
A better future with KSS
kneath
238
17k
Adopting Sorbet at Scale
ufuk
74
9.2k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
44
7k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.1k
The Straight Up "How To Draw Better" Workshop
denniskardys
232
140k
What’s in a name? Adding method to the madness
productmarketing
PRO
22
3.3k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
656
59k
Done Done
chrislema
182
16k
Transcript
ΫϥυωΠςΟϒͳࢹΛ Mackerel Ͱ 2019.12.23 Mackerel Day #2 @fujiwara
@fujiwara .BDLFSFMΞϯόαμʔ ʙ HJUIVCDPNLBZBDFDTQSFTTP "NB[PO&$4σϓϩΠπʔϧ HJUIVCDPNGVKJXBSBMBNCSPMM "84-BNCEBσϓϩΠπʔ ϧ
Game & Community
None
ΫϥυωΠςΟϒʁ $/$'$MPVE/BUJWF%FpOJUJPOW ΫϥυωΠςΟϒٕज़ɺύϒϦοΫΫϥυɺϓϥΠϕʔτΫϥυɺϋΠϒ ϦουΫϥυͳͲͷۙతͰμΠφϛοΫͳڥʹ͓͍ͯɺεέʔϥϒϧͳΞϓ ϦέʔγϣϯΛߏங͓Αͼ࣮ߦ͢ΔͨΊͷೳྗΛ৫ʹͨΒ͠·͢ɻ ͜ͷΞϓϩʔνͷදྫʹɺίϯςφɺαʔϏεϝογϡɺϚΠΫϩαʔϏεɺΠ ϛϡʔλϒϧΠϯϑϥετϥΫνϟɺ͓ΑͼએݴܕAPI͕͋Γ·͢ɻ IUUQTHJUIVCDPNDODGUPDCMPCNBTUFS%&'*/*5*0/NE
ίϯςφɺαʔϏεϝογϡɺϚΠΫϩαʔϏε… ͦΕΒͷٕज़Λ͍ͬͯΔ㱠ΫϥυωΠςΟϒ "Design for Failure" ͱͦΕΛ࣮ݱ͢ΔΈίϯϙʔωϯτɺͦΕΛ͍ ͜ͳ͢͜ͱ͕ͦ͜ΫϥυωΠςΟϒ োͷൃੜʹରͯࣗ͠ಈ෮چͰ͖ΔΑ͏ʹσβΠϯ͢ΔɻোͷൃੜʹΑΔϢʔ βʔӨڹ͕ͦͦͳ͍Α͏ʹΞʔΩςΫνϟΛσβΠϯ͢Δ2 IUUQTTQFBLFSEFDLDPNUPSJDMTEFTJHOGPSGBJMVSFJTUIFUSVFDMPVEOBUJWF
"Design for Failure" োΛఆͯ͠γεςϜΛσβΠϯ͢Δ w Πϯελϯεࢮ͵ w ϚωʔδυαʔϏεࢮ͵ ͚ͲେGBJMPWFS͢Δ w
σʔληϯλʔ͝ͱࢮ͵ كʹ͋Δ Ͳ͜·ͰΛఆ͠ɺͲ͔͜ΒఘΊΔ͔ ࢹͦΕΒΛંΓࠐΜͰઃܭ͢ΔˠΫϥυωΠςΟϒͳࢹ
ࢹରಈతʹ૿͑ͨΓݮͬͨΓ͢Δ ίϯςφσϓϩΠ͝ͱʹੜ·ΕมΘΔ ίϯςφͰͳͯ͘ w &$ͷΦʔτεέʔϦϯάͰ4QPUΠϯελϯεΛ͏ w ΘΓͱ͙͢མͪΔɺଞͷ্͕͕Δ ϚωʔδυαʔϏεෛՙʹԠͯ͡૿ݮͰ͖ΔΑ͏ʹͳ͖ͬͯͨ "VSPSB"VUP4DBMJOHͳͲ
Mackerel ͰΫϥυωΠςΟϒͷୈҰา ʮDPOOFDUJWJUZݕΛΊΔʯ
connec%vity ݕΛΊΔ ͜ͷΞϥʔτ ! $SJUJDBM͔͠ͳ͍ w ։ൃڥ͕ਂʹམͪͯඈͼى͖Δඞཁͳ͍ w ຊ൪ڥͰམͪͯαʔϏεʹӨڹ͢Δϗετͳ͍ Α͏ʹ࡞Δ
ϗετ͕ࢮ͵͜ͱΛલఏʹσβΠϯ͢ΔʹΫϥυωΠςΟϒ $SJUJDBMͳΞϥʔτʮαʔϏεͷܧଓੑʹӨڹ͢ΔͷʯͷΈ
Mackerel ͷΫϥυωΠςΟϒ͞ ! ϗετ͕૿͑ͯݮͬͯࣗಈͰैͰ͖Δ NBDLFSFMDPOUBJOFSBHFOU "84"[VSFΠϯςάϨʔγϣϯ ! ୀ͢ΔͱϗετϝτϦοΫ͕ݟ͑ͳ͘ͳΔ $16ͳͲҰ෦ͷϝτϦοΫͷΈΔɺΧελϜϝτϦοΫ ߹ܭฏۉͳͲظͰ͍͍͕ͨɺফ͑ͯ͠·͏
! ϗετ୯Ґ՝ۚ ϚΠΫϩϗετ ԁ݄ Ͱ424-BNCEBͷΑ͏ʹ ͍҆ɺϝτϦοΫ͕গͳ͍ରͷࢹʹͪΐͬͱߴ͍ʜ ૯ϝτϦοΫ՝ۚϓϥϯ͕΄͍͠
ϚωʔδυαʔϏεͷਐԽʹظͭͭ͠ ΫϥυωΠςΟϒͳࢹΛਐΊΔͨΊʹ ʮܺؒՈ۩044ʯ ϚωʔδυαʔϏεͷػೳαʔϏεؒ࿈ܞ͕ࣗͨͪͷӡ༻ʹ͓͍ͯෆेͳ ߹ʹɺͦͷ伱ؒΛຒΊͯΑΓΑ͍ӡ༻Λ࣮ݱ͢ΔͨΊʹ։ൃ͞ΕͨιϑτΣΞɻ ಛʹOSSͷͷΛࢦ͢ɻ3 IUUQTTQFBLFSEFDLDPNGVKJXBSBYJKJBOKJBKVPTTGBMTFTVTVNF IUUQTTQFBLFSEFDLDPNGVKJXBSBBXTEFWEBZUPLZP
ࠓհ͢Δ伱ؒՈ۩ OSS w NBQSPCF w NBDLFSFMQMVHJOQSPNFUIFVTRVFSZ IUUQTHJUIVCDPNGVKJXBSBNBDLFSFMQMVHJOQSPNFUIFVTRVFSZ IUUQTHJUIVCDPNGVKJXBSBNBQSPCF
ʲ՝ʳAWSΠϯςάϨʔγϣϯͰొ͞ΕͨϗετͰ mackerel-plugin ͰͷϝτϦοΫऔΓ͍ͨ
ྫɿAmazon RDS(MySQL)ʹରͯ͠ mackerel-plugin-mysql Λ࣮ߦ
ʲղ๏ʳͲ͔͜ͷϗετͷ mackerel-agent Ͱ plugin ࣮ߦʁ [plugin.metrics.rds01] command = "mackerel-plugin-mysql -host='rds01.***.ap-northeast-1.rds.amazonaws.com'
(ུ)" custom_identifier = "rds01.***.ap-northeast-1.rds.amazonaws.com" [plugin.metrics.rds02] command = "mackerel-plugin-mysql -host='rds02.***.ap-northeast-1.rds.amazonaws.com' (ུ)" custom_identifier = "rds02.***.ap-northeast-1.rds.amazonaws.com" ! ͜ͷϗετ͕མͪͨΒϝτϦοΫऩू͕ࢭ·Δ " ͋ͱ͔Β૿͑ͨϗετΛࢹ͢Δͷʹઃఆมߋ͕໘ # ࠷ۙNBDLFSFMBHFOU Λಈ͔͢ϗετ ͕ͳ͍͜ͱʜ ࢹରͷ૿ݮʹࣗಈै͍ͨ͠ʂ
maprobe w .BDLFSFMʹొ͞Εͨϗετʹରͯ͠ w ֎ܗࢹQJOHUDQIUUQ w NBDLFSFMQMVHJOΛ࣮ߦ ϗετϝτϦοΫͱͯ͠ߘ w ొࡁΈͷϗετϝτϦοΫΛू͠
αʔϏεϝτϦοΫͱͯ͠ߘ Λߦ͏ͨΊͷΤʔδΣϯτ
ʲղ๏ʳ maprobe Ͱ plugin ࣮ߦ probes: - service: production role:
RDS command: command: - 'mackerel-plugin-mysql' - '-host={{.Host.CustomIdentifier}}' - '-username=root' - '-password={{env "RDS_PASSWORD"}}' αʔϏεQSPEVDUJPO ϩʔϧ3%4ͷϗετશͯʹରͯ͠ NBDLFSFMQMVHJONZTRMΛ࣮ߦ ݁ՌΛݸʑͷϗετϝτϦοΫͱͯ͠.BDLFSFMૹ৴͢Δ
maprobe ରϗετͷ૿ݮʹࣗಈै ຖ.BDLFSFM"1*Λୟ͍ͯϗετΛݕࡧ ! ϗετͷ૿ݮʹࣗಈͰै %PDLFSίϯςφΞϦ㽂 docker pull fujiwara/maprobe 4ʹஔ͍ͨઃఆϑΝΠϧΛࣗಈͰ࠶ಡΈࠐΈ
! 4Λߋ৽͢ΕίϯςφϏϧυɾσϓϩΠෆ༻Ͱઃఆө maprobe agent --config s3://example.com/config.yaml IUUQTIVCEPDLFSDPNSGVKJXBSBNBQSPCF
ʲ՝ʳconnec%vity ΛΊͨΒ ϗετͷࢮ׆ࢹͲ͏͢Δʁ
ʲղ๏ʳmaprobe ͷϔϧενΣοΫػೳ ΈࠐΈͷϔϧενΣοΫػೳ QJOH 5$1 )551͕͋Δ probes: - service: production
role: EC2 ping: address: "{{ .Host.IPAddresses.eth0 }}" - service: production role: ElastiCacheRedis tcp: host: "{{ .Host.CustomIdentifier }}" port: 6379 send: "PING\n" expect_pattern: "PONG"
maprobe ͰͷϔϧενΣοΫ NBQSPCFͷϔϧενΣοΫ݁ՌϗετϝτϦοΫʹͳΔ DIFDLࢹͰͳ͍
check ࢹ͕Α͘ͳ͍ͱ͜Ζ(ࢲݟ) ઃఆมߋ͕ϑΝΠϧͷमਖ਼ σϓϩΠ ʮͪΐͬͱ͍·͚ͩࢹP⒎ᮢมߋʯ͕͍͠ ᮢͷධՁํ๏͕ϓϥάΠϯ͝ͱʹ·ͪ·ͪ --critical-underʮҎԼʯ͔ʮະຬʯ͔ʜ ҰʹଟͷϗετͰൃใ͕ͪ͠ ेϗετ͔ΒDIFDLࢹࣦഊ͕དྷͯݪҼݸͩͬͨΓ OUQͷ࣌ࠁͣΕɺEBFNPOͷઃఆ
EFQMPZ ϛεʜ
check ࢹ = metric ࢹͷಛघͳύλʔϯ ϝτϦοΫΛอଘͯͦ͠ΕΛධՁ͢Εಉ͜͡ͱ͕Ͱ͖Δ ϝτϦοΫࢹɺࣜࢹΛ׆༻͢Δ
ྫɿ ping ʹΑΔࢮ׆ࢹ sum(role(production:EC2, ping.count.failure)) QSPDVUDJPO&$ͷ͍ͣΕ͔ͷϗετʹQJOH͕ࣦഊͨ͠ΒXBSO Կ͔མͪͯαʔϏε͕ఏڙͰ͖͍ͯΕ$SJUJDBMͰͳ͍
ྫɿ job queue ͷཹ job ΛΞϥʔτ sum(role(production:job-queue, custom.gearmand.queue.*.total)) ෳͷϗετʹKPCRVFVF͕͋Δ ཹKPCΛϝτϦοΫʹ͍ͯ͠Δ
ཷ·Δͱ͖શͯͷRVFVF͕ཷ·Δ͜ͱ͕ଟ͍ DIFDLࢹͰݸผʹΞϥʔτ͢Δͱશ෦ͷϗετͰൃใ͕ͪ͠ ߹ܭΛݟΔ͜ͱͰશମͷॲཧঢ়گΛѲ͢Δ
ʲ՝ʳୀ͢Δͱফ͑ͯ͠·͏ ϗετϝτϦοΫΛ͍͍ͨ
ΧελϜϝτϦοΫϗετ͕ୀ͢Δͱফ͑ͯ͠·͏
ʲղ๏ʳmaprobe ͰϗετϝτϦοΫΛूอଘ aggregates: - service: production role: push-server metrics: -
name: custom.push.messages.sent outputs: - func: sum name: custom.push.messages.total_sent αʔϏεQSPEVDUJPO ϩʔϧQVTITFSWFSʹରͯ͠ ϗετϝτϦοΫͷQVTINFTTBHFTTFOUΛશऔಘ ˠԋࢉͨ݁͠ՌΛαʔϏεϝτϦοΫͱͯ͠อଘ͢Δ
maprobe aggregate func/ons ݱࡏTVN NJO NBY BWFSBHF DPVOUΛαϙʔτ QFSDFOUJMF͋ͬͨ΄͏͕Αͦ͞͏͚ͩͲະ࣮ ΧελϜϝτϦοΫ͕ফ͑ͳ͚ΕࣜάϥϑͰ݁ͳͷͰԿଔ
ʲ՝ʳͬͱΫϥυωΠςΟϒͳ Ϧιʔεͷࢹ
ϚΠΫϩαʔϏεʂ αʔϏεϝογϡʂ &OWPZΛ͍࢝Ί͍ͯΔͷͰɺϝτϦΫεΛऔΓ͍ͨ ͱ͋Δ&OWPZͷ/statsΛୟ͘ͱʜ $ curl -s x.x.x.x:9901/stats ... cluster.web.default.total_match_count:
1 cluster.web.external.upstream_rq_200: 988 cluster.web.external.upstream_rq_2xx: 988 cluster.web.external.upstream_rq_302: 13 cluster.web.external.upstream_rq_3xx: 13 cluster.web.external.upstream_rq_400: 3 cluster.web.external.upstream_rq_403: 26 cluster.web.external.upstream_rq_404: 5 cluster.web.external.upstream_rq_4xx: 34 .... IUUQTFOWPZQSPYZJP
͜ΕΛશ෦ Mackerel ʹૹΕΑ͍ʁ cluster.web.default.total_match_count: 1 cluster.web.external.upstream_rq_200: 988 cluster.web.external.upstream_rq_2xx: 988 cluster.web.external.upstream_rq_302:
13 cluster.web.external.upstream_rq_3xx: 13 cluster.web.external.upstream_rq_400: 3 cluster.web.external.upstream_rq_403: 26 cluster.web.external.upstream_rq_404: 5 cluster.web.external.upstream_rq_4xx: 34 ...
͍߹Θͤ ϑΟʔυόοΫϑΥʔϜ͔Βؾܰʹฉ͍ͯΈͨ ۙʑenvoy(https://www.envoyproxy.io/)Λಋೖ༧ఆͳͷͰ͕͢ɺMackerel ެࣜͱͯ͠envoy statsऔಘϓϥάΠϯΛެ։͞ΕΔ༧ఆ͋Γ·͢Ͱ͠ΐ͏͔ʁ envoyͷϓϥάΠϯͰ͕͢ɺݱ࣌Ͱެࣜͱͯ͠ެ։͢Δ༧ఆ͍͟͝ ·ͤΜɻ ! ࡞Δ͔ʜʜ
͔͠͠ Envoy େྔͷϝτϦοΫΛు͖ग़͢ $ curl -s x.x.x.x:9901/stats | wc -l
337 ͜ΕΛશ෦.BDLFSFMʹ͍࣋ͬͯ͘ͱʜ ϚΠΫϩϗετϝτϦοΫϗετˠϗετ૬ &OWPZ͕͍Δϗετ͝ͱʹYԁ ੫ผ
࡞ઓมߋ શ෦Λ͍࣋ͬͯ͘ͷίετ͕ݫ͍͠ ͱ͍͑ͲΜͳ͕΄͍͔͠Α͔͍ͬͯ͘ͳ͍ &OWPZӡ༻ܦݧ͕ઙ͍ͷͰɺӡ༻͠ͳ͕ΒݟΔΛܾΊ͍ͨ QMVHJOΛ࡞ͬͯ༗༻ͳϝτϦοΫΛऔΔʹӡ༻ܦݧ͕ඞཁ ͭ·ΓͱΓ͋͑ͣશ෦औΓ͍ͨ
Prometheus Ͳ͏͔ʁ 1VMMܕϝτϦοΫऩूɾࢹπʔϧ อଘͨ͠Λ1SPN2-ΫΤϦͰॊೈʹՃͯ͠औಘͰ͖Δ ظؒͷϝτϦοΫอଘ͋·Γߟྀ͞Ε͍ͯͳ͍ (SBGBOBͱ͔ͰՄࢹԽ͕ී௨ʁ
! Envoy → Prometheus → Mackerel զʑ.BDLFSFMͰΞϥʔτ͍ͨ͠͠άϥϑݟ͍ͨ ͲͷΛऔΔ͖͔͕ݟ͑ͳ͍ͷͰɺۙશ෦औ͓͖͍ͬͯͨ &OWPZˠ1SPNFUIFVT
͜Ε͋Δ 1SPNFUIFVTˠΫΤϦ݁ՌΛNFUSJDQMVHJOܗࣜͰग़ྗ NFUSJDQMVHJOܗࣜͷग़ྗΛ.BDLFSFMʹอଘ ͜Ε͋Δ ͜ͷ͚ͩ࡞ΕΑ͍ͷͰʂ Γͳ͍1SPNFUIFVTʹΫΤϦͯ͠औΕΑ͍
mackrel-plugin-prometheus-query ࡞Γ·ͨ͠ HJUIVCDPNGVKJXBSBNBDLFSFMQMVHJOQSPNFUIFVTRVFSZ $ mackerel-plugin-prometheus-query \ -query "up" \ -metric-key-format
"promq.{job}.{instance}" promq.web.10_1_129_175_9901 1 1575941187 promq.web.10_1_130_170_9901 1 1575941187 promq.web.10_1_131_53_9901 1 1575941187 promq.prometheus.localhost_9090 1 1575941187
Prometheus ͷΫΤϦྫ ྫɿؒͷVQTUSFBNͷϦτϥΠճΛٻΊΔΫΤϦ sum( delta( envoy_cluster_upstream_rq_retry{envoy_cluster_name="web"}[1m] ) ) .BDLFSFMʹϗετ୯ҐͰͳ͘શମͷΛૹΔ ݸʑͷ1SPNFUIFVTΛݟΕ͋ΔͷͰ
plugin ͷग़ྗΛ mkr throw Ͱ͛Δ $ mackerel-plugin-prometheus-query \ -query 'sum(delta(envoy_cluster_upstream_rq_retry_success
{envoy_cluster_name="web"}[1m]))' \ -metric-key-format "envoy.web.upstream.retry.success" \ | mkr throw --service production
·ͱΊ ΫϥυωΠςΟϒͳࢹͱ োΛڐ༰͢Δઃܭશମͷ݈શੑΛݟΔ ͦͷಓ۩ͱͯ͠ͷܺؒՈ۩044 w NBQSPCF ࣗಈैɺ֎ܗࢹɺQMVHJOࢹɺϝτϦοΫू w NBDLFSFMQMVHJOQSPNFUIFVTRVFSZ TIPSUUFSN1SPNFUIFVTͰ
MPOHUFSN BMFSUJOH.BDLFSFMͰ