Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
BASEにおけるMackerel利用上の工夫と困りごとのご紹介 / Mackerel Me...
Search
Fumitaka Yamane
March 01, 2019
Technology
0
16k
BASEにおけるMackerel利用上の工夫と困りごとのご紹介 / Mackerel Meetup #13 Tokyo
2019/03/01 Mackerel Meetup #13 Tokyo
Fumitaka Yamane
March 01, 2019
Tweet
Share
Other Decks in Technology
See All in Technology
権威ドキュメントで振り返る2024 #年忘れセキュリティ2024
hirotomotaguchi
2
730
How to be an AWS Community Builder | 君もAWS Community Builderになろう!〜2024 冬 CB募集直前対策編?!〜
coosuke
PRO
2
2.8k
KubeCon NA 2024 Recap: How to Move from Ingress to Gateway API with Minimal Hassle
ysakotch
0
200
ハイテク休憩
sat
PRO
2
120
PHPからGoへのマイグレーション for DMMアフィリエイト
yabakokobayashi
1
160
継続的にアウトカムを生み出し ビジネスにつなげる、 戦略と運営に対するタイミーのQUEST(探求)
zigorou
0
510
コンテナセキュリティのためのLandlock入門
nullpo_head
2
320
バクラクのドキュメント解析技術と実データにおける課題 / layerx-ccc-winter-2024
shimacos
2
1k
Microsoft Azure全冠になってみた ~アレを使い倒した者が試験を制す!?~/Obtained all Microsoft Azure certifications Those who use "that" to the full will win the exam! ?
yuj1osm
1
110
マイクロサービスにおける容易なトランザクション管理に向けて
scalar
0
110
10個のフィルタをAXI4-Streamでつなげてみた
marsee101
0
160
C++26 エラー性動作
faithandbrave
2
680
Featured
See All Featured
Making Projects Easy
brettharned
116
5.9k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
5
440
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
169
50k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
365
25k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
280
13k
Docker and Python
trallard
41
3.1k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
127
18k
Designing Experiences People Love
moore
138
23k
The Art of Programming - Codeland 2020
erikaheidi
53
13k
Typedesign – Prime Four
hannesfritz
40
2.4k
How GitHub (no longer) Works
holman
311
140k
Stop Working from a Prison Cell
hatefulcrawdad
267
20k
Transcript
BASEʹ͓͚ΔMackerelར༻্ͷͱࠔΓ͝ͱ ͷ͝հ #"4&גࣜձࣾ43& ࢁ࢙ࠜਹ .BDLFSFM.FFUVQ5PLZP
ຊ… BASEͰMackerelΛར༻͍ͯ͠ΔதͰͷɺͪΐͬͱͨ͠ ɺࠔͬͨ͜ͱͱͦΕʹର͢ΔϫʔΫΞϥϯυɺ MackerelͷཁͳͲΛհ͠·͢ ࢀߟʹͳΕخ͍͠Ͱ͢
ࣗݾհ ࢁ࢙ࠜਹʢ·Ͷ;Έ͔ͨʣ BASEגࣜձࣾɹSREάϧʔϓ 2016ೖࣾ ೖࣾ࣌ʹMackerel͕ಋೖࡁΈͩͬͨ લ৬ʢιʔγϟϧήʔϜʣͰMackerelͬͯͨ twitter / ͯͳID: @fumikony
BASEͷϓϩμΫτʹ͍ͭͯ ωοτγϣοϓ࡞αʔϏε ʮBASEʯ γϣοϐϯάΞϓϦ ʮBASEʯ ՁͷަΛΑΓγϯϓϧʹ͠ɺ ੈքதͷਓʑ͕࠷దͳܦࡁ׆ಈΛߦ͑ΔΑ͏ʹ͢Δɻ MISSION
BASEͷϓϩμΫτʹ͍ͭͯ ωοτγϣοϓ࡞αʔϏε ʮBASEʯ γϣοϐϯάΞϓϦ ʮBASEʯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ
BASEͷϓϩμΫτʹ͍ͭͯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ αʔϏεͷن֦େʹͱͳͬͯোͷӨڹେ͖͘
BASEͷϓϩμΫτʹ͍ͭͯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ ҟৗ͕͋ͬͨΒ͙͢ʹؾ͖͍ͮͨ MackerelͰࢹ
࣍ 1. BASEͷΠϯϑϥߏʹ͍ͭͯ 2. Mackerelར༻্ͷ 3. Mackerelར༻্ͷࠔΓ͝ͱ 4. Mackerelͷ͍͍ͱ͜Ζ
1. BASEͷΠϯϑϥߏʹ͍ͭͯ
BASEͷΠϯϑϥߏʹ͍ͭͯ ֓ཁ ͍͍ͩͨAWS EC2, ECS, ELB, Lambda, ElastiCache, RDS, SQS,
CloudFront, S3, SES, etc… MackerelͰࢹ SlackʹΞϥʔτ௨
BASEͷΠϯϑϥߏʹ͍ͭͯ αʔόͷछྨ B/GσϓϩΠ͢Δαʔό σϓϩΠ͢Δͨͼʹ Πϯελϯε࡞ ݹ͍αʔόফ͍͑ͯ͘ ΞϓϦέʔγϣϯαʔό ʢApache+PHPʣ
BASEͷΠϯϑϥߏʹ͍ͭͯ αʔόͷछྨ B/GσϓϩΠ͠ͳ͍αʔό ͣͬͱ͋Δ όοναʔόͳͲ
2. Mackerelར༻্ͷ
Mackerelར༻্ͷ /health endpoint pattern OKͩͬͨΒεςʔλε200Λฦ͚ͩ͢ͷURL ֎ܗࢹͷର /health ͱ /health/deep /health
αʔό/ίϯςφͷੜଘ /health/deep DBͷଓ·Ͱ֬ೝ ͪ͜ΒΛ֎ܗࢹͰ༻ See also: ʮೖ ࢹ ―ϞμϯͳϞχλϦϯάͷͨΊͷσβΠϯύλʔϯʯ ɹɹɹɹɹ(https://www.oreilly.co.jp/books/9784873118642/) ɹɹɹɹɹஶ: Mike Julian ɹɹɹɹɹ7.3 health ΤϯυϙΠϯτύλʔϯ
Mackerelར༻্ͷ /health endpoint pattern ֎ܗࢹ͕Τϥʔʹͳͬͨͱ͖ͷ௨
Mackerelར༻্ͷ monitoringαʔό MackerelͷΧελϜϝτϦοΫαʔϏεϝτϦοΫΛऔಘɺ ߘ͢ΔͨΊͷαʔό AWSΠϯςάϨʔγϣϯະରԠͷϚωʔδυαʔϏεͷϝτϦο ΫΛऔಘ ରɿSES ҎલCloudFrontର͕ͩͬͨɺAWSΠϯςάϨʔγϣϯ͕ ରԠͨ͠ͷͰෆཁʹ
Mackerelར༻্ͷ monitoringαʔό AWSΠϯςάϨʔγϣϯ+ϓϥάΠϯ ର: RDS AWSΠϯςάϨʔγϣϯͰϝτϦοΫΛऔಘ monitoringαʔόͰmackerel-plugin-mysqlͰϝτϦοΫΛ औಘ custom_identifier Λࢦఆͯ͠ಉ͡ϗετʹू
Mackerelར༻্ͷ monitoringαʔό [plugin.metrics.mysql-prd-main] command = "mackerel-plugin-mysql -host=db1.xxx.ap-northeast-1.rds.amazonaws.com" custom_identifier = "db1.xxx.ap-northeast-1.rds.amazonaws.com"
See also: https://mackerel.io/ja/docs/entry/integrations/aws mackerel-agent.confʢൈਮʣ
Mackerelར༻্ͷ ࣜʹΑΔࢹ scale(divide( host(xxxxxxxxxxx, custom.ses.stats.Bounces), host(xxxxxxxxxxx, custom.ses.stats.DeliveryAttempts)), 100) mackerel-plugin-sesͰऔಘͰ͖Δͷ৴ͱόϯεͦΕͧΕͷ ݅ͳͷͰɺࣜͰܭࢉͨ͠Λࢹ
όϯε݅ / ৴݅ * 100 See Also: https://mackerel.io/ja/docs/entry/expression-monitoring SESͷBounce Rate (%)
Mackerelར༻্ͷ ࣜʹΑΔࢹ scale(divide( host(xxxxxxxxxxx, custom.ses.stats.Bounces), host(xxxxxxxxxxx, custom.ses.stats.DeliveryAttempts)), 100) SESͷBounce Rate
(%)
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ mkr MackerelͷCLIπʔϧ mkr monitors ࢹϧʔϧΛૢ࡞͢ΔαϒίϚϯυ mkr monitors pull
monitors.json ͱͯ͠ࢹϧʔϧΛอଘ mkr monitors diff monitors.json ͱࢹϧʔϧͷࠩΛදࣔ mkr monitors push monitors.json Λࢹϧʔϧʹө
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ mkr MackerelͷCLIπʔϧ mkr monitors ࢹϧʔϧΛૢ࡞͢ΔαϒίϚϯυ mkr monitors pull
monitors.json ͱͯ͠ࢹϧʔϧΛอଘ mkr monitors diff monitors.json ͱࢹϧʔϧͷࠩΛදࣔ mkr monitors push monitors.json Λࢹϧʔϧʹө
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ mkr monitors pull → git commit → GitHubʹgit
push CircleCIͰσΠϦʔ࣮ߦ มߋཤྺΛ͑Δ ·͕ͪ͑ͯյͯͤ͠Δ όοΫΞοϓతͳ͍ํʹཹΊ͍ͯΔ mkr monitors pushΘͣɺWebUIͰͷมߋͷΈ WebUIศར
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code
|| git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code
|| git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code
|| git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ
Mackerelར༻্ͷ ͍߹Θͤ༰ΛSlackͰڞ༗ ʮϝʔϧઃఆʯͰɺ 4MBDLͷʮ&NBJMʯ"QQͷ ϝʔϧΞυϨεΛઃఆ ͍߹Θͤͨ༰͕ 4MBDLʹྲྀΕ͖ͯͯศར
• ίϯςφΤʔδΣϯτࢼͯ͠Έ·ͨ͠ • ECSͷઃఆϛεΛൃݟʂ • ϝϞϦ͕Γͯͳ͔ͬͨʂ
• ϩʔϧҟৗݕ ࢼͯ͠Έ·ͨ͠
• ෛՙࢼݧͷ࣌ʹͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 ɹʮEίϚʔεϓϥοτϑΥʔϜͷΛࢧ͑Δ ɹɹෛՙࢼݧͷऔΓΈʯ •
• ෛՙࢼݧͷ࣌ʹͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 • ApacheͷίωΫγϣϯ͕ுΓ͍ͯͨͷ͕Ϙτϧ ωοΫʹͳ͍ͬͯͨ • MackerelΛΈͨΒ͙͢ʹΘ͔ͬͨʂʢ୲ऀஊʣ
• ෛՙࢼݧͷ࣌ʹͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 • Apache͕ϘτϧωοΫʹͳ͍ͬͯͨ • MackerelΛΈͨΒ͙͢ʹΘ͔ͬͨʂʢ୲ऀஊʣ
ʢMackerelΛݟͳ͕ΒΘ͍Θ͍͍ͬͯΔʣ
3. Mackerelར༻্ͷࠔΓ͝ͱ
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̍ʣ ͠ΐͬͪΎ͏ग़ͬͺͳ͠ͷΞϥʔτ र͏͖Ξϥʔτ͕ຒΕΔ ͱ͍͑ࢹΛશ͘Δͷɻɻɻ ௨νϟϯωϧΛ͚ΔϫʔΫΞϥϯυ ຊɺͦΜͳΞϥʔτແ͍Α͏ʹ͢Δ͖͕ͩ…
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̍ʣ ॏཁʹԠͯ͡௨νϟϯωϧΛ͚Δ #base_infra_alert ॏཁɿߴ #base_infra_alert_warn ॏཁɿத #base_infra_alert_info ॏཁɿ
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̎ʣ #base_infra_alert ʹMackerel͔Βͷ௨Ҏ֎ͷͷྲྀ͍ͯ͠Δ ϝʔϧͰདྷΔͷɺRSSϑΟʔυͳͲ ͜ΕΒʹ mention ͳ͍ͷͰɺSlackͰ͜ͷνϟϯωϧͷ௨ઃఆ Λڧͯ͘͠ؾ͚ͮΔΑ͏ʹ͢Δ͕…
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̎ʣ ௨͞Ε͗͢Δ #base_infra_alert ্ͷձ͕શ෦௨͞ΕΔʂ #base_infra_alert ʹ௨Λྲྀ͚ͩ͢ʹ͢Δ ٞผͷνϟϯωϧͰ ʮReacji ChannelerʯͰٞ༻ͷνϟϯωϧʹ
ϦϯΫΛίϐʔ https://get.slack.help/hc/ja/articles/360000482666-ଞνϟϯωϧϝοηʔδΛαΫοͱίϐʔ
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̎ʣ Ξϥʔτ௨ʹֆจࣈͰԠ͢Δͱ…
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̎ʣ ٞ༻ͷνϟϯωϧʹϦϯΫ͕ίϐʔ͞ΕΔʂ ௨͕ݮͬͯগ͠ฏʹ…
Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ αʔϏεͷͱͱʹαʔόͷछྨ͕૿͑ͨ αϒγεςϜͷ૿Ճ ϩʔϧͷԽ ϚωʔδυαʔϏεΛMackerelͷࢹରʹՃ ϩʔϧ/ϗετͷߏʹ͖Ε͍ʹରԠ͚ʹ͍͘ Lambda, SQSͳͲ
Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ MackerelͰʮϩʔϧʯ͕ࢹΛ͢Δ୯ҐͰ͋Γɺά ϥϑΛ·ͱΊΔ୯ҐͰ͋Δ ࢹ͍ͨ͠୯ҐͱάϥϑͰݟ͍ͨ୯Ґඞͣ͠Ұக ͠ͳ͍ ࢹ͍ͨ͠୯ҐͰϩʔϧΛ͚͍ͯ͘ͱɺ ʮαʔϏεʯͷը໘͕ͪ͝Όͪ͝Ό͖ͯͨ͠
Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ
Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ ࠓϩʔϧ࠶ฤத… Ұ؏ੑͷ͋Δϧʔϧʹج͍ͮͯઃܭ͢͠ ̎ͭͷʮαʔϏεʯΛͬͯϩʔϧΛཧ͢Δ ϩʔϧάϥϑΛҰཡͰΈ͍ͨϩʔϧ → ϝΠϯͷʮαʔϏεʯ ྫɿdb-primary, db-replica
ͦΕҎ֎ͷࢹ୯Ґͱͯ͠ͷϩʔϧ → αϒͷʮαʔϏεʯ ྫɿrds
Mackerelར༻্ͷࠔΓ͝ͱ ΧελϜϝτϦοΫϩʔϧʹͬͯ΄͍͠ αʔό͕ফ͑ΔͱΧελϜϝτϦοΫ͕ফ͑Δ σϓϩΠ͢Δͨͼʹαʔό͕ফ͑ΔͷͰΘΓͱਏ͍ γεςϜϝτϦοΫ͍ͬͯΔ͕… ΧελϜϝτϦοΫϩʔϧʹͬͯ΄͍͠ʂ ͳʹͱͧ…
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ աڈͷάϥϑͷϐʔΫ͕Θ͔Γʹ͍͘ ྫ͑ɺաڈͰҰ൪ϦΫΤετ͕དྷͯͨͱ͖ͷ ؒϦΫΤετͲͷ͘Β͍͔ͩͬͨͳʁͱ͍͏࣌ ͱΓ͋͑ͣʮ1YʯͷάϥϑΛݟͯΈΔ…ʁ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ࣌ؒൣғΛ͘औΔͱϐʔΫ͕ͳ·Δ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ࣌ؒൣғΛ͘औΔͱϐʔΫ͕ͳ·Δ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ࣌ؒൣғΛ͘औΔͱϐʔΫ͕ͳ·Δ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ࣌ؒൣғΛ͘औΔͱϐʔΫ͕ͳ·Δ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ʮ1YʯͷάϥϑͰϐʔΫ͕ݟ͑ͯ΄͍͠ ͳʹͱͧ…
4. Mackerelͷ͍͍ͱ͜Ζ
Mackerelͷ͍͍ͱ͜Ζ ࣗͰӡ༻͠ͳ͍͍ͯ͘ ͲΜͲΜػೳڧԽ͞Ε͍ͯ͘ AWSΠϯςάϨʔγϣϯ͕ศར ϓϥάΠϯͷՃָ͕ https://github.com/mackerelio/plugin-registry https://github.com/mackerelio/mackerel-agent-plugins ࣗ࡞؆୯ ʮαϙʔτνʔϜ࿈བྷʯͷϨεϙϯε͕Α͍
Mackerelͷ͍͍ͱ͜Ζ WebUI͕Θ͔Γ͍͢ άϥϑ͕͖Ε͍ API͕͋ͬͯϓϩάϥϜ͔Βૢ࡞Ͱ͖Δ CLI͕͋Δ ͍͡Ί͍͢ ΤʔδΣϯτΛΠϯετʔϧͯ͠ϩʔϧΛઃఆ͢Δ͚ͩͰɺ ͍͍ͩͨྑ͍ײ͡ͷάϥϑ͕ग़ͯ͘Δ ຊޠͩ UIɺυΩϡϝϯτ
ࢀߟਤॻ ʮMackerel αʔόࢹ [࣮ફ] ೖʯ ஶ: Ҫ্େีɼപ୩େีɼਿࢁ௨ɼాத৻࢘ɼ௶༎थɼদխ (http://gihyo.jp/book/2017/978-4-7741-9213-0) ʮೖ ࢹ
―ϞμϯͳϞχλϦϯάͷͨΊͷσβΠϯύλʔϯʯ ஶ: Mike Julian (https://www.oreilly.co.jp/books/9784873118642/) ʮιϑτΣΞΤϯδχΞͷͨΊͷɹITΠϯϑϥࢹʦ࣮ફʧೖʯ ஶ: ࡈ౻༞Ұ (https://gihyo.jp/book/2016/978-4-7741-7865-3)
We are hiring! SREͪΖΜɺ༷ʑͳ৬छΛืू͍ͯ͠·͢ ΧδϡΞϧʹձࣾʹ༡ͼʹདྷ͍ͨํ·Ͱ https://jobs.binc.jp/
ςοΫϒϩάߋ৽ͯ͠·͢ https://devblog.thebase.in/
ΦϯυϝσΟΞߋ৽ͯ͠·͢ https://basebook.binc.jp/
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠