Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
BASEにおけるMackerel利用上の工夫と困りごとのご紹介 / Mackerel Me...
Search
Fumitaka Yamane
March 01, 2019
Technology
0
15k
BASEにおけるMackerel利用上の工夫と困りごとのご紹介 / Mackerel Meetup #13 Tokyo
2019/03/01 Mackerel Meetup #13 Tokyo
Fumitaka Yamane
March 01, 2019
Tweet
Share
Other Decks in Technology
See All in Technology
受託開発でもアジャイル開発できました / Agile in Contract Development
takaking22
9
4.5k
CData Virtuality 日本ローンチイベントのKeynote
cdataj
0
230
電子辞書にステータスバーを実装する
puhitaku
0
110
【shownet.conf_】ShowNet伝送改めShowNet APN 2024
shownet
PRO
0
450
KubeVirt Networking ONIC 2024
orimanabu
4
620
LINEヤフー新卒採用 コーディングテスト解説 アルゴリズム問題編
lycorp_recruit_jp
0
13k
Slackbot × RAG で実現する社内情報検索の最適化
howdy39
2
340
LINEヤフー新卒採用 コーディングテスト解説 実装問題編
lycorp_recruit_jp
1
12k
スタサプ ForSCHOOLアプリのシンプルな設計
recruitengineers
PRO
3
630
New Relicを活用したシステム監視の強化とオブザーバビリティ向上
sugoto911
1
100
Microsoft 365 でデータセキュリティを強化しよう
sophiakunii
1
100
入社半年(合計1年)でGoogle Cloud 認定を全冠した秘訣🤫
risatube
1
230
Featured
See All Featured
GraphQLとの向き合い方2022年版
quramy
43
13k
Imperfection Machines: The Place of Print at Facebook
scottboms
264
13k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
280
13k
Raft: Consensus for Rubyists
vanstee
136
6.6k
In The Pink: A Labor of Love
frogandcode
139
22k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
4
120
YesSQL, Process and Tooling at Scale
rocio
167
14k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
1
290
VelocityConf: Rendering Performance Case Studies
addyosmani
325
23k
Robots, Beer and Maslow
schacon
PRO
157
8.2k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
231
17k
Stop Working from a Prison Cell
hatefulcrawdad
267
20k
Transcript
BASEʹ͓͚ΔMackerelར༻্ͷͱࠔΓ͝ͱ ͷ͝հ #"4&גࣜձࣾ43& ࢁ࢙ࠜਹ .BDLFSFM.FFUVQ5PLZP
ຊ… BASEͰMackerelΛར༻͍ͯ͠ΔதͰͷɺͪΐͬͱͨ͠ ɺࠔͬͨ͜ͱͱͦΕʹର͢ΔϫʔΫΞϥϯυɺ MackerelͷཁͳͲΛհ͠·͢ ࢀߟʹͳΕخ͍͠Ͱ͢
ࣗݾհ ࢁ࢙ࠜਹʢ·Ͷ;Έ͔ͨʣ BASEגࣜձࣾɹSREάϧʔϓ 2016ೖࣾ ೖࣾ࣌ʹMackerel͕ಋೖࡁΈͩͬͨ લ৬ʢιʔγϟϧήʔϜʣͰMackerelͬͯͨ twitter / ͯͳID: @fumikony
BASEͷϓϩμΫτʹ͍ͭͯ ωοτγϣοϓ࡞αʔϏε ʮBASEʯ γϣοϐϯάΞϓϦ ʮBASEʯ ՁͷަΛΑΓγϯϓϧʹ͠ɺ ੈքதͷਓʑ͕࠷దͳܦࡁ׆ಈΛߦ͑ΔΑ͏ʹ͢Δɻ MISSION
BASEͷϓϩμΫτʹ͍ͭͯ ωοτγϣοϓ࡞αʔϏε ʮBASEʯ γϣοϐϯάΞϓϦ ʮBASEʯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ
BASEͷϓϩμΫτʹ͍ͭͯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ αʔϏεͷن֦େʹͱͳͬͯোͷӨڹେ͖͘
BASEͷϓϩμΫτʹ͍ͭͯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ ҟৗ͕͋ͬͨΒ͙͢ʹؾ͖͍ͮͨ MackerelͰࢹ
࣍ 1. BASEͷΠϯϑϥߏʹ͍ͭͯ 2. Mackerelར༻্ͷ 3. Mackerelར༻্ͷࠔΓ͝ͱ 4. Mackerelͷ͍͍ͱ͜Ζ
1. BASEͷΠϯϑϥߏʹ͍ͭͯ
BASEͷΠϯϑϥߏʹ͍ͭͯ ֓ཁ ͍͍ͩͨAWS EC2, ECS, ELB, Lambda, ElastiCache, RDS, SQS,
CloudFront, S3, SES, etc… MackerelͰࢹ SlackʹΞϥʔτ௨
BASEͷΠϯϑϥߏʹ͍ͭͯ αʔόͷछྨ B/GσϓϩΠ͢Δαʔό σϓϩΠ͢Δͨͼʹ Πϯελϯε࡞ ݹ͍αʔόফ͍͑ͯ͘ ΞϓϦέʔγϣϯαʔό ʢApache+PHPʣ
BASEͷΠϯϑϥߏʹ͍ͭͯ αʔόͷछྨ B/GσϓϩΠ͠ͳ͍αʔό ͣͬͱ͋Δ όοναʔόͳͲ
2. Mackerelར༻্ͷ
Mackerelར༻্ͷ /health endpoint pattern OKͩͬͨΒεςʔλε200Λฦ͚ͩ͢ͷURL ֎ܗࢹͷର /health ͱ /health/deep /health
αʔό/ίϯςφͷੜଘ /health/deep DBͷଓ·Ͱ֬ೝ ͪ͜ΒΛ֎ܗࢹͰ༻ See also: ʮೖ ࢹ ―ϞμϯͳϞχλϦϯάͷͨΊͷσβΠϯύλʔϯʯ ɹɹɹɹɹ(https://www.oreilly.co.jp/books/9784873118642/) ɹɹɹɹɹஶ: Mike Julian ɹɹɹɹɹ7.3 health ΤϯυϙΠϯτύλʔϯ
Mackerelར༻্ͷ /health endpoint pattern ֎ܗࢹ͕Τϥʔʹͳͬͨͱ͖ͷ௨
Mackerelར༻্ͷ monitoringαʔό MackerelͷΧελϜϝτϦοΫαʔϏεϝτϦοΫΛऔಘɺ ߘ͢ΔͨΊͷαʔό AWSΠϯςάϨʔγϣϯະରԠͷϚωʔδυαʔϏεͷϝτϦο ΫΛऔಘ ରɿSES ҎલCloudFrontର͕ͩͬͨɺAWSΠϯςάϨʔγϣϯ͕ ରԠͨ͠ͷͰෆཁʹ
Mackerelར༻্ͷ monitoringαʔό AWSΠϯςάϨʔγϣϯ+ϓϥάΠϯ ର: RDS AWSΠϯςάϨʔγϣϯͰϝτϦοΫΛऔಘ monitoringαʔόͰmackerel-plugin-mysqlͰϝτϦοΫΛ औಘ custom_identifier Λࢦఆͯ͠ಉ͡ϗετʹू
Mackerelར༻্ͷ monitoringαʔό [plugin.metrics.mysql-prd-main] command = "mackerel-plugin-mysql -host=db1.xxx.ap-northeast-1.rds.amazonaws.com" custom_identifier = "db1.xxx.ap-northeast-1.rds.amazonaws.com"
See also: https://mackerel.io/ja/docs/entry/integrations/aws mackerel-agent.confʢൈਮʣ
Mackerelར༻্ͷ ࣜʹΑΔࢹ scale(divide( host(xxxxxxxxxxx, custom.ses.stats.Bounces), host(xxxxxxxxxxx, custom.ses.stats.DeliveryAttempts)), 100) mackerel-plugin-sesͰऔಘͰ͖Δͷ৴ͱόϯεͦΕͧΕͷ ݅ͳͷͰɺࣜͰܭࢉͨ͠Λࢹ
όϯε݅ / ৴݅ * 100 See Also: https://mackerel.io/ja/docs/entry/expression-monitoring SESͷBounce Rate (%)
Mackerelར༻্ͷ ࣜʹΑΔࢹ scale(divide( host(xxxxxxxxxxx, custom.ses.stats.Bounces), host(xxxxxxxxxxx, custom.ses.stats.DeliveryAttempts)), 100) SESͷBounce Rate
(%)
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ mkr MackerelͷCLIπʔϧ mkr monitors ࢹϧʔϧΛૢ࡞͢ΔαϒίϚϯυ mkr monitors pull
monitors.json ͱͯ͠ࢹϧʔϧΛอଘ mkr monitors diff monitors.json ͱࢹϧʔϧͷࠩΛදࣔ mkr monitors push monitors.json Λࢹϧʔϧʹө
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ mkr MackerelͷCLIπʔϧ mkr monitors ࢹϧʔϧΛૢ࡞͢ΔαϒίϚϯυ mkr monitors pull
monitors.json ͱͯ͠ࢹϧʔϧΛอଘ mkr monitors diff monitors.json ͱࢹϧʔϧͷࠩΛදࣔ mkr monitors push monitors.json Λࢹϧʔϧʹө
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ mkr monitors pull → git commit → GitHubʹgit
push CircleCIͰσΠϦʔ࣮ߦ มߋཤྺΛ͑Δ ·͕ͪ͑ͯյͯͤ͠Δ όοΫΞοϓతͳ͍ํʹཹΊ͍ͯΔ mkr monitors pushΘͣɺWebUIͰͷมߋͷΈ WebUIศར
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code
|| git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code
|| git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code
|| git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ
Mackerelར༻্ͷ ͍߹Θͤ༰ΛSlackͰڞ༗ ʮϝʔϧઃఆʯͰɺ 4MBDLͷʮ&NBJMʯ"QQͷ ϝʔϧΞυϨεΛઃఆ ͍߹Θͤͨ༰͕ 4MBDLʹྲྀΕ͖ͯͯศར
• ίϯςφΤʔδΣϯτࢼͯ͠Έ·ͨ͠ • ECSͷઃఆϛεΛൃݟʂ • ϝϞϦ͕Γͯͳ͔ͬͨʂ
• ϩʔϧҟৗݕ ࢼͯ͠Έ·ͨ͠
• ෛՙࢼݧͷ࣌ʹͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 ɹʮEίϚʔεϓϥοτϑΥʔϜͷΛࢧ͑Δ ɹɹෛՙࢼݧͷऔΓΈʯ •
• ෛՙࢼݧͷ࣌ʹͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 • ApacheͷίωΫγϣϯ͕ுΓ͍ͯͨͷ͕Ϙτϧ ωοΫʹͳ͍ͬͯͨ • MackerelΛΈͨΒ͙͢ʹΘ͔ͬͨʂʢ୲ऀஊʣ
• ෛՙࢼݧͷ࣌ʹͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 • Apache͕ϘτϧωοΫʹͳ͍ͬͯͨ • MackerelΛΈͨΒ͙͢ʹΘ͔ͬͨʂʢ୲ऀஊʣ
ʢMackerelΛݟͳ͕ΒΘ͍Θ͍͍ͬͯΔʣ
3. Mackerelར༻্ͷࠔΓ͝ͱ
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̍ʣ ͠ΐͬͪΎ͏ग़ͬͺͳ͠ͷΞϥʔτ र͏͖Ξϥʔτ͕ຒΕΔ ͱ͍͑ࢹΛશ͘Δͷɻɻɻ ௨νϟϯωϧΛ͚ΔϫʔΫΞϥϯυ ຊɺͦΜͳΞϥʔτແ͍Α͏ʹ͢Δ͖͕ͩ…
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̍ʣ ॏཁʹԠͯ͡௨νϟϯωϧΛ͚Δ #base_infra_alert ॏཁɿߴ #base_infra_alert_warn ॏཁɿத #base_infra_alert_info ॏཁɿ
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̎ʣ #base_infra_alert ʹMackerel͔Βͷ௨Ҏ֎ͷͷྲྀ͍ͯ͠Δ ϝʔϧͰདྷΔͷɺRSSϑΟʔυͳͲ ͜ΕΒʹ mention ͳ͍ͷͰɺSlackͰ͜ͷνϟϯωϧͷ௨ઃఆ Λڧͯ͘͠ؾ͚ͮΔΑ͏ʹ͢Δ͕…
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̎ʣ ௨͞Ε͗͢Δ #base_infra_alert ্ͷձ͕શ෦௨͞ΕΔʂ #base_infra_alert ʹ௨Λྲྀ͚ͩ͢ʹ͢Δ ٞผͷνϟϯωϧͰ ʮReacji ChannelerʯͰٞ༻ͷνϟϯωϧʹ
ϦϯΫΛίϐʔ https://get.slack.help/hc/ja/articles/360000482666-ଞνϟϯωϧϝοηʔδΛαΫοͱίϐʔ
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̎ʣ Ξϥʔτ௨ʹֆจࣈͰԠ͢Δͱ…
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̎ʣ ٞ༻ͷνϟϯωϧʹϦϯΫ͕ίϐʔ͞ΕΔʂ ௨͕ݮͬͯগ͠ฏʹ…
Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ αʔϏεͷͱͱʹαʔόͷछྨ͕૿͑ͨ αϒγεςϜͷ૿Ճ ϩʔϧͷԽ ϚωʔδυαʔϏεΛMackerelͷࢹରʹՃ ϩʔϧ/ϗετͷߏʹ͖Ε͍ʹରԠ͚ʹ͍͘ Lambda, SQSͳͲ
Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ MackerelͰʮϩʔϧʯ͕ࢹΛ͢Δ୯ҐͰ͋Γɺά ϥϑΛ·ͱΊΔ୯ҐͰ͋Δ ࢹ͍ͨ͠୯ҐͱάϥϑͰݟ͍ͨ୯Ґඞͣ͠Ұக ͠ͳ͍ ࢹ͍ͨ͠୯ҐͰϩʔϧΛ͚͍ͯ͘ͱɺ ʮαʔϏεʯͷը໘͕ͪ͝Όͪ͝Ό͖ͯͨ͠
Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ
Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ ࠓϩʔϧ࠶ฤத… Ұ؏ੑͷ͋Δϧʔϧʹج͍ͮͯઃܭ͢͠ ̎ͭͷʮαʔϏεʯΛͬͯϩʔϧΛཧ͢Δ ϩʔϧάϥϑΛҰཡͰΈ͍ͨϩʔϧ → ϝΠϯͷʮαʔϏεʯ ྫɿdb-primary, db-replica
ͦΕҎ֎ͷࢹ୯Ґͱͯ͠ͷϩʔϧ → αϒͷʮαʔϏεʯ ྫɿrds
Mackerelར༻্ͷࠔΓ͝ͱ ΧελϜϝτϦοΫϩʔϧʹͬͯ΄͍͠ αʔό͕ফ͑ΔͱΧελϜϝτϦοΫ͕ফ͑Δ σϓϩΠ͢Δͨͼʹαʔό͕ফ͑ΔͷͰΘΓͱਏ͍ γεςϜϝτϦοΫ͍ͬͯΔ͕… ΧελϜϝτϦοΫϩʔϧʹͬͯ΄͍͠ʂ ͳʹͱͧ…
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ աڈͷάϥϑͷϐʔΫ͕Θ͔Γʹ͍͘ ྫ͑ɺաڈͰҰ൪ϦΫΤετ͕དྷͯͨͱ͖ͷ ؒϦΫΤετͲͷ͘Β͍͔ͩͬͨͳʁͱ͍͏࣌ ͱΓ͋͑ͣʮ1YʯͷάϥϑΛݟͯΈΔ…ʁ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ࣌ؒൣғΛ͘औΔͱϐʔΫ͕ͳ·Δ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ࣌ؒൣғΛ͘औΔͱϐʔΫ͕ͳ·Δ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ࣌ؒൣғΛ͘औΔͱϐʔΫ͕ͳ·Δ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ࣌ؒൣғΛ͘औΔͱϐʔΫ͕ͳ·Δ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ʮ1YʯͷάϥϑͰϐʔΫ͕ݟ͑ͯ΄͍͠ ͳʹͱͧ…
4. Mackerelͷ͍͍ͱ͜Ζ
Mackerelͷ͍͍ͱ͜Ζ ࣗͰӡ༻͠ͳ͍͍ͯ͘ ͲΜͲΜػೳڧԽ͞Ε͍ͯ͘ AWSΠϯςάϨʔγϣϯ͕ศར ϓϥάΠϯͷՃָ͕ https://github.com/mackerelio/plugin-registry https://github.com/mackerelio/mackerel-agent-plugins ࣗ࡞؆୯ ʮαϙʔτνʔϜ࿈བྷʯͷϨεϙϯε͕Α͍
Mackerelͷ͍͍ͱ͜Ζ WebUI͕Θ͔Γ͍͢ άϥϑ͕͖Ε͍ API͕͋ͬͯϓϩάϥϜ͔Βૢ࡞Ͱ͖Δ CLI͕͋Δ ͍͡Ί͍͢ ΤʔδΣϯτΛΠϯετʔϧͯ͠ϩʔϧΛઃఆ͢Δ͚ͩͰɺ ͍͍ͩͨྑ͍ײ͡ͷάϥϑ͕ग़ͯ͘Δ ຊޠͩ UIɺυΩϡϝϯτ
ࢀߟਤॻ ʮMackerel αʔόࢹ [࣮ફ] ೖʯ ஶ: Ҫ্େีɼപ୩େีɼਿࢁ௨ɼాத৻࢘ɼ௶༎थɼদխ (http://gihyo.jp/book/2017/978-4-7741-9213-0) ʮೖ ࢹ
―ϞμϯͳϞχλϦϯάͷͨΊͷσβΠϯύλʔϯʯ ஶ: Mike Julian (https://www.oreilly.co.jp/books/9784873118642/) ʮιϑτΣΞΤϯδχΞͷͨΊͷɹITΠϯϑϥࢹʦ࣮ફʧೖʯ ஶ: ࡈ౻༞Ұ (https://gihyo.jp/book/2016/978-4-7741-7865-3)
We are hiring! SREͪΖΜɺ༷ʑͳ৬छΛืू͍ͯ͠·͢ ΧδϡΞϧʹձࣾʹ༡ͼʹདྷ͍ͨํ·Ͱ https://jobs.binc.jp/
ςοΫϒϩάߋ৽ͯ͠·͢ https://devblog.thebase.in/
ΦϯυϝσΟΞߋ৽ͯ͠·͢ https://basebook.binc.jp/
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠