Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
BASEにおけるMackerel利用上の工夫と困りごとのご紹介 / Mackerel Meetup #13 Tokyo
Search
Fumitaka Yamane
March 01, 2019
Technology
0
14k
BASEにおけるMackerel利用上の工夫と困りごとのご紹介 / Mackerel Meetup #13 Tokyo
2019/03/01 Mackerel Meetup #13 Tokyo
Fumitaka Yamane
March 01, 2019
Tweet
Share
Other Decks in Technology
See All in Technology
Postman v10リリース後を振り返る / Looking back at Postman v10 after release
yokawasa
1
160
地理空間データ可視化・解析・活用ソリューション Pacific Spatial Solutions (PSS)
pacificspatialsolutions
0
300
APIファーストなプロダクトマネジメントの実践 〜SaaSus Platformでの例〜 / "Practicing API-First Product Management - An Example with SaaSus Platform
oztick139
0
110
ワールドカフェI /チューターを改良する / World Café I and Improving the Tutors
ks91
PRO
0
120
DevOpsメトリクスとアウトカムの接続にトライ!開発プロセスを通して計測できるメトリクスの活用方法
ham0215
2
240
LangSmith入門―トレース/評価/プロンプト管理などを担うLLMアプリ開発プラットフォーム
os1ma
3
350
Janus
bkuhlmann
1
490
データベース02: データベースの概念
trycycle
0
160
エンジニア候補者向け資料2024.04.24.pdf
macloud
0
3.3k
現代CSSフレームワークの内部実装とその仕組み
poteboy
7
3.6k
複雑な構成要素を持つUIとの向き合い方 〜新・支出グラフでの実例〜 / B43 TECH TALK
nakamuuu
0
140
Cloud Native Java with Spring Boot (CNCF Aarhus, April 2024)
thomasvitale
1
180
Featured
See All Featured
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
125
32k
Design by the Numbers
sachag
274
18k
Become a Pro
speakerdeck
PRO
11
4.5k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
357
22k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
274
13k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
187
16k
What's in a price? How to price your products and services
michaelherold
237
11k
Imperfection Machines: The Place of Print at Facebook
scottboms
260
12k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
322
20k
Product Roadmaps are Hard
iamctodd
44
9.7k
Atom: Resistance is Futile
akmur
259
25k
How to train your dragon (web standard)
notwaldorf
73
5.2k
Transcript
BASEʹ͓͚ΔMackerelར༻্ͷͱࠔΓ͝ͱ ͷ͝հ #"4&גࣜձࣾ43& ࢁ࢙ࠜਹ .BDLFSFM.FFUVQ5PLZP
ຊ… BASEͰMackerelΛར༻͍ͯ͠ΔதͰͷɺͪΐͬͱͨ͠ ɺࠔͬͨ͜ͱͱͦΕʹର͢ΔϫʔΫΞϥϯυɺ MackerelͷཁͳͲΛհ͠·͢ ࢀߟʹͳΕخ͍͠Ͱ͢
ࣗݾհ ࢁ࢙ࠜਹʢ·Ͷ;Έ͔ͨʣ BASEגࣜձࣾɹSREάϧʔϓ 2016ೖࣾ ೖࣾ࣌ʹMackerel͕ಋೖࡁΈͩͬͨ લ৬ʢιʔγϟϧήʔϜʣͰMackerelͬͯͨ twitter / ͯͳID: @fumikony
BASEͷϓϩμΫτʹ͍ͭͯ ωοτγϣοϓ࡞αʔϏε ʮBASEʯ γϣοϐϯάΞϓϦ ʮBASEʯ ՁͷަΛΑΓγϯϓϧʹ͠ɺ ੈքதͷਓʑ͕࠷దͳܦࡁ׆ಈΛߦ͑ΔΑ͏ʹ͢Δɻ MISSION
BASEͷϓϩμΫτʹ͍ͭͯ ωοτγϣοϓ࡞αʔϏε ʮBASEʯ γϣοϐϯάΞϓϦ ʮBASEʯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ
BASEͷϓϩμΫτʹ͍ͭͯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ αʔϏεͷن֦େʹͱͳͬͯোͷӨڹେ͖͘
BASEͷϓϩμΫτʹ͍ͭͯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ ҟৗ͕͋ͬͨΒ͙͢ʹؾ͖͍ͮͨ MackerelͰࢹ
࣍ 1. BASEͷΠϯϑϥߏʹ͍ͭͯ 2. Mackerelར༻্ͷ 3. Mackerelར༻্ͷࠔΓ͝ͱ 4. Mackerelͷ͍͍ͱ͜Ζ
1. BASEͷΠϯϑϥߏʹ͍ͭͯ
BASEͷΠϯϑϥߏʹ͍ͭͯ ֓ཁ ͍͍ͩͨAWS EC2, ECS, ELB, Lambda, ElastiCache, RDS, SQS,
CloudFront, S3, SES, etc… MackerelͰࢹ SlackʹΞϥʔτ௨
BASEͷΠϯϑϥߏʹ͍ͭͯ αʔόͷछྨ B/GσϓϩΠ͢Δαʔό σϓϩΠ͢Δͨͼʹ Πϯελϯε࡞ ݹ͍αʔόফ͍͑ͯ͘ ΞϓϦέʔγϣϯαʔό ʢApache+PHPʣ
BASEͷΠϯϑϥߏʹ͍ͭͯ αʔόͷछྨ B/GσϓϩΠ͠ͳ͍αʔό ͣͬͱ͋Δ όοναʔόͳͲ
2. Mackerelར༻্ͷ
Mackerelར༻্ͷ /health endpoint pattern OKͩͬͨΒεςʔλε200Λฦ͚ͩ͢ͷURL ֎ܗࢹͷର /health ͱ /health/deep /health
αʔό/ίϯςφͷੜଘ /health/deep DBͷଓ·Ͱ֬ೝ ͪ͜ΒΛ֎ܗࢹͰ༻ See also: ʮೖ ࢹ ―ϞμϯͳϞχλϦϯάͷͨΊͷσβΠϯύλʔϯʯ ɹɹɹɹɹ(https://www.oreilly.co.jp/books/9784873118642/) ɹɹɹɹɹஶ: Mike Julian ɹɹɹɹɹ7.3 health ΤϯυϙΠϯτύλʔϯ
Mackerelར༻্ͷ /health endpoint pattern ֎ܗࢹ͕Τϥʔʹͳͬͨͱ͖ͷ௨
Mackerelར༻্ͷ monitoringαʔό MackerelͷΧελϜϝτϦοΫαʔϏεϝτϦοΫΛऔಘɺ ߘ͢ΔͨΊͷαʔό AWSΠϯςάϨʔγϣϯະରԠͷϚωʔδυαʔϏεͷϝτϦο ΫΛऔಘ ରɿSES ҎલCloudFrontର͕ͩͬͨɺAWSΠϯςάϨʔγϣϯ͕ ରԠͨ͠ͷͰෆཁʹ
Mackerelར༻্ͷ monitoringαʔό AWSΠϯςάϨʔγϣϯ+ϓϥάΠϯ ର: RDS AWSΠϯςάϨʔγϣϯͰϝτϦοΫΛऔಘ monitoringαʔόͰmackerel-plugin-mysqlͰϝτϦοΫΛ औಘ custom_identifier Λࢦఆͯ͠ಉ͡ϗετʹू
Mackerelར༻্ͷ monitoringαʔό [plugin.metrics.mysql-prd-main] command = "mackerel-plugin-mysql -host=db1.xxx.ap-northeast-1.rds.amazonaws.com" custom_identifier = "db1.xxx.ap-northeast-1.rds.amazonaws.com"
See also: https://mackerel.io/ja/docs/entry/integrations/aws mackerel-agent.confʢൈਮʣ
Mackerelར༻্ͷ ࣜʹΑΔࢹ scale(divide( host(xxxxxxxxxxx, custom.ses.stats.Bounces), host(xxxxxxxxxxx, custom.ses.stats.DeliveryAttempts)), 100) mackerel-plugin-sesͰऔಘͰ͖Δͷ৴ͱόϯεͦΕͧΕͷ ݅ͳͷͰɺࣜͰܭࢉͨ͠Λࢹ
όϯε݅ / ৴݅ * 100 See Also: https://mackerel.io/ja/docs/entry/expression-monitoring SESͷBounce Rate (%)
Mackerelར༻্ͷ ࣜʹΑΔࢹ scale(divide( host(xxxxxxxxxxx, custom.ses.stats.Bounces), host(xxxxxxxxxxx, custom.ses.stats.DeliveryAttempts)), 100) SESͷBounce Rate
(%)
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ mkr MackerelͷCLIπʔϧ mkr monitors ࢹϧʔϧΛૢ࡞͢ΔαϒίϚϯυ mkr monitors pull
monitors.json ͱͯ͠ࢹϧʔϧΛอଘ mkr monitors diff monitors.json ͱࢹϧʔϧͷࠩΛදࣔ mkr monitors push monitors.json Λࢹϧʔϧʹө
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ mkr MackerelͷCLIπʔϧ mkr monitors ࢹϧʔϧΛૢ࡞͢ΔαϒίϚϯυ mkr monitors pull
monitors.json ͱͯ͠ࢹϧʔϧΛอଘ mkr monitors diff monitors.json ͱࢹϧʔϧͷࠩΛදࣔ mkr monitors push monitors.json Λࢹϧʔϧʹө
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ mkr monitors pull → git commit → GitHubʹgit
push CircleCIͰσΠϦʔ࣮ߦ มߋཤྺΛ͑Δ ·͕ͪ͑ͯյͯͤ͠Δ όοΫΞοϓతͳ͍ํʹཹΊ͍ͯΔ mkr monitors pushΘͣɺWebUIͰͷมߋͷΈ WebUIศར
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code
|| git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code
|| git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code
|| git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ
Mackerelར༻্ͷ ͍߹Θͤ༰ΛSlackͰڞ༗ ʮϝʔϧઃఆʯͰɺ 4MBDLͷʮ&NBJMʯ"QQͷ ϝʔϧΞυϨεΛઃఆ ͍߹Θͤͨ༰͕ 4MBDLʹྲྀΕ͖ͯͯศར
• ίϯςφΤʔδΣϯτࢼͯ͠Έ·ͨ͠ • ECSͷઃఆϛεΛൃݟʂ • ϝϞϦ͕Γͯͳ͔ͬͨʂ
• ϩʔϧҟৗݕ ࢼͯ͠Έ·ͨ͠
• ෛՙࢼݧͷ࣌ʹͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 ɹʮEίϚʔεϓϥοτϑΥʔϜͷΛࢧ͑Δ ɹɹෛՙࢼݧͷऔΓΈʯ •
• ෛՙࢼݧͷ࣌ʹͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 • ApacheͷίωΫγϣϯ͕ுΓ͍ͯͨͷ͕Ϙτϧ ωοΫʹͳ͍ͬͯͨ • MackerelΛΈͨΒ͙͢ʹΘ͔ͬͨʂʢ୲ऀஊʣ
• ෛՙࢼݧͷ࣌ʹͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 • Apache͕ϘτϧωοΫʹͳ͍ͬͯͨ • MackerelΛΈͨΒ͙͢ʹΘ͔ͬͨʂʢ୲ऀஊʣ
ʢMackerelΛݟͳ͕ΒΘ͍Θ͍͍ͬͯΔʣ
3. Mackerelར༻্ͷࠔΓ͝ͱ
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̍ʣ ͠ΐͬͪΎ͏ग़ͬͺͳ͠ͷΞϥʔτ र͏͖Ξϥʔτ͕ຒΕΔ ͱ͍͑ࢹΛશ͘Δͷɻɻɻ ௨νϟϯωϧΛ͚ΔϫʔΫΞϥϯυ ຊɺͦΜͳΞϥʔτແ͍Α͏ʹ͢Δ͖͕ͩ…
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̍ʣ ॏཁʹԠͯ͡௨νϟϯωϧΛ͚Δ #base_infra_alert ॏཁɿߴ #base_infra_alert_warn ॏཁɿத #base_infra_alert_info ॏཁɿ
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̎ʣ #base_infra_alert ʹMackerel͔Βͷ௨Ҏ֎ͷͷྲྀ͍ͯ͠Δ ϝʔϧͰདྷΔͷɺRSSϑΟʔυͳͲ ͜ΕΒʹ mention ͳ͍ͷͰɺSlackͰ͜ͷνϟϯωϧͷ௨ઃఆ Λڧͯ͘͠ؾ͚ͮΔΑ͏ʹ͢Δ͕…
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̎ʣ ௨͞Ε͗͢Δ #base_infra_alert ্ͷձ͕શ෦௨͞ΕΔʂ #base_infra_alert ʹ௨Λྲྀ͚ͩ͢ʹ͢Δ ٞผͷνϟϯωϧͰ ʮReacji ChannelerʯͰٞ༻ͷνϟϯωϧʹ
ϦϯΫΛίϐʔ https://get.slack.help/hc/ja/articles/360000482666-ଞνϟϯωϧϝοηʔδΛαΫοͱίϐʔ
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̎ʣ Ξϥʔτ௨ʹֆจࣈͰԠ͢Δͱ…
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̎ʣ ٞ༻ͷνϟϯωϧʹϦϯΫ͕ίϐʔ͞ΕΔʂ ௨͕ݮͬͯগ͠ฏʹ…
Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ αʔϏεͷͱͱʹαʔόͷछྨ͕૿͑ͨ αϒγεςϜͷ૿Ճ ϩʔϧͷԽ ϚωʔδυαʔϏεΛMackerelͷࢹରʹՃ ϩʔϧ/ϗετͷߏʹ͖Ε͍ʹରԠ͚ʹ͍͘ Lambda, SQSͳͲ
Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ MackerelͰʮϩʔϧʯ͕ࢹΛ͢Δ୯ҐͰ͋Γɺά ϥϑΛ·ͱΊΔ୯ҐͰ͋Δ ࢹ͍ͨ͠୯ҐͱάϥϑͰݟ͍ͨ୯Ґඞͣ͠Ұக ͠ͳ͍ ࢹ͍ͨ͠୯ҐͰϩʔϧΛ͚͍ͯ͘ͱɺ ʮαʔϏεʯͷը໘͕ͪ͝Όͪ͝Ό͖ͯͨ͠
Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ
Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ ࠓϩʔϧ࠶ฤத… Ұ؏ੑͷ͋Δϧʔϧʹج͍ͮͯઃܭ͢͠ ̎ͭͷʮαʔϏεʯΛͬͯϩʔϧΛཧ͢Δ ϩʔϧάϥϑΛҰཡͰΈ͍ͨϩʔϧ → ϝΠϯͷʮαʔϏεʯ ྫɿdb-primary, db-replica
ͦΕҎ֎ͷࢹ୯Ґͱͯ͠ͷϩʔϧ → αϒͷʮαʔϏεʯ ྫɿrds
Mackerelར༻্ͷࠔΓ͝ͱ ΧελϜϝτϦοΫϩʔϧʹͬͯ΄͍͠ αʔό͕ফ͑ΔͱΧελϜϝτϦοΫ͕ফ͑Δ σϓϩΠ͢Δͨͼʹαʔό͕ফ͑ΔͷͰΘΓͱਏ͍ γεςϜϝτϦοΫ͍ͬͯΔ͕… ΧελϜϝτϦοΫϩʔϧʹͬͯ΄͍͠ʂ ͳʹͱͧ…
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ աڈͷάϥϑͷϐʔΫ͕Θ͔Γʹ͍͘ ྫ͑ɺաڈͰҰ൪ϦΫΤετ͕དྷͯͨͱ͖ͷ ؒϦΫΤετͲͷ͘Β͍͔ͩͬͨͳʁͱ͍͏࣌ ͱΓ͋͑ͣʮ1YʯͷάϥϑΛݟͯΈΔ…ʁ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ࣌ؒൣғΛ͘औΔͱϐʔΫ͕ͳ·Δ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ࣌ؒൣғΛ͘औΔͱϐʔΫ͕ͳ·Δ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ࣌ؒൣғΛ͘औΔͱϐʔΫ͕ͳ·Δ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ࣌ؒൣғΛ͘औΔͱϐʔΫ͕ͳ·Δ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ʮ1YʯͷάϥϑͰϐʔΫ͕ݟ͑ͯ΄͍͠ ͳʹͱͧ…
4. Mackerelͷ͍͍ͱ͜Ζ
Mackerelͷ͍͍ͱ͜Ζ ࣗͰӡ༻͠ͳ͍͍ͯ͘ ͲΜͲΜػೳڧԽ͞Ε͍ͯ͘ AWSΠϯςάϨʔγϣϯ͕ศར ϓϥάΠϯͷՃָ͕ https://github.com/mackerelio/plugin-registry https://github.com/mackerelio/mackerel-agent-plugins ࣗ࡞؆୯ ʮαϙʔτνʔϜ࿈བྷʯͷϨεϙϯε͕Α͍
Mackerelͷ͍͍ͱ͜Ζ WebUI͕Θ͔Γ͍͢ άϥϑ͕͖Ε͍ API͕͋ͬͯϓϩάϥϜ͔Βૢ࡞Ͱ͖Δ CLI͕͋Δ ͍͡Ί͍͢ ΤʔδΣϯτΛΠϯετʔϧͯ͠ϩʔϧΛઃఆ͢Δ͚ͩͰɺ ͍͍ͩͨྑ͍ײ͡ͷάϥϑ͕ग़ͯ͘Δ ຊޠͩ UIɺυΩϡϝϯτ
ࢀߟਤॻ ʮMackerel αʔόࢹ [࣮ફ] ೖʯ ஶ: Ҫ্େีɼപ୩େีɼਿࢁ௨ɼాத৻࢘ɼ௶༎थɼদխ (http://gihyo.jp/book/2017/978-4-7741-9213-0) ʮೖ ࢹ
―ϞμϯͳϞχλϦϯάͷͨΊͷσβΠϯύλʔϯʯ ஶ: Mike Julian (https://www.oreilly.co.jp/books/9784873118642/) ʮιϑτΣΞΤϯδχΞͷͨΊͷɹITΠϯϑϥࢹʦ࣮ફʧೖʯ ஶ: ࡈ౻༞Ұ (https://gihyo.jp/book/2016/978-4-7741-7865-3)
We are hiring! SREͪΖΜɺ༷ʑͳ৬छΛืू͍ͯ͠·͢ ΧδϡΞϧʹձࣾʹ༡ͼʹདྷ͍ͨํ·Ͱ https://jobs.binc.jp/
ςοΫϒϩάߋ৽ͯ͠·͢ https://devblog.thebase.in/
ΦϯυϝσΟΞߋ৽ͯ͠·͢ https://basebook.binc.jp/
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠