Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
BASEにおけるMackerel利用上の工夫と困りごとのご紹介 / Mackerel Meetup #13 Tokyo
Search
Fumitaka Yamane
March 01, 2019
Technology
0
15k
BASEにおけるMackerel利用上の工夫と困りごとのご紹介 / Mackerel Meetup #13 Tokyo
2019/03/01 Mackerel Meetup #13 Tokyo
Fumitaka Yamane
March 01, 2019
Tweet
Share
Other Decks in Technology
See All in Technology
データ分析を支える技術 生成AI再入門
ishikawa_satoru
0
380
[2024最新版]AWS Control Towerを使ったセキュアなマルチアカウント環境の作り方
hiashisan
0
270
コンテナ・K8s研修 - 前半 コンテナ基礎・ハンズオン【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
170
CEL(Common Expression Language)で書いた条件にマッチしたIAM Policyを見つける / iam-policy-finder
fujiwara3
0
720
Github Actions 로 Android 팀의 효율성 극대화
hadonghyun
0
160
データベース研修 DB基礎【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
220
Matterport を使ってクラスメソッド各拠点のバーチャルオフィスツアーを作成してみた
wakatsuki
0
160
AWSサービスメニュー開発をしていてAWSを好きだ!と感じた瞬間
toru_kubota
0
130
[NIKKEI Tech Talk]Bias for Action!! 実践から学ぶための仕組とコミュニティ / Community for Practice and Learning
kanamasa
0
280
「単なる OAuth 2.0 を認証に使うと、車が通れるほどのどでかいセキュリティー・ホールができる」のか検証してみた
terara
0
380
JBUG岡山 #6 WordCamp男木島の チームビルディング
takeshifurusato
0
150
クラウド利用者の「責任」をどう果たす?AWSセキュリティ対策のススメ #AWSSummit
hiashisan
0
280
Featured
See All Featured
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
121
18k
Navigating Team Friction
lara
181
13k
Building Adaptive Systems
keathley
34
2k
GraphQLの誤解/rethinking-graphql
sonatard
59
9.6k
4 Signs Your Business is Dying
shpigford
178
21k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
248
20k
Into the Great Unknown - MozCon
thekraken
20
1.3k
Building Applications with DynamoDB
mza
89
5.8k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
20
7.2k
The World Runs on Bad Software
bkeepers
PRO
63
11k
How to train your dragon (web standard)
notwaldorf
79
5.5k
The Pragmatic Product Professional
lauravandoore
29
6.1k
Transcript
BASEʹ͓͚ΔMackerelར༻্ͷͱࠔΓ͝ͱ ͷ͝հ #"4&גࣜձࣾ43& ࢁ࢙ࠜਹ .BDLFSFM.FFUVQ5PLZP
ຊ… BASEͰMackerelΛར༻͍ͯ͠ΔதͰͷɺͪΐͬͱͨ͠ ɺࠔͬͨ͜ͱͱͦΕʹର͢ΔϫʔΫΞϥϯυɺ MackerelͷཁͳͲΛհ͠·͢ ࢀߟʹͳΕخ͍͠Ͱ͢
ࣗݾհ ࢁ࢙ࠜਹʢ·Ͷ;Έ͔ͨʣ BASEגࣜձࣾɹSREάϧʔϓ 2016ೖࣾ ೖࣾ࣌ʹMackerel͕ಋೖࡁΈͩͬͨ લ৬ʢιʔγϟϧήʔϜʣͰMackerelͬͯͨ twitter / ͯͳID: @fumikony
BASEͷϓϩμΫτʹ͍ͭͯ ωοτγϣοϓ࡞αʔϏε ʮBASEʯ γϣοϐϯάΞϓϦ ʮBASEʯ ՁͷަΛΑΓγϯϓϧʹ͠ɺ ੈքதͷਓʑ͕࠷దͳܦࡁ׆ಈΛߦ͑ΔΑ͏ʹ͢Δɻ MISSION
BASEͷϓϩμΫτʹ͍ͭͯ ωοτγϣοϓ࡞αʔϏε ʮBASEʯ γϣοϐϯάΞϓϦ ʮBASEʯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ
BASEͷϓϩμΫτʹ͍ͭͯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ αʔϏεͷن֦େʹͱͳͬͯোͷӨڹେ͖͘
BASEͷϓϩμΫτʹ͍ͭͯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ ҟৗ͕͋ͬͨΒ͙͢ʹؾ͖͍ͮͨ MackerelͰࢹ
࣍ 1. BASEͷΠϯϑϥߏʹ͍ͭͯ 2. Mackerelར༻্ͷ 3. Mackerelར༻্ͷࠔΓ͝ͱ 4. Mackerelͷ͍͍ͱ͜Ζ
1. BASEͷΠϯϑϥߏʹ͍ͭͯ
BASEͷΠϯϑϥߏʹ͍ͭͯ ֓ཁ ͍͍ͩͨAWS EC2, ECS, ELB, Lambda, ElastiCache, RDS, SQS,
CloudFront, S3, SES, etc… MackerelͰࢹ SlackʹΞϥʔτ௨
BASEͷΠϯϑϥߏʹ͍ͭͯ αʔόͷछྨ B/GσϓϩΠ͢Δαʔό σϓϩΠ͢Δͨͼʹ Πϯελϯε࡞ ݹ͍αʔόফ͍͑ͯ͘ ΞϓϦέʔγϣϯαʔό ʢApache+PHPʣ
BASEͷΠϯϑϥߏʹ͍ͭͯ αʔόͷछྨ B/GσϓϩΠ͠ͳ͍αʔό ͣͬͱ͋Δ όοναʔόͳͲ
2. Mackerelར༻্ͷ
Mackerelར༻্ͷ /health endpoint pattern OKͩͬͨΒεςʔλε200Λฦ͚ͩ͢ͷURL ֎ܗࢹͷର /health ͱ /health/deep /health
αʔό/ίϯςφͷੜଘ /health/deep DBͷଓ·Ͱ֬ೝ ͪ͜ΒΛ֎ܗࢹͰ༻ See also: ʮೖ ࢹ ―ϞμϯͳϞχλϦϯάͷͨΊͷσβΠϯύλʔϯʯ ɹɹɹɹɹ(https://www.oreilly.co.jp/books/9784873118642/) ɹɹɹɹɹஶ: Mike Julian ɹɹɹɹɹ7.3 health ΤϯυϙΠϯτύλʔϯ
Mackerelར༻্ͷ /health endpoint pattern ֎ܗࢹ͕Τϥʔʹͳͬͨͱ͖ͷ௨
Mackerelར༻্ͷ monitoringαʔό MackerelͷΧελϜϝτϦοΫαʔϏεϝτϦοΫΛऔಘɺ ߘ͢ΔͨΊͷαʔό AWSΠϯςάϨʔγϣϯະରԠͷϚωʔδυαʔϏεͷϝτϦο ΫΛऔಘ ରɿSES ҎલCloudFrontର͕ͩͬͨɺAWSΠϯςάϨʔγϣϯ͕ ରԠͨ͠ͷͰෆཁʹ
Mackerelར༻্ͷ monitoringαʔό AWSΠϯςάϨʔγϣϯ+ϓϥάΠϯ ର: RDS AWSΠϯςάϨʔγϣϯͰϝτϦοΫΛऔಘ monitoringαʔόͰmackerel-plugin-mysqlͰϝτϦοΫΛ औಘ custom_identifier Λࢦఆͯ͠ಉ͡ϗετʹू
Mackerelར༻্ͷ monitoringαʔό [plugin.metrics.mysql-prd-main] command = "mackerel-plugin-mysql -host=db1.xxx.ap-northeast-1.rds.amazonaws.com" custom_identifier = "db1.xxx.ap-northeast-1.rds.amazonaws.com"
See also: https://mackerel.io/ja/docs/entry/integrations/aws mackerel-agent.confʢൈਮʣ
Mackerelར༻্ͷ ࣜʹΑΔࢹ scale(divide( host(xxxxxxxxxxx, custom.ses.stats.Bounces), host(xxxxxxxxxxx, custom.ses.stats.DeliveryAttempts)), 100) mackerel-plugin-sesͰऔಘͰ͖Δͷ৴ͱόϯεͦΕͧΕͷ ݅ͳͷͰɺࣜͰܭࢉͨ͠Λࢹ
όϯε݅ / ৴݅ * 100 See Also: https://mackerel.io/ja/docs/entry/expression-monitoring SESͷBounce Rate (%)
Mackerelར༻্ͷ ࣜʹΑΔࢹ scale(divide( host(xxxxxxxxxxx, custom.ses.stats.Bounces), host(xxxxxxxxxxx, custom.ses.stats.DeliveryAttempts)), 100) SESͷBounce Rate
(%)
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ mkr MackerelͷCLIπʔϧ mkr monitors ࢹϧʔϧΛૢ࡞͢ΔαϒίϚϯυ mkr monitors pull
monitors.json ͱͯ͠ࢹϧʔϧΛอଘ mkr monitors diff monitors.json ͱࢹϧʔϧͷࠩΛදࣔ mkr monitors push monitors.json Λࢹϧʔϧʹө
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ mkr MackerelͷCLIπʔϧ mkr monitors ࢹϧʔϧΛૢ࡞͢ΔαϒίϚϯυ mkr monitors pull
monitors.json ͱͯ͠ࢹϧʔϧΛอଘ mkr monitors diff monitors.json ͱࢹϧʔϧͷࠩΛදࣔ mkr monitors push monitors.json Λࢹϧʔϧʹө
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ mkr monitors pull → git commit → GitHubʹgit
push CircleCIͰσΠϦʔ࣮ߦ มߋཤྺΛ͑Δ ·͕ͪ͑ͯյͯͤ͠Δ όοΫΞοϓతͳ͍ํʹཹΊ͍ͯΔ mkr monitors pushΘͣɺWebUIͰͷมߋͷΈ WebUIศར
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code
|| git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code
|| git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code
|| git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ
Mackerelར༻্ͷ ͍߹Θͤ༰ΛSlackͰڞ༗ ʮϝʔϧઃఆʯͰɺ 4MBDLͷʮ&NBJMʯ"QQͷ ϝʔϧΞυϨεΛઃఆ ͍߹Θͤͨ༰͕ 4MBDLʹྲྀΕ͖ͯͯศར
• ίϯςφΤʔδΣϯτࢼͯ͠Έ·ͨ͠ • ECSͷઃఆϛεΛൃݟʂ • ϝϞϦ͕Γͯͳ͔ͬͨʂ
• ϩʔϧҟৗݕ ࢼͯ͠Έ·ͨ͠
• ෛՙࢼݧͷ࣌ʹͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 ɹʮEίϚʔεϓϥοτϑΥʔϜͷΛࢧ͑Δ ɹɹෛՙࢼݧͷऔΓΈʯ •
• ෛՙࢼݧͷ࣌ʹͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 • ApacheͷίωΫγϣϯ͕ுΓ͍ͯͨͷ͕Ϙτϧ ωοΫʹͳ͍ͬͯͨ • MackerelΛΈͨΒ͙͢ʹΘ͔ͬͨʂʢ୲ऀஊʣ
• ෛՙࢼݧͷ࣌ʹͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 • Apache͕ϘτϧωοΫʹͳ͍ͬͯͨ • MackerelΛΈͨΒ͙͢ʹΘ͔ͬͨʂʢ୲ऀஊʣ
ʢMackerelΛݟͳ͕ΒΘ͍Θ͍͍ͬͯΔʣ
3. Mackerelར༻্ͷࠔΓ͝ͱ
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̍ʣ ͠ΐͬͪΎ͏ग़ͬͺͳ͠ͷΞϥʔτ र͏͖Ξϥʔτ͕ຒΕΔ ͱ͍͑ࢹΛશ͘Δͷɻɻɻ ௨νϟϯωϧΛ͚ΔϫʔΫΞϥϯυ ຊɺͦΜͳΞϥʔτແ͍Α͏ʹ͢Δ͖͕ͩ…
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̍ʣ ॏཁʹԠͯ͡௨νϟϯωϧΛ͚Δ #base_infra_alert ॏཁɿߴ #base_infra_alert_warn ॏཁɿத #base_infra_alert_info ॏཁɿ
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̎ʣ #base_infra_alert ʹMackerel͔Βͷ௨Ҏ֎ͷͷྲྀ͍ͯ͠Δ ϝʔϧͰདྷΔͷɺRSSϑΟʔυͳͲ ͜ΕΒʹ mention ͳ͍ͷͰɺSlackͰ͜ͷνϟϯωϧͷ௨ઃఆ Λڧͯ͘͠ؾ͚ͮΔΑ͏ʹ͢Δ͕…
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̎ʣ ௨͞Ε͗͢Δ #base_infra_alert ্ͷձ͕શ෦௨͞ΕΔʂ #base_infra_alert ʹ௨Λྲྀ͚ͩ͢ʹ͢Δ ٞผͷνϟϯωϧͰ ʮReacji ChannelerʯͰٞ༻ͷνϟϯωϧʹ
ϦϯΫΛίϐʔ https://get.slack.help/hc/ja/articles/360000482666-ଞνϟϯωϧϝοηʔδΛαΫοͱίϐʔ
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̎ʣ Ξϥʔτ௨ʹֆจࣈͰԠ͢Δͱ…
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̎ʣ ٞ༻ͷνϟϯωϧʹϦϯΫ͕ίϐʔ͞ΕΔʂ ௨͕ݮͬͯগ͠ฏʹ…
Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ αʔϏεͷͱͱʹαʔόͷछྨ͕૿͑ͨ αϒγεςϜͷ૿Ճ ϩʔϧͷԽ ϚωʔδυαʔϏεΛMackerelͷࢹରʹՃ ϩʔϧ/ϗετͷߏʹ͖Ε͍ʹରԠ͚ʹ͍͘ Lambda, SQSͳͲ
Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ MackerelͰʮϩʔϧʯ͕ࢹΛ͢Δ୯ҐͰ͋Γɺά ϥϑΛ·ͱΊΔ୯ҐͰ͋Δ ࢹ͍ͨ͠୯ҐͱάϥϑͰݟ͍ͨ୯Ґඞͣ͠Ұக ͠ͳ͍ ࢹ͍ͨ͠୯ҐͰϩʔϧΛ͚͍ͯ͘ͱɺ ʮαʔϏεʯͷը໘͕ͪ͝Όͪ͝Ό͖ͯͨ͠
Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ
Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ ࠓϩʔϧ࠶ฤத… Ұ؏ੑͷ͋Δϧʔϧʹج͍ͮͯઃܭ͢͠ ̎ͭͷʮαʔϏεʯΛͬͯϩʔϧΛཧ͢Δ ϩʔϧάϥϑΛҰཡͰΈ͍ͨϩʔϧ → ϝΠϯͷʮαʔϏεʯ ྫɿdb-primary, db-replica
ͦΕҎ֎ͷࢹ୯Ґͱͯ͠ͷϩʔϧ → αϒͷʮαʔϏεʯ ྫɿrds
Mackerelར༻্ͷࠔΓ͝ͱ ΧελϜϝτϦοΫϩʔϧʹͬͯ΄͍͠ αʔό͕ফ͑ΔͱΧελϜϝτϦοΫ͕ফ͑Δ σϓϩΠ͢Δͨͼʹαʔό͕ফ͑ΔͷͰΘΓͱਏ͍ γεςϜϝτϦοΫ͍ͬͯΔ͕… ΧελϜϝτϦοΫϩʔϧʹͬͯ΄͍͠ʂ ͳʹͱͧ…
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ աڈͷάϥϑͷϐʔΫ͕Θ͔Γʹ͍͘ ྫ͑ɺաڈͰҰ൪ϦΫΤετ͕དྷͯͨͱ͖ͷ ؒϦΫΤετͲͷ͘Β͍͔ͩͬͨͳʁͱ͍͏࣌ ͱΓ͋͑ͣʮ1YʯͷάϥϑΛݟͯΈΔ…ʁ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ࣌ؒൣғΛ͘औΔͱϐʔΫ͕ͳ·Δ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ࣌ؒൣғΛ͘औΔͱϐʔΫ͕ͳ·Δ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ࣌ؒൣғΛ͘औΔͱϐʔΫ͕ͳ·Δ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ࣌ؒൣғΛ͘औΔͱϐʔΫ͕ͳ·Δ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ʮ1YʯͷάϥϑͰϐʔΫ͕ݟ͑ͯ΄͍͠ ͳʹͱͧ…
4. Mackerelͷ͍͍ͱ͜Ζ
Mackerelͷ͍͍ͱ͜Ζ ࣗͰӡ༻͠ͳ͍͍ͯ͘ ͲΜͲΜػೳڧԽ͞Ε͍ͯ͘ AWSΠϯςάϨʔγϣϯ͕ศར ϓϥάΠϯͷՃָ͕ https://github.com/mackerelio/plugin-registry https://github.com/mackerelio/mackerel-agent-plugins ࣗ࡞؆୯ ʮαϙʔτνʔϜ࿈བྷʯͷϨεϙϯε͕Α͍
Mackerelͷ͍͍ͱ͜Ζ WebUI͕Θ͔Γ͍͢ άϥϑ͕͖Ε͍ API͕͋ͬͯϓϩάϥϜ͔Βૢ࡞Ͱ͖Δ CLI͕͋Δ ͍͡Ί͍͢ ΤʔδΣϯτΛΠϯετʔϧͯ͠ϩʔϧΛઃఆ͢Δ͚ͩͰɺ ͍͍ͩͨྑ͍ײ͡ͷάϥϑ͕ग़ͯ͘Δ ຊޠͩ UIɺυΩϡϝϯτ
ࢀߟਤॻ ʮMackerel αʔόࢹ [࣮ફ] ೖʯ ஶ: Ҫ্େีɼപ୩େีɼਿࢁ௨ɼాத৻࢘ɼ௶༎थɼদխ (http://gihyo.jp/book/2017/978-4-7741-9213-0) ʮೖ ࢹ
―ϞμϯͳϞχλϦϯάͷͨΊͷσβΠϯύλʔϯʯ ஶ: Mike Julian (https://www.oreilly.co.jp/books/9784873118642/) ʮιϑτΣΞΤϯδχΞͷͨΊͷɹITΠϯϑϥࢹʦ࣮ફʧೖʯ ஶ: ࡈ౻༞Ұ (https://gihyo.jp/book/2016/978-4-7741-7865-3)
We are hiring! SREͪΖΜɺ༷ʑͳ৬छΛืू͍ͯ͠·͢ ΧδϡΞϧʹձࣾʹ༡ͼʹདྷ͍ͨํ·Ͱ https://jobs.binc.jp/
ςοΫϒϩάߋ৽ͯ͠·͢ https://devblog.thebase.in/
ΦϯυϝσΟΞߋ৽ͯ͠·͢ https://basebook.binc.jp/
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠