Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
BASEにおけるMackerel利用上の工夫と困りごとのご紹介 / Mackerel Meetup #13 Tokyo
Search
Fumitaka Yamane
March 01, 2019
Technology
0
14k
BASEにおけるMackerel利用上の工夫と困りごとのご紹介 / Mackerel Meetup #13 Tokyo
2019/03/01 Mackerel Meetup #13 Tokyo
Fumitaka Yamane
March 01, 2019
Tweet
Share
Other Decks in Technology
See All in Technology
関数型DDDの理論と実践:「決定を遅らせる」を先につくり、 ビジネスの機動力と価値をあげる
knih
2
480
ハイパフォーマンスな組織をつくるための開発生産性の考え方 / developer-productivity-high-performer-link-and-motivation
lmi
3
260
これまでのキャリアとこれからMLエンジニアとしてどう動くか
masatakashiwagi
0
470
統計的学習理論読み Chapter 1
kmatsui
3
890
統計的学習理論読み Chapter 2
kmatsui
1
240
Cloud Friendly(?) Jenkins. How we failed to make Jenkins cloud native and what we learned?
onenashev
PRO
0
110
エンジニアブランディングチームの KPI / KPI's of engineer branding team
chaspy
1
140
10分でわかるfreeeのQA
freee
0
250
サービスメッシュ環境における OpenTelemetry 活用 / OpenTelemetry in Service Mesh
k6s4i53rx
2
850
大規模なアジャイル開発の現場と技術負債 / Technical Debt
yoshiitaka
20
4.1k
LLM + RAG を使った SORACOM Support Bot の裏側の歴史
soracom
PRO
1
640
やっていきテスト
k6s4i53rx
0
160
Featured
See All Featured
Raft: Consensus for Rubyists
vanstee
130
6.2k
VelocityConf: Rendering Performance Case Studies
addyosmani
319
23k
The Art of Programming - Codeland 2020
erikaheidi
40
12k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
39
4.3k
4 Signs Your Business is Dying
shpigford
174
21k
We Have a Design System, Now What?
morganepeng
42
6.7k
Clear Off the Table
cherdarchuk
82
310k
Pencils Down: Stop Designing & Start Developing
hursman
115
11k
Rails Girls Zürich Keynote
gr2m
91
13k
Building an army of robots
kneath
300
41k
How GitHub (no longer) Works
holman
301
140k
GraphQLの誤解/rethinking-graphql
sonatard
48
9.1k
Transcript
BASEʹ͓͚ΔMackerelར༻্ͷͱࠔΓ͝ͱ ͷ͝հ #"4&גࣜձࣾ43& ࢁ࢙ࠜਹ .BDLFSFM.FFUVQ5PLZP
ຊ… BASEͰMackerelΛར༻͍ͯ͠ΔதͰͷɺͪΐͬͱͨ͠ ɺࠔͬͨ͜ͱͱͦΕʹର͢ΔϫʔΫΞϥϯυɺ MackerelͷཁͳͲΛհ͠·͢ ࢀߟʹͳΕخ͍͠Ͱ͢
ࣗݾհ ࢁ࢙ࠜਹʢ·Ͷ;Έ͔ͨʣ BASEגࣜձࣾɹSREάϧʔϓ 2016ೖࣾ ೖࣾ࣌ʹMackerel͕ಋೖࡁΈͩͬͨ લ৬ʢιʔγϟϧήʔϜʣͰMackerelͬͯͨ twitter / ͯͳID: @fumikony
BASEͷϓϩμΫτʹ͍ͭͯ ωοτγϣοϓ࡞αʔϏε ʮBASEʯ γϣοϐϯάΞϓϦ ʮBASEʯ ՁͷަΛΑΓγϯϓϧʹ͠ɺ ੈքதͷਓʑ͕࠷దͳܦࡁ׆ಈΛߦ͑ΔΑ͏ʹ͢Δɻ MISSION
BASEͷϓϩμΫτʹ͍ͭͯ ωοτγϣοϓ࡞αʔϏε ʮBASEʯ γϣοϐϯάΞϓϦ ʮBASEʯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ
BASEͷϓϩμΫτʹ͍ͭͯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ αʔϏεͷن֦େʹͱͳͬͯোͷӨڹେ͖͘
BASEͷϓϩμΫτʹ͍ͭͯ 70ສγϣοϓ 500ສͷΞϓϦϢʔβʔ ҟৗ͕͋ͬͨΒ͙͢ʹؾ͖͍ͮͨ MackerelͰࢹ
࣍ 1. BASEͷΠϯϑϥߏʹ͍ͭͯ 2. Mackerelར༻্ͷ 3. Mackerelར༻্ͷࠔΓ͝ͱ 4. Mackerelͷ͍͍ͱ͜Ζ
1. BASEͷΠϯϑϥߏʹ͍ͭͯ
BASEͷΠϯϑϥߏʹ͍ͭͯ ֓ཁ ͍͍ͩͨAWS EC2, ECS, ELB, Lambda, ElastiCache, RDS, SQS,
CloudFront, S3, SES, etc… MackerelͰࢹ SlackʹΞϥʔτ௨
BASEͷΠϯϑϥߏʹ͍ͭͯ αʔόͷछྨ B/GσϓϩΠ͢Δαʔό σϓϩΠ͢Δͨͼʹ Πϯελϯε࡞ ݹ͍αʔόফ͍͑ͯ͘ ΞϓϦέʔγϣϯαʔό ʢApache+PHPʣ
BASEͷΠϯϑϥߏʹ͍ͭͯ αʔόͷछྨ B/GσϓϩΠ͠ͳ͍αʔό ͣͬͱ͋Δ όοναʔόͳͲ
2. Mackerelར༻্ͷ
Mackerelར༻্ͷ /health endpoint pattern OKͩͬͨΒεςʔλε200Λฦ͚ͩ͢ͷURL ֎ܗࢹͷର /health ͱ /health/deep /health
αʔό/ίϯςφͷੜଘ /health/deep DBͷଓ·Ͱ֬ೝ ͪ͜ΒΛ֎ܗࢹͰ༻ See also: ʮೖ ࢹ ―ϞμϯͳϞχλϦϯάͷͨΊͷσβΠϯύλʔϯʯ ɹɹɹɹɹ(https://www.oreilly.co.jp/books/9784873118642/) ɹɹɹɹɹஶ: Mike Julian ɹɹɹɹɹ7.3 health ΤϯυϙΠϯτύλʔϯ
Mackerelར༻্ͷ /health endpoint pattern ֎ܗࢹ͕Τϥʔʹͳͬͨͱ͖ͷ௨
Mackerelར༻্ͷ monitoringαʔό MackerelͷΧελϜϝτϦοΫαʔϏεϝτϦοΫΛऔಘɺ ߘ͢ΔͨΊͷαʔό AWSΠϯςάϨʔγϣϯະରԠͷϚωʔδυαʔϏεͷϝτϦο ΫΛऔಘ ରɿSES ҎલCloudFrontର͕ͩͬͨɺAWSΠϯςάϨʔγϣϯ͕ ରԠͨ͠ͷͰෆཁʹ
Mackerelར༻্ͷ monitoringαʔό AWSΠϯςάϨʔγϣϯ+ϓϥάΠϯ ର: RDS AWSΠϯςάϨʔγϣϯͰϝτϦοΫΛऔಘ monitoringαʔόͰmackerel-plugin-mysqlͰϝτϦοΫΛ औಘ custom_identifier Λࢦఆͯ͠ಉ͡ϗετʹू
Mackerelར༻্ͷ monitoringαʔό [plugin.metrics.mysql-prd-main] command = "mackerel-plugin-mysql -host=db1.xxx.ap-northeast-1.rds.amazonaws.com" custom_identifier = "db1.xxx.ap-northeast-1.rds.amazonaws.com"
See also: https://mackerel.io/ja/docs/entry/integrations/aws mackerel-agent.confʢൈਮʣ
Mackerelར༻্ͷ ࣜʹΑΔࢹ scale(divide( host(xxxxxxxxxxx, custom.ses.stats.Bounces), host(xxxxxxxxxxx, custom.ses.stats.DeliveryAttempts)), 100) mackerel-plugin-sesͰऔಘͰ͖Δͷ৴ͱόϯεͦΕͧΕͷ ݅ͳͷͰɺࣜͰܭࢉͨ͠Λࢹ
όϯε݅ / ৴݅ * 100 See Also: https://mackerel.io/ja/docs/entry/expression-monitoring SESͷBounce Rate (%)
Mackerelར༻্ͷ ࣜʹΑΔࢹ scale(divide( host(xxxxxxxxxxx, custom.ses.stats.Bounces), host(xxxxxxxxxxx, custom.ses.stats.DeliveryAttempts)), 100) SESͷBounce Rate
(%)
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ mkr MackerelͷCLIπʔϧ mkr monitors ࢹϧʔϧΛૢ࡞͢ΔαϒίϚϯυ mkr monitors pull
monitors.json ͱͯ͠ࢹϧʔϧΛอଘ mkr monitors diff monitors.json ͱࢹϧʔϧͷࠩΛදࣔ mkr monitors push monitors.json Λࢹϧʔϧʹө
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ mkr MackerelͷCLIπʔϧ mkr monitors ࢹϧʔϧΛૢ࡞͢ΔαϒίϚϯυ mkr monitors pull
monitors.json ͱͯ͠ࢹϧʔϧΛอଘ mkr monitors diff monitors.json ͱࢹϧʔϧͷࠩΛදࣔ mkr monitors push monitors.json Λࢹϧʔϧʹө
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ mkr monitors pull → git commit → GitHubʹgit
push CircleCIͰσΠϦʔ࣮ߦ มߋཤྺΛ͑Δ ·͕ͪ͑ͯյͯͤ͠Δ όοΫΞοϓతͳ͍ํʹཹΊ͍ͯΔ mkr monitors pushΘͣɺWebUIͰͷมߋͷΈ WebUIศར
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code
|| git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code
|| git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ
Mackerelར༻্ͷ ࢹϧʔϧͷίʔυԽ git pull mkr monitors pull git diff --exit-code
|| git commit -am '[ci skip]' git push origin master CircleCIͰσΠϦʔ࣮ߦ
Mackerelར༻্ͷ ͍߹Θͤ༰ΛSlackͰڞ༗ ʮϝʔϧઃఆʯͰɺ 4MBDLͷʮ&NBJMʯ"QQͷ ϝʔϧΞυϨεΛઃఆ ͍߹Θͤͨ༰͕ 4MBDLʹྲྀΕ͖ͯͯศར
• ίϯςφΤʔδΣϯτࢼͯ͠Έ·ͨ͠ • ECSͷઃఆϛεΛൃݟʂ • ϝϞϦ͕Γͯͳ͔ͬͨʂ
• ϩʔϧҟৗݕ ࢼͯ͠Έ·ͨ͠
• ෛՙࢼݧͷ࣌ʹͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 ɹʮEίϚʔεϓϥοτϑΥʔϜͷΛࢧ͑Δ ɹɹෛՙࢼݧͷऔΓΈʯ •
• ෛՙࢼݧͷ࣌ʹͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 • ApacheͷίωΫγϣϯ͕ுΓ͍ͯͨͷ͕Ϙτϧ ωοΫʹͳ͍ͬͯͨ • MackerelΛΈͨΒ͙͢ʹΘ͔ͬͨʂʢ୲ऀஊʣ
• ෛՙࢼݧͷ࣌ʹͬͯΈ·ͨ͠ • https://devblog.thebase.in/entry/2019/01/23/110000 • Apache͕ϘτϧωοΫʹͳ͍ͬͯͨ • MackerelΛΈͨΒ͙͢ʹΘ͔ͬͨʂʢ୲ऀஊʣ
ʢMackerelΛݟͳ͕ΒΘ͍Θ͍͍ͬͯΔʣ
3. Mackerelར༻্ͷࠔΓ͝ͱ
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̍ʣ ͠ΐͬͪΎ͏ग़ͬͺͳ͠ͷΞϥʔτ र͏͖Ξϥʔτ͕ຒΕΔ ͱ͍͑ࢹΛશ͘Δͷɻɻɻ ௨νϟϯωϧΛ͚ΔϫʔΫΞϥϯυ ຊɺͦΜͳΞϥʔτແ͍Α͏ʹ͢Δ͖͕ͩ…
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̍ʣ ॏཁʹԠͯ͡௨νϟϯωϧΛ͚Δ #base_infra_alert ॏཁɿߴ #base_infra_alert_warn ॏཁɿத #base_infra_alert_info ॏཁɿ
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̎ʣ #base_infra_alert ʹMackerel͔Βͷ௨Ҏ֎ͷͷྲྀ͍ͯ͠Δ ϝʔϧͰདྷΔͷɺRSSϑΟʔυͳͲ ͜ΕΒʹ mention ͳ͍ͷͰɺSlackͰ͜ͷνϟϯωϧͷ௨ઃఆ Λڧͯ͘͠ؾ͚ͮΔΑ͏ʹ͢Δ͕…
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̎ʣ ௨͞Ε͗͢Δ #base_infra_alert ্ͷձ͕શ෦௨͞ΕΔʂ #base_infra_alert ʹ௨Λྲྀ͚ͩ͢ʹ͢Δ ٞผͷνϟϯωϧͰ ʮReacji ChannelerʯͰٞ༻ͷνϟϯωϧʹ
ϦϯΫΛίϐʔ https://get.slack.help/hc/ja/articles/360000482666-ଞνϟϯωϧϝοηʔδΛαΫοͱίϐʔ
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̎ʣ Ξϥʔτ௨ʹֆจࣈͰԠ͢Δͱ…
Mackerelར༻্ͷࠔΓ͝ͱ ΦΦΧϛগʢ̎ʣ ٞ༻ͷνϟϯωϧʹϦϯΫ͕ίϐʔ͞ΕΔʂ ௨͕ݮͬͯগ͠ฏʹ…
Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ αʔϏεͷͱͱʹαʔόͷछྨ͕૿͑ͨ αϒγεςϜͷ૿Ճ ϩʔϧͷԽ ϚωʔδυαʔϏεΛMackerelͷࢹରʹՃ ϩʔϧ/ϗετͷߏʹ͖Ε͍ʹରԠ͚ʹ͍͘ Lambda, SQSͳͲ
Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ MackerelͰʮϩʔϧʯ͕ࢹΛ͢Δ୯ҐͰ͋Γɺά ϥϑΛ·ͱΊΔ୯ҐͰ͋Δ ࢹ͍ͨ͠୯ҐͱάϥϑͰݟ͍ͨ୯Ґඞͣ͠Ұக ͠ͳ͍ ࢹ͍ͨ͠୯ҐͰϩʔϧΛ͚͍ͯ͘ͱɺ ʮαʔϏεʯͷը໘͕ͪ͝Όͪ͝Ό͖ͯͨ͠
Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ
Mackerelར༻্ͷࠔΓ͝ͱ ϩʔϧ͕ࢄΒ͔ͬͯ͠·ͬͨ ࠓϩʔϧ࠶ฤத… Ұ؏ੑͷ͋Δϧʔϧʹج͍ͮͯઃܭ͢͠ ̎ͭͷʮαʔϏεʯΛͬͯϩʔϧΛཧ͢Δ ϩʔϧάϥϑΛҰཡͰΈ͍ͨϩʔϧ → ϝΠϯͷʮαʔϏεʯ ྫɿdb-primary, db-replica
ͦΕҎ֎ͷࢹ୯Ґͱͯ͠ͷϩʔϧ → αϒͷʮαʔϏεʯ ྫɿrds
Mackerelར༻্ͷࠔΓ͝ͱ ΧελϜϝτϦοΫϩʔϧʹͬͯ΄͍͠ αʔό͕ফ͑ΔͱΧελϜϝτϦοΫ͕ফ͑Δ σϓϩΠ͢Δͨͼʹαʔό͕ফ͑ΔͷͰΘΓͱਏ͍ γεςϜϝτϦοΫ͍ͬͯΔ͕… ΧελϜϝτϦοΫϩʔϧʹͬͯ΄͍͠ʂ ͳʹͱͧ…
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ աڈͷάϥϑͷϐʔΫ͕Θ͔Γʹ͍͘ ྫ͑ɺաڈͰҰ൪ϦΫΤετ͕དྷͯͨͱ͖ͷ ؒϦΫΤετͲͷ͘Β͍͔ͩͬͨͳʁͱ͍͏࣌ ͱΓ͋͑ͣʮ1YʯͷάϥϑΛݟͯΈΔ…ʁ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ࣌ؒൣғΛ͘औΔͱϐʔΫ͕ͳ·Δ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ࣌ؒൣғΛ͘औΔͱϐʔΫ͕ͳ·Δ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ࣌ؒൣғΛ͘औΔͱϐʔΫ͕ͳ·Δ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ࣌ؒൣғΛ͘औΔͱϐʔΫ͕ͳ·Δ
Mackerelར༻্ͷࠔΓ͝ͱ άϥϑͷϐʔΫΛΘ͔Γͯ͘͢͠΄͍͠ ʮ1YʯͷάϥϑͰϐʔΫ͕ݟ͑ͯ΄͍͠ ͳʹͱͧ…
4. Mackerelͷ͍͍ͱ͜Ζ
Mackerelͷ͍͍ͱ͜Ζ ࣗͰӡ༻͠ͳ͍͍ͯ͘ ͲΜͲΜػೳڧԽ͞Ε͍ͯ͘ AWSΠϯςάϨʔγϣϯ͕ศར ϓϥάΠϯͷՃָ͕ https://github.com/mackerelio/plugin-registry https://github.com/mackerelio/mackerel-agent-plugins ࣗ࡞؆୯ ʮαϙʔτνʔϜ࿈བྷʯͷϨεϙϯε͕Α͍
Mackerelͷ͍͍ͱ͜Ζ WebUI͕Θ͔Γ͍͢ άϥϑ͕͖Ε͍ API͕͋ͬͯϓϩάϥϜ͔Βૢ࡞Ͱ͖Δ CLI͕͋Δ ͍͡Ί͍͢ ΤʔδΣϯτΛΠϯετʔϧͯ͠ϩʔϧΛઃఆ͢Δ͚ͩͰɺ ͍͍ͩͨྑ͍ײ͡ͷάϥϑ͕ग़ͯ͘Δ ຊޠͩ UIɺυΩϡϝϯτ
ࢀߟਤॻ ʮMackerel αʔόࢹ [࣮ફ] ೖʯ ஶ: Ҫ্େีɼപ୩େีɼਿࢁ௨ɼాத৻࢘ɼ௶༎थɼদխ (http://gihyo.jp/book/2017/978-4-7741-9213-0) ʮೖ ࢹ
―ϞμϯͳϞχλϦϯάͷͨΊͷσβΠϯύλʔϯʯ ஶ: Mike Julian (https://www.oreilly.co.jp/books/9784873118642/) ʮιϑτΣΞΤϯδχΞͷͨΊͷɹITΠϯϑϥࢹʦ࣮ફʧೖʯ ஶ: ࡈ౻༞Ұ (https://gihyo.jp/book/2016/978-4-7741-7865-3)
We are hiring! SREͪΖΜɺ༷ʑͳ৬छΛืू͍ͯ͠·͢ ΧδϡΞϧʹձࣾʹ༡ͼʹདྷ͍ͨํ·Ͱ https://jobs.binc.jp/
ςοΫϒϩάߋ৽ͯ͠·͢ https://devblog.thebase.in/
ΦϯυϝσΟΞߋ৽ͯ͠·͢ https://basebook.binc.jp/
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠