Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
メルカリのシステム・サービス監視について/Monitoring Mercari service...
Search
kazeburo
November 29, 2017
Technology
5
7.1k
メルカリのシステム・サービス監視について/Monitoring Mercari service and servers
メルカリのシステム・サービス監視について
Monitoring seminar in Mercari
kazeburo
November 29, 2017
Tweet
Share
More Decks by kazeburo
See All by kazeburo
さくらのクラウド 開発の挑戦とその舞台裏
kazeburo
1
930
[SRE kaigi 2025] ガバメントクラウドに向けた開発と変化するSRE組織のあり方 / Development for Government Cloud and the Evolving Role of SRE Teams
kazeburo
4
3.1k
[さくらのTech Day] ガバメントクラウド開発と変化と成長する組織 / sakura techday, Develop govcloud and the team
kazeburo
0
6.8k
ガバメントクラウド開発と変化と成長する組織 / Organizational change and growth in developing a government cloud
kazeburo
4
3k
DNS水責め攻撃と監視 / DNS water torture attack Monitoring and SLO
kazeburo
5
4.3k
DBやめてみた / DNS water torture attack and countermeasures
kazeburo
13
14k
IaaSにおけるPlatform Engineeringとこれから / Platform engineering in IaaS
kazeburo
2
1.4k
高信頼IaaSを実現するDevOps / DevOps for Highly Reliable IaaS
kazeburo
1
700
権威DNSサービスへのDDoSと ハイパフォーマンスなベンチマーカ / DNS Pseudo random subdomain attack and High performance Benchmarker
kazeburo
3
5.4k
Other Decks in Technology
See All in Technology
AIの全社活用を推進するための安全なレールを敷いた話
shoheimitani
2
450
生成AI時代 文字コードを学ぶ意義を見出せるか?
hrsued
1
810
自律的なスケーリング手法FASTにおけるVPoEとしてのアカウンタビリティ / dev-productivity-con-2025
yoshikiiida
1
15k
American airlines ®️ USA Contact Numbers: Complete 2025 Support Guide
airhelpsupport
0
350
論文紹介:LLMDet (CVPR2025 Highlight)
tattaka
0
310
Delta airlines®️ USA Contact Numbers: Complete 2025 Support Guide
airtravelguide
0
340
United Airlines Customer Service– Call 1-833-341-3142 Now!
airhelp
0
160
使いたいMCPサーバーはWeb APIをラップして自分で作る #QiitaBash
bengo4com
0
1.6k
Understanding_Thread_Tuning_for_Inference_Servers_of_Deep_Models.pdf
lycorptech_jp
PRO
0
170
Tech-Verse 2025 Global CTO Session
lycorptech_jp
PRO
0
1.7k
Operating Operator
shhnjk
1
510
Zero Data Loss Autonomous Recovery Service サービス概要
oracle4engineer
PRO
2
7.7k
Featured
See All Featured
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.3k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
138
34k
A better future with KSS
kneath
239
17k
Designing for humans not robots
tammielis
253
25k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.7k
Rails Girls Zürich Keynote
gr2m
94
14k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
5
270
Git: the NoSQL Database
bkeepers
PRO
430
65k
Large-scale JavaScript Application Architecture
addyosmani
512
110k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
34
5.9k
Building an army of robots
kneath
306
45k
Transcript
ϝϧΧϦͷγεςϜɾαʔϏε ࢹʹ͍ͭͯ Monitoring Seminar in Mercari 2017/Good/Meat @kazeburo
Me • Masahiro Nagano / խ • id:kazeburo • Mercari,
Inc Principal Engineer Site Reliability Engineering (SRE) Team
Agenda • Mercariͷ͜Ε·Ͱͱࢹπʔϧ • MackerelͰͷαʔόࢹ
~ 2014/9 JP ੴङ ΠϯϑϥνʔϜ!5PLZP
~2014/9 • ͘͞ΒΠϯλʔωοτੴङDCͷઐ༻αʔόͱΫϥυΛར༻ • ઐ༻αʔόʹͯZabbixαʔόΛߏங • ʮτϦΨʔʯΛ׆༻ͯ͠͞·͟·ͳࢹΛߦ͏ • ͍݅ࣜͩ͠Ͱෳࡶͳࢹ͕࣮ݱ •
ݱࡏͷࢹ߲ͷϕʔε͕Ͱ͖Δ
2014/9~ US JP ੴङ ΠϯϑϥνʔϜ!5PLZP
2014/9~ • USαʔϏε͕ AWS Oregon region ʹͯ։࢝ • ઐ༻αʔόͱΫϥυͷϚϧνˍϋΠϒϦουߏ •
USʹZabbix ServerΛߏஙͯ͠ɺ౦ژ͔Βࢹ
ଟRegion Zabbixͷ՝ • Zabbixͷઃఆ͕ͣΕ͍ͯ͘ • ӡ༻͍ͯ͠ΔZabbixͷόʔδϣϯ͕ҟͳΔ • ઐ༻αʔόͱAWSͰएׯҧ͏ࢹ߲ • JPͰ࡞ΓࠐΜͩࢹ͕USͰ࠶ݱͰ͖͍ͯͳ͍
• USͰ͚ͩى͖Δࢹ࿙ΕʹΑΔࣄނ • Zabbix ProxyΛར༻͠ɺ1ݸͷZabbix ServerूͳͲͷҊ
Zabbixͷ՝ • Zabbix ࣗମͷӡ༻ • όʔδϣϯΞοϓͷෛ୲ • MySQL ͷෛՙ͕େ͖͘ɺࢹԆͳͲൃੜ •
ෳࡶͳτϦΨʔͷཧ • ϚεΫϦοΫओମͷઃఆ • όʔδϣϯཧͳͲΛߦ͍͍ͨ • ࢹͷ௨Λվળ͍ͨ͠
2016/1~ US JP ੴङ 43&!5PLZP
mackerel ಋೖ • Service Metrics͔Βಋೖ • ؆୯ʹάϥϑ͕ඳ͚ɺࢹᮢͷઃఆ͕Ͱ͖Δ • fluentdɺNorikraͱͷΈ߹Θͤ •
ZabbixͷτϦΨʔͷҠ২ • τϦΨʔΛPluginͱ࣮ͯ͠ • Plugin GitͰཧ͠ɺAnsibleͰ
mackerel ಋೖ • ࢹπʔϧɺ࣌ܥྻDBͷӡ༻ͷΦϑϩʔυ • Կ͠ͳͯ͘ຖिόʔδϣϯΞοϓ • JP/USͰͷࢹ߲Λ߹ΘͤΔ • ҟͳΔͱ͜Ζ
Ansible templateͳͲͰٵऩ
2017/3~ US JP ੴङ UK 43&!5PLZP
2017/3~ • UK ͰͷαʔϏε։࢝ • UK Λ։࢝͢Δʹ͋ͨͬͯɺ͞Βʹ͏ҰͭͷΫϥυΛ࠾༻ • ࢹ͕ΫϥυԽ͞Ε͓ͯΓɺ৽ͨͳࢹαʔόͷՃඞཁͳ͠ •
JP/US ͷࢹ߲͕ͦͷ··ద༻Ͱ͖ɺΠϯϑϥετϥΫνϟͷߏஙྃͱͱ ʹࢹͷઃఆ͕ྃ
ݱࡏ US JP ੴङ UK 43&&OHJOFFST!+1646, Stackdriver Prometheus
ݱࡏ • ϚΠΫϩαʔϏεԽ • GKE ্ͷίϯςφɾαʔϏεͷࢹͷͨΊʹ Stack DriverɺPrometheusɺ DataDog ͷ׆༻
• αʔόαΠυΤϯδχΞࢹπʔϧΛར༻
ͦͷଞͷࢹ New Relic Kurado άϥϑը૾ΛҰؾʹݟΕΔͷͰศར جຊతͳϝτϦΫεͪ͜ΒͰݟΔ PHPͷ෦ͷτϨʔε ΞϓϦέʔγϣϯͷνϡʔχϯάͷࢀߟ
MackerelͰͷαʔόࢹ
https://speakerdeck.com/kazeburo/mackerel-day
ࢹʹ·ͭΘΔࣈ • ࢹϧʔϧ: 278 • Hostຖͷࢹϧʔϧ • MySQL: 34 •
Application: 39 • Search: 37 • Custom Plugin: 50+ (check + metrics + utils)
MySQLͷࢹ߲(1/4) • Connectivity • FileSystem % >85% >88% • Swap
% >50% >70% • ssh-alive • sshdͷϓϩηεࢹ • global-ip-and-iptables • global ipͷ༗ແͱiptablesͷঢ়ଶ • unbound-resolv • localͷunboudͰ໊લղܾ͕Ͱ͖Δ͔ • unbound-process • unboundͷϓϩηεࢹ • crond-process • crondͷϓϩηεࢹ • uptime • ࠶ىಈࢹ
MySQLͷࢹ߲(2/4) • inode-usage • inode༻ >80% >90% • uname-change •
unameίϚϯυͷ݁Ռͷdiffࢹ • passwd-change • passwdϑΝΠϧͷdiffࢹ • hostname-changed • hostnameίϚϯυͷ݁Ռͷdiffࢹ • custom.ntpq.synced.remote <0.1 <0.1 • custom.ntpq.offset.seconds >300 >300 (msec) • ntpͷಉظαʔόͱ࣌ࠁͷζϨ • custom.linux-lite.memory.avail <50MB <20MB • ۭ͖ϝϞϦ • custom.linux-lite.cpu-usage.cpu-steal >20% >20% • custom.linux-lite.cpu-usage.cpu-iowait >30% >50% • custom.linux-lite-cpu-usage.cpu-system >8% >8% • ͦΕͧΕͷCPU༻(100%্͕ݶ)
MySQLͷࢹ߲(3/4) • cutom.linux-lite.loadavg.per-cpu >3 >3 • ίΞͰׂͬͨϩʔυΞϕϨʔδ • postfix-smtp-alive •
SMTPϙʔτͷ֬ೝ • postfix-master-process • postfix masterϓϩηεࢹ • custom.postfix.mailq.queue >100 >5k • postfix mail Ωϡʔཹ • custom.linux-lite.process.all >2k >2k • custom.linux-lite.process.running >60 >100 • શϓϩηεͱ࣮ߦதͷϓϩηε • mysql-uptime • mysqlͷuptime • custom.mysql-lite.replication-threads.io <0.2 <0.2 • custom.mysql-lite.replication-threads.sql <0.2 <0.2 • ϨϓϦέʔγϣϯͷ֤threadͷঢ়ଶ
MySQLͷࢹ߲(4/4) • custom.mysql-lite.replication-behind- master.second >5 >5 • mysqlͷϨϓϦέʔγϣϯԆ • custom.mysql-lite.connections.utilization
>90 >90 • max_connectionsʹର͢ΔίωΫγϣϯ • custom.mysql-lite.threads.running >1k >2k • mysql্Ͱ࣮ߦதͷεϨου • mysql-slave-sql-error • replicationΤϥʔͷࢹ • machine-exceptions • αʔόͷϝϞϦΤϥʔࢹ • raid-disks • αʔόͷRAID/Diskঢ়ଶͷࢹ
ࢹͷҭ͔ͯͨ • ʮࢹܧଓతͳςετʯby kazuho • ʮςετϑΝʔετʯͰγεςϜΛߏங͠ͳ͕ΒࢹΛ࡞Δ • ϓϩηεࢹϙʔτࢹ • ʮϙετϞʔςϜʯͷରԠࡦͱͯ͠ࢹΛҭͯΔ
• ᮢͷௐɺL7Ϩϕϧͷࢹ
end