Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
メルカリのシステム・サービス監視について/Monitoring Mercari service and servers
Search
kazeburo
November 29, 2017
Technology
5
6.7k
メルカリのシステム・サービス監視について/Monitoring Mercari service and servers
メルカリのシステム・サービス監視について
Monitoring seminar in Mercari
kazeburo
November 29, 2017
Tweet
Share
More Decks by kazeburo
See All by kazeburo
DNS水責め攻撃と監視 / DNS water torture attack Monitoring and SLO
kazeburo
4
3.6k
DBやめてみた / DNS water torture attack and countermeasures
kazeburo
13
11k
IaaSにおけるPlatform Engineeringとこれから / Platform engineering in IaaS
kazeburo
2
1.1k
高信頼IaaSを実現するDevOps / DevOps for Highly Reliable IaaS
kazeburo
1
460
権威DNSサービスへのDDoSと ハイパフォーマンスなベンチマーカ / DNS Pseudo random subdomain attack and High performance Benchmarker
kazeburo
3
4.5k
DNS権威サーバのクラウドサービス向けに行われた攻撃および対策 / DNS Pseudo-Random Subdomain Attack and mitigations
kazeburo
7
11k
sacloudns
kazeburo
2
280
「orchestratorとGTID運用を支える監視」の勉強 / Monitoring orchestrator and GTID operation
kazeburo
2
1.2k
最近の監視(仮)/Recent system monitoring with mackerel
kazeburo
3
4.5k
Other Decks in Technology
See All in Technology
ServiceNow Knowledge 24の歩き方 EYストラテジー・アンド・コンサルティング
manarobot
0
200
GraphQL 成熟度モデルの紹介と、プロダクトに当てはめた事例 / GraphQL maturity model
mh4gf
7
1.3k
Cloud Native Java with Spring Boot (CNCF Aarhus, April 2024)
thomasvitale
1
170
FrontDoorとWebAppsを組み合わせた際のリダイレクト処理の注意点
kenichirokimura
1
530
Building a RAG-poweredAI chat appwith Python and VS Code
pamelafox
0
100
プロンプトエンジニアリングでがんばらない-Agentic Workflow へ-近藤憲児
kenjikondobai
3
860
いつか使うかも貯金してたらめちゃめちゃ機能が増えてた話
riyaamemiya
0
300
IaCジェネレーターとBedrockで詳細設計書を生成してみた
tsukasa_ishimaru
1
280
MapLibreとAmazon Location Service
dayjournal
1
160
EMとして2023年度に頑張ったこと / What we did well in FY2023 as a EM
pauli
1
170
ワールドカフェI /チューターを改良する / World Café I and Improving the Tutors
ks91
PRO
0
120
LayerXにおけるLLMプロダクト開発の今までとこれから
layerx
PRO
1
370
Featured
See All Featured
Pencils Down: Stop Designing & Start Developing
hursman
117
11k
Building Flexible Design Systems
yeseniaperezcruz
319
37k
The MySQL Ecosystem @ GitHub 2015
samlambert
243
12k
Making the Leap to Tech Lead
cromwellryan
124
8.5k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
78
43k
Bootstrapping a Software Product
garrettdimon
PRO
302
110k
4 Signs Your Business is Dying
shpigford
175
21k
A designer walks into a library…
pauljervisheath
200
23k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
17
1.4k
Happy Clients
brianwarren
92
6.4k
YesSQL, Process and Tooling at Scale
rocio
164
13k
10 Git Anti Patterns You Should be Aware of
lemiorhan
648
58k
Transcript
ϝϧΧϦͷγεςϜɾαʔϏε ࢹʹ͍ͭͯ Monitoring Seminar in Mercari 2017/Good/Meat @kazeburo
Me • Masahiro Nagano / խ • id:kazeburo • Mercari,
Inc Principal Engineer Site Reliability Engineering (SRE) Team
Agenda • Mercariͷ͜Ε·Ͱͱࢹπʔϧ • MackerelͰͷαʔόࢹ
~ 2014/9 JP ੴङ ΠϯϑϥνʔϜ!5PLZP
~2014/9 • ͘͞ΒΠϯλʔωοτੴङDCͷઐ༻αʔόͱΫϥυΛར༻ • ઐ༻αʔόʹͯZabbixαʔόΛߏங • ʮτϦΨʔʯΛ׆༻ͯ͠͞·͟·ͳࢹΛߦ͏ • ͍݅ࣜͩ͠Ͱෳࡶͳࢹ͕࣮ݱ •
ݱࡏͷࢹ߲ͷϕʔε͕Ͱ͖Δ
2014/9~ US JP ੴङ ΠϯϑϥνʔϜ!5PLZP
2014/9~ • USαʔϏε͕ AWS Oregon region ʹͯ։࢝ • ઐ༻αʔόͱΫϥυͷϚϧνˍϋΠϒϦουߏ •
USʹZabbix ServerΛߏஙͯ͠ɺ౦ژ͔Βࢹ
ଟRegion Zabbixͷ՝ • Zabbixͷઃఆ͕ͣΕ͍ͯ͘ • ӡ༻͍ͯ͠ΔZabbixͷόʔδϣϯ͕ҟͳΔ • ઐ༻αʔόͱAWSͰएׯҧ͏ࢹ߲ • JPͰ࡞ΓࠐΜͩࢹ͕USͰ࠶ݱͰ͖͍ͯͳ͍
• USͰ͚ͩى͖Δࢹ࿙ΕʹΑΔࣄނ • Zabbix ProxyΛར༻͠ɺ1ݸͷZabbix ServerूͳͲͷҊ
Zabbixͷ՝ • Zabbix ࣗମͷӡ༻ • όʔδϣϯΞοϓͷෛ୲ • MySQL ͷෛՙ͕େ͖͘ɺࢹԆͳͲൃੜ •
ෳࡶͳτϦΨʔͷཧ • ϚεΫϦοΫओମͷઃఆ • όʔδϣϯཧͳͲΛߦ͍͍ͨ • ࢹͷ௨Λվળ͍ͨ͠
2016/1~ US JP ੴङ 43&!5PLZP
mackerel ಋೖ • Service Metrics͔Βಋೖ • ؆୯ʹάϥϑ͕ඳ͚ɺࢹᮢͷઃఆ͕Ͱ͖Δ • fluentdɺNorikraͱͷΈ߹Θͤ •
ZabbixͷτϦΨʔͷҠ২ • τϦΨʔΛPluginͱ࣮ͯ͠ • Plugin GitͰཧ͠ɺAnsibleͰ
mackerel ಋೖ • ࢹπʔϧɺ࣌ܥྻDBͷӡ༻ͷΦϑϩʔυ • Կ͠ͳͯ͘ຖिόʔδϣϯΞοϓ • JP/USͰͷࢹ߲Λ߹ΘͤΔ • ҟͳΔͱ͜Ζ
Ansible templateͳͲͰٵऩ
2017/3~ US JP ੴङ UK 43&!5PLZP
2017/3~ • UK ͰͷαʔϏε։࢝ • UK Λ։࢝͢Δʹ͋ͨͬͯɺ͞Βʹ͏ҰͭͷΫϥυΛ࠾༻ • ࢹ͕ΫϥυԽ͞Ε͓ͯΓɺ৽ͨͳࢹαʔόͷՃඞཁͳ͠ •
JP/US ͷࢹ߲͕ͦͷ··ద༻Ͱ͖ɺΠϯϑϥετϥΫνϟͷߏஙྃͱͱ ʹࢹͷઃఆ͕ྃ
ݱࡏ US JP ੴङ UK 43&&OHJOFFST!+1646, Stackdriver Prometheus
ݱࡏ • ϚΠΫϩαʔϏεԽ • GKE ্ͷίϯςφɾαʔϏεͷࢹͷͨΊʹ Stack DriverɺPrometheusɺ DataDog ͷ׆༻
• αʔόαΠυΤϯδχΞࢹπʔϧΛར༻
ͦͷଞͷࢹ New Relic Kurado άϥϑը૾ΛҰؾʹݟΕΔͷͰศར جຊతͳϝτϦΫεͪ͜ΒͰݟΔ PHPͷ෦ͷτϨʔε ΞϓϦέʔγϣϯͷνϡʔχϯάͷࢀߟ
MackerelͰͷαʔόࢹ
https://speakerdeck.com/kazeburo/mackerel-day
ࢹʹ·ͭΘΔࣈ • ࢹϧʔϧ: 278 • Hostຖͷࢹϧʔϧ • MySQL: 34 •
Application: 39 • Search: 37 • Custom Plugin: 50+ (check + metrics + utils)
MySQLͷࢹ߲(1/4) • Connectivity • FileSystem % >85% >88% • Swap
% >50% >70% • ssh-alive • sshdͷϓϩηεࢹ • global-ip-and-iptables • global ipͷ༗ແͱiptablesͷঢ়ଶ • unbound-resolv • localͷunboudͰ໊લղܾ͕Ͱ͖Δ͔ • unbound-process • unboundͷϓϩηεࢹ • crond-process • crondͷϓϩηεࢹ • uptime • ࠶ىಈࢹ
MySQLͷࢹ߲(2/4) • inode-usage • inode༻ >80% >90% • uname-change •
unameίϚϯυͷ݁Ռͷdiffࢹ • passwd-change • passwdϑΝΠϧͷdiffࢹ • hostname-changed • hostnameίϚϯυͷ݁Ռͷdiffࢹ • custom.ntpq.synced.remote <0.1 <0.1 • custom.ntpq.offset.seconds >300 >300 (msec) • ntpͷಉظαʔόͱ࣌ࠁͷζϨ • custom.linux-lite.memory.avail <50MB <20MB • ۭ͖ϝϞϦ • custom.linux-lite.cpu-usage.cpu-steal >20% >20% • custom.linux-lite.cpu-usage.cpu-iowait >30% >50% • custom.linux-lite-cpu-usage.cpu-system >8% >8% • ͦΕͧΕͷCPU༻(100%্͕ݶ)
MySQLͷࢹ߲(3/4) • cutom.linux-lite.loadavg.per-cpu >3 >3 • ίΞͰׂͬͨϩʔυΞϕϨʔδ • postfix-smtp-alive •
SMTPϙʔτͷ֬ೝ • postfix-master-process • postfix masterϓϩηεࢹ • custom.postfix.mailq.queue >100 >5k • postfix mail Ωϡʔཹ • custom.linux-lite.process.all >2k >2k • custom.linux-lite.process.running >60 >100 • શϓϩηεͱ࣮ߦதͷϓϩηε • mysql-uptime • mysqlͷuptime • custom.mysql-lite.replication-threads.io <0.2 <0.2 • custom.mysql-lite.replication-threads.sql <0.2 <0.2 • ϨϓϦέʔγϣϯͷ֤threadͷঢ়ଶ
MySQLͷࢹ߲(4/4) • custom.mysql-lite.replication-behind- master.second >5 >5 • mysqlͷϨϓϦέʔγϣϯԆ • custom.mysql-lite.connections.utilization
>90 >90 • max_connectionsʹର͢ΔίωΫγϣϯ • custom.mysql-lite.threads.running >1k >2k • mysql্Ͱ࣮ߦதͷεϨου • mysql-slave-sql-error • replicationΤϥʔͷࢹ • machine-exceptions • αʔόͷϝϞϦΤϥʔࢹ • raid-disks • αʔόͷRAID/Diskঢ়ଶͷࢹ
ࢹͷҭ͔ͯͨ • ʮࢹܧଓతͳςετʯby kazuho • ʮςετϑΝʔετʯͰγεςϜΛߏங͠ͳ͕ΒࢹΛ࡞Δ • ϓϩηεࢹϙʔτࢹ • ʮϙετϞʔςϜʯͷରԠࡦͱͯ͠ࢹΛҭͯΔ
• ᮢͷௐɺL7Ϩϕϧͷࢹ
end