Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
メルカリのシステム・サービス監視について/Monitoring Mercari service and servers
kazeburo
November 29, 2017
Technology
5
5.8k
メルカリのシステム・サービス監視について/Monitoring Mercari service and servers
メルカリのシステム・サービス監視について
Monitoring seminar in Mercari
kazeburo
November 29, 2017
Tweet
Share
More Decks by kazeburo
See All by kazeburo
kazeburo
1
120
kazeburo
2
810
kazeburo
3
3.1k
kazeburo
2
2k
kazeburo
23
4.1k
kazeburo
21
5.1k
kazeburo
9
9.7k
kazeburo
8
4.5k
kazeburo
26
8.4k
Other Decks in Technology
See All in Technology
am7cinnamon
1
300
kanaugust
PRO
0
210
tatsy
2
430
yashi8484
0
130
surumegohan
1
170
legalforce
PRO
0
160
ahana
0
400
5ebec
0
160
tsuyo
0
490
line_developers
PRO
2
160
daisukehirama41
1
330
clustervr
PRO
0
210
Featured
See All Featured
tanoku
86
8.6k
sugarenia
233
880k
ufuk
56
5.5k
addyosmani
1346
200k
holman
288
130k
jonrohan
1021
390k
sferik
613
55k
mthomps
38
2.3k
mza
80
4.2k
reverentgeek
167
7.3k
mojombo
359
62k
addyosmani
312
21k
Transcript
ϝϧΧϦͷγεςϜɾαʔϏε ࢹʹ͍ͭͯ Monitoring Seminar in Mercari 2017/Good/Meat @kazeburo
Me • Masahiro Nagano / խ • id:kazeburo • Mercari,
Inc Principal Engineer Site Reliability Engineering (SRE) Team
Agenda • Mercariͷ͜Ε·Ͱͱࢹπʔϧ • MackerelͰͷαʔόࢹ
~ 2014/9 JP ੴङ ΠϯϑϥνʔϜ!5PLZP
~2014/9 • ͘͞ΒΠϯλʔωοτੴङDCͷઐ༻αʔόͱΫϥυΛར༻ • ઐ༻αʔόʹͯZabbixαʔόΛߏங • ʮτϦΨʔʯΛ׆༻ͯ͠͞·͟·ͳࢹΛߦ͏ • ͍݅ࣜͩ͠Ͱෳࡶͳࢹ͕࣮ݱ •
ݱࡏͷࢹ߲ͷϕʔε͕Ͱ͖Δ
2014/9~ US JP ੴङ ΠϯϑϥνʔϜ!5PLZP
2014/9~ • USαʔϏε͕ AWS Oregon region ʹͯ։࢝ • ઐ༻αʔόͱΫϥυͷϚϧνˍϋΠϒϦουߏ •
USʹZabbix ServerΛߏஙͯ͠ɺ౦ژ͔Βࢹ
ଟRegion Zabbixͷ՝ • Zabbixͷઃఆ͕ͣΕ͍ͯ͘ • ӡ༻͍ͯ͠ΔZabbixͷόʔδϣϯ͕ҟͳΔ • ઐ༻αʔόͱAWSͰएׯҧ͏ࢹ߲ • JPͰ࡞ΓࠐΜͩࢹ͕USͰ࠶ݱͰ͖͍ͯͳ͍
• USͰ͚ͩى͖Δࢹ࿙ΕʹΑΔࣄނ • Zabbix ProxyΛར༻͠ɺ1ݸͷZabbix ServerूͳͲͷҊ
Zabbixͷ՝ • Zabbix ࣗମͷӡ༻ • όʔδϣϯΞοϓͷෛ୲ • MySQL ͷෛՙ͕େ͖͘ɺࢹԆͳͲൃੜ •
ෳࡶͳτϦΨʔͷཧ • ϚεΫϦοΫओମͷઃఆ • όʔδϣϯཧͳͲΛߦ͍͍ͨ • ࢹͷ௨Λվળ͍ͨ͠
2016/1~ US JP ੴङ 43&!5PLZP
mackerel ಋೖ • Service Metrics͔Βಋೖ • ؆୯ʹάϥϑ͕ඳ͚ɺࢹᮢͷઃఆ͕Ͱ͖Δ • fluentdɺNorikraͱͷΈ߹Θͤ •
ZabbixͷτϦΨʔͷҠ২ • τϦΨʔΛPluginͱ࣮ͯ͠ • Plugin GitͰཧ͠ɺAnsibleͰ
mackerel ಋೖ • ࢹπʔϧɺ࣌ܥྻDBͷӡ༻ͷΦϑϩʔυ • Կ͠ͳͯ͘ຖिόʔδϣϯΞοϓ • JP/USͰͷࢹ߲Λ߹ΘͤΔ • ҟͳΔͱ͜Ζ
Ansible templateͳͲͰٵऩ
2017/3~ US JP ੴङ UK 43&!5PLZP
2017/3~ • UK ͰͷαʔϏε։࢝ • UK Λ։࢝͢Δʹ͋ͨͬͯɺ͞Βʹ͏ҰͭͷΫϥυΛ࠾༻ • ࢹ͕ΫϥυԽ͞Ε͓ͯΓɺ৽ͨͳࢹαʔόͷՃඞཁͳ͠ •
JP/US ͷࢹ߲͕ͦͷ··ద༻Ͱ͖ɺΠϯϑϥετϥΫνϟͷߏஙྃͱͱ ʹࢹͷઃఆ͕ྃ
ݱࡏ US JP ੴङ UK 43&&OHJOFFST!+1646, Stackdriver Prometheus
ݱࡏ • ϚΠΫϩαʔϏεԽ • GKE ্ͷίϯςφɾαʔϏεͷࢹͷͨΊʹ Stack DriverɺPrometheusɺ DataDog ͷ׆༻
• αʔόαΠυΤϯδχΞࢹπʔϧΛར༻
ͦͷଞͷࢹ New Relic Kurado άϥϑը૾ΛҰؾʹݟΕΔͷͰศར جຊతͳϝτϦΫεͪ͜ΒͰݟΔ PHPͷ෦ͷτϨʔε ΞϓϦέʔγϣϯͷνϡʔχϯάͷࢀߟ
MackerelͰͷαʔόࢹ
https://speakerdeck.com/kazeburo/mackerel-day
ࢹʹ·ͭΘΔࣈ • ࢹϧʔϧ: 278 • Hostຖͷࢹϧʔϧ • MySQL: 34 •
Application: 39 • Search: 37 • Custom Plugin: 50+ (check + metrics + utils)
MySQLͷࢹ߲(1/4) • Connectivity • FileSystem % >85% >88% • Swap
% >50% >70% • ssh-alive • sshdͷϓϩηεࢹ • global-ip-and-iptables • global ipͷ༗ແͱiptablesͷঢ়ଶ • unbound-resolv • localͷunboudͰ໊લղܾ͕Ͱ͖Δ͔ • unbound-process • unboundͷϓϩηεࢹ • crond-process • crondͷϓϩηεࢹ • uptime • ࠶ىಈࢹ
MySQLͷࢹ߲(2/4) • inode-usage • inode༻ >80% >90% • uname-change •
unameίϚϯυͷ݁Ռͷdiffࢹ • passwd-change • passwdϑΝΠϧͷdiffࢹ • hostname-changed • hostnameίϚϯυͷ݁Ռͷdiffࢹ • custom.ntpq.synced.remote <0.1 <0.1 • custom.ntpq.offset.seconds >300 >300 (msec) • ntpͷಉظαʔόͱ࣌ࠁͷζϨ • custom.linux-lite.memory.avail <50MB <20MB • ۭ͖ϝϞϦ • custom.linux-lite.cpu-usage.cpu-steal >20% >20% • custom.linux-lite.cpu-usage.cpu-iowait >30% >50% • custom.linux-lite-cpu-usage.cpu-system >8% >8% • ͦΕͧΕͷCPU༻(100%্͕ݶ)
MySQLͷࢹ߲(3/4) • cutom.linux-lite.loadavg.per-cpu >3 >3 • ίΞͰׂͬͨϩʔυΞϕϨʔδ • postfix-smtp-alive •
SMTPϙʔτͷ֬ೝ • postfix-master-process • postfix masterϓϩηεࢹ • custom.postfix.mailq.queue >100 >5k • postfix mail Ωϡʔཹ • custom.linux-lite.process.all >2k >2k • custom.linux-lite.process.running >60 >100 • શϓϩηεͱ࣮ߦதͷϓϩηε • mysql-uptime • mysqlͷuptime • custom.mysql-lite.replication-threads.io <0.2 <0.2 • custom.mysql-lite.replication-threads.sql <0.2 <0.2 • ϨϓϦέʔγϣϯͷ֤threadͷঢ়ଶ
MySQLͷࢹ߲(4/4) • custom.mysql-lite.replication-behind- master.second >5 >5 • mysqlͷϨϓϦέʔγϣϯԆ • custom.mysql-lite.connections.utilization
>90 >90 • max_connectionsʹର͢ΔίωΫγϣϯ • custom.mysql-lite.threads.running >1k >2k • mysql্Ͱ࣮ߦதͷεϨου • mysql-slave-sql-error • replicationΤϥʔͷࢹ • machine-exceptions • αʔόͷϝϞϦΤϥʔࢹ • raid-disks • αʔόͷRAID/Diskঢ়ଶͷࢹ
ࢹͷҭ͔ͯͨ • ʮࢹܧଓతͳςετʯby kazuho • ʮςετϑΝʔετʯͰγεςϜΛߏங͠ͳ͕ΒࢹΛ࡞Δ • ϓϩηεࢹϙʔτࢹ • ʮϙετϞʔςϜʯͷରԠࡦͱͯ͠ࢹΛҭͯΔ
• ᮢͷௐɺL7Ϩϕϧͷࢹ
end