Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
最近の監視(仮)/Recent system monitoring with mackerel
kazeburo
December 23, 2019
Technology
3
3.1k
最近の監視(仮)/Recent system monitoring with mackerel
mackerel day #2
kazeburo
December 23, 2019
Tweet
Share
More Decks by kazeburo
See All by kazeburo
kazeburo
2
130
kazeburo
2
810
kazeburo
2
2k
kazeburo
23
4.1k
kazeburo
21
5.1k
kazeburo
9
9.7k
kazeburo
8
4.5k
kazeburo
26
8.4k
kazeburo
5
5.8k
Other Decks in Technology
See All in Technology
nnstt1
2
120
chipstar_light
0
490
autifyhq
1
780
akabekobeko
0
210
shwars
0
110
yuhta28
0
150
kanaugust
PRO
0
120
kanaugust
PRO
0
210
hmatsu47
1
170
miura55
0
130
ayatokura
1
170
kentaro
2
520
Featured
See All Featured
shlominoach
176
7.6k
dougneiner
56
5.4k
hursman
107
9.3k
62gerente
586
200k
skipperchong
8
790
frogandcode
128
20k
pauljervisheath
195
15k
akmur
252
19k
matthewcrist
73
7.5k
maggiecrowley
10
550
jacobian
257
20k
zenorocha
296
40k
Transcript
࠷ۙͷࢹ(Ծ) Mackerel Day #2 kazeburo
Me • Masahiro Nagano • @kazeburo • Mackerel Ξϯόαμʔ •
Ұ൪͖ͳ mackerelϓϥάΠϯ mackerel-plugin-axslog (ࣗ࡞)ͩͳɻ ʹཱͭͱ͜ΖͰ diff-detector(ࣗ࡞)
https://ihara2525.tumblr.com/ “Ұߦͷϩάͷ͜͏ʹɺҰਓͷϢʔβ͕͍Δ”
• ΤϥʔόδΣοτͱ͍͏ߟ͑ • ೦ʹࢥ͏ϢʔβΛগͳ͍͖͍ͯͨ͘͠ • ࢹϢʔβͷظʹԠ͑ΔͨΊͷπʔϧ
• ʮࢹʯʹΑΓ • োΛະવʹ͙ • োʹૉૣ͘ؾͮ͘ • োͷݪҼಛఆͷώϯτ/MTTRͷॖ
mackerel-plugin-axslog ͱੜ
None
• mackerel-plugin-axslog • https://github.com/kazeburo/mackerel-plugin-axslog • mackerel-plugin-accesslog ͱಉ͘͡accesslogͷՄࢹԽ • status͝ͱͷͱׂ߹ɺresponse timeͷpercentileΛܭଌ
• ूܭ͢ΔlabelΛมߋՄೳɾJSONαϙʔτ • ߴ
0 2 4 6 8 accesslog(ltsv) axslog(ltsv) axslog(json) 1.17s 0.584s
7.976s 120ສߦ(20000req/sec)ͷϩάΛॲཧ͢Δ࣌ؒ
• ߴԽͷར • γεςϜෛՙΛؾʹͤͣɺʮΨϯΨϯʯϝτϦΫεΛ૿ͤΔ • ϑΟϧλػೳͷՃ
[plugin.metrics.axslog-web-googlebot] command = "/usr/local/bin/mackerel-plugin-axslog --key-prefix=googlebot \ --logfile /var/log/nginx/access.log --filter Googlebot"
[plugin.metrics.axslog-web-adsbot] command = "/usr/local/bin/mackerel-plugin-axslog --key-prefix=adsbot \ --logfile /var/log/nginx/access.log --filter AdsBot-Google"
• mackerel-plugin-postfix-log • https://github.com/kazeburo/mackerel-plugin-postfix-log • SMTP • AxlogͰlogrotateΛै͢Δίʔυॻ͍ͨͷͰ͙͢Ͱ͖ͨ
None
postfixlog.total_delay.average 0.240476 1555681849 postfixlog.total_delay.99_percentile 0.250000 1555681849 postfixlog.total_delay.95_percentile 0.240000 1555681849 postfixlog.total_delay.90_percentile
0.240000 1555681849 postfixlog.recving_delay.average 0.040000 1555681849 postfixlog.recving_delay.99_percentile 0.040000 1555681849 postfixlog.recving_delay.95_percentile 0.040000 1555681849 postfixlog.recving_delay.90_percentile 0.040000 1555681849 postfixlog.queuing_delay.average 0.000476 1555681849 postfixlog.queuing_delay.99_percentile 0.010000 1555681849 postfixlog.queuing_delay.95_percentile 0.000000 1555681849 postfixlog.queuing_delay.90_percentile 0.000000 1555681849 postfixlog.connection_delay.average 0.090000 1555681849 postfixlog.connection_delay.99_percentile 0.090000 1555681849 postfixlog.connection_delay.95_percentile 0.090000 1555681849 postfixlog.connection_delay.90_percentile 0.090000 1555681849 postfixlog.transmission_delay.average 0.090000 1555681849 postfixlog.transmission_delay.99_percentile 0.090000 1555681849 postfixlog.transmission_delay.95_percentile 0.090000 1555681849 postfixlog.transmission_delay.90_percentile 0.090000 1555681849 postfixlog.transfer_num.2xx_count 1.615385 1555681849 postfixlog.transfer_num.4xx_count 0.000000 1555681849 postfixlog.transfer_num.5xx_count 0.000000 1555681849 postfixlog.transfer_total.count 1.615385 1555681849 postfixlog.transfer_ratio.2xx_percentage 100.000000 1555681849 postfixlog.transfer_ratio.4xx_percentage 0.000000 1555681849 postfixlog.transfer_ratio.5xx_percentage 0.000000 1555681849 ड৴࣌ͷԆ Ωϡʔʹೖ͍ͬͯΔ࣌ؒ ૹ৴ઌͷଓʹ͔͔Δ࣌ؒ ૹ৴ͷԆ
• mackerel-plugin-log-incr-rate • https://github.com/kazeburo/mackerel-plugin-log-incr-rate • ͋Δϩάʹର͢Δผͷϩάͷߦͷ૿Ճׂ߹Λࢹ͢Δ • accesslogʹର͢Δapplication_log, errorlogͷ݅ͳͲ •
application_logͷॻ͖ࠐΈ͕രൃͯ͠ϩάܭଌʹӨڹͨ͠োʹର͢Δࢹ
None
• mackerel-plugin-gzip-response • Not OSS • ϨεϙϯεͷGzipѹॖ͕ߦΘΕ͍ͯΔ͔ࢹ • Ұ෦ΫϥΠΞϯτͷ࣮ͰGzip͕ແޮʹͳͬͨ··ɺେ͖ͳ৴ͱͳΔϚε λʔߋ৽͕ߦΘΕɺωοτϫʔΫʹෛՙ͕͔͔Δࣄଶ͕͋ͬͨ
• ϩάʹϨεϙϯεͷ Conent-Encoding ϔομΛه͠ूܭ
None
• Mackerel Plugin ࡞ύλʔϯ • ো/োʹܨ͕Δͷൃݟ • Metrics Plugin ͷ࡞ͱαʔόͷө
• MetricsΛΈͯͷɺᮢઃఆɾਵ࣌ݟ͠
DB ӡ༻ͱࢹ
• ϗετͷεςʔλεΛࢹ͢Δ • ϗετεςʔλε Standby ͷDB(slave)αʔόΛ production ʹೖ • ϝϯςφϯε࡞ۀͷͨΊϨϓϦέʔγϣϯ͕ࢭ·͍͕ͬͯͨɺ௨དྷͳ͍
App App consul consul DNS consul consul 1) look up
DNS 2) connect DB check mysql uptime before slave slave slave
App App consul consul DNS consul consul 1) look up
DNS 2) connect DB A) check mysql uptime After Mackerel batch batch batch B) check host status file B’ ) update host status file by cron slave slave slave
• Consul ͷαʔϏε͕༗ޮʹͳΔͨΊʹɺmackerelͷεςʔλεͷมߋ͕ ඞཁɻΞϥʔτʹؾ͕ͭ͘Մೳੑˢ • ϗετεςʔλεΛϩʔΧϧͷͲ͔͜ʹॻ͖ࠐΉػೳɺ mackerel-agent ʹ͋Δͱศར?
• Slave Ͱͷ Trigger ࣮ߦΛकΔ • ProductionͷҰ෦ͷslaveΛɺௐࠪɾूܭ༻ͱͯ͑͠ΔΑ͏ʹ͍ͯ͠Δ • ηϯγςΟϒͳใʹ͍ͭͯSlave্ͷTriggerͰআ
• MySQLͱϨϓϦέʔγϣϯͱTrigger • MySQLͷϨϓϦέʔγϣϯͷํࣜʹSQL͕సૹ͞Εͯ͘ΔStatementํࣜͱɺ σʔλ͕ૹΒΕͯ͘ΔRow Basedํࣜɺ·ͨͲͪΒ͔͕ࣗಈͰબ͞ΕΔMixed ํ͕ࣜ͋ΔɻMixedํࣜͰجຊStatementํ͕ࣜͩɺඇܾఆੑΫΤϦ͕Θ ΕΔͱɺRow BasedͱͳΔ •
ϨϓϦέʔγϣϯΛड͚Δslaveαʔό্ͰͷTriggerɺStatementํࣜͷ߹ͷ Έ࣮ߦ͞ΕΔ • SlaveͰtriggerΛ҆શʹར༻͢ΔͨΊʹҙਤ͠ͳ͍ Row basedͳϨϓϦέʔ γϣϯΛൃݟ͢Δඞཁ͕͋Δ
• check-mysqlbinlog-rbr-fallbacks • No OSS • mysqlbinlog ίϚϯυ͍ɺRow BasedʹͳͬͨςʔϒϧΛݕग़
Master Backup Anon Slave Trigger binlog binlog check-mysqlbinlog-rbr-fallbacks Detect the
SQL and fix data and applications if necessary App
• ઓ͍ͯ͠ΔݶΓɺࢹ૿͑Δ • Application Server: 175 Metrics • DB Server:
245 Metrics
• Ҏ্