最近の監視(仮)/Recent system monitoring with mackerel

Transcript

1. ࠷ۙͷ؂ࢹ(Ծ) Mackerel Day #2 kazeburo

2. Me • Masahiro Nagano • @kazeburo • Mackerel Ξϯόαμʔ •

   Ұ൪޷͖ͳ mackerelϓϥάΠϯ͸ mackerel-plugin-axslog (ࣗ࡞)ͩͳɻ໾ ʹཱͭͱ͜ΖͰ͸ diff-detector(ࣗ࡞)

3. https://ihara2525.tumblr.com/ “Ұߦͷϩάͷ޲͜͏ʹ͸ɺҰਓͷϢʔβ͕͍Δ”

4. • Τϥʔ͸όδΣοτͱ͍͏ߟ͑ • ࢒೦ʹࢥ͏ϢʔβΛগͳ͍͖͍ͯͨ͘͠ • ؂ࢹ͸Ϣʔβͷظ଴ʹԠ͑ΔͨΊͷπʔϧ

5. • ʮ؂ࢹʯʹΑΓ • ো֐Λະવʹ๷͙ • ো֐ʹૉૣ͘ؾͮ͘ • ো֐ͷݪҼಛఆͷώϯτ/MTTRͷ୹ॖ

6. mackerel-plugin-axslog ͱ೿ੜ

7. None

8. • mackerel-plugin-axslog • https://github.com/kazeburo/mackerel-plugin-axslog • mackerel-plugin-accesslog ͱಉ͘͡accesslogͷՄࢹԽ • status͝ͱͷ਺ͱׂ߹ɺresponse timeͷpercentile஋Λܭଌ

   • ूܭ͢ΔlabelΛมߋՄೳɾJSONαϙʔτ • ߴ଎

9. 0 2 4 6 8 accesslog(ltsv) axslog(ltsv) axslog(json) 1.17s 0.584s

   7.976s 120ສߦ(20000req/sec)ͷϩάΛॲཧ͢Δ࣌ؒ

10. • ߴ଎Խͷར఺ • γεςϜෛՙΛؾʹͤͣɺʮΨϯΨϯʯϝτϦΫεΛ૿΍ͤΔ • ϑΟϧλػೳͷ௥Ճ

11. [plugin.metrics.axslog-web-googlebot] command = "/usr/local/bin/mackerel-plugin-axslog --key-prefix=googlebot \
 --logfile /var/log/nginx/access.log --filter Googlebot"

    [plugin.metrics.axslog-web-adsbot] command = "/usr/local/bin/mackerel-plugin-axslog --key-prefix=adsbot \
 --logfile /var/log/nginx/access.log --filter AdsBot-Google"

12. • mackerel-plugin-postfix-log • https://github.com/kazeburo/mackerel-plugin-postfix-log • SMTP • AxlogͰlogrotateΛ௥ै͢Δίʔυॻ͍ͨͷͰ͙͢Ͱ͖ͨ

13. None

14. postfixlog.total_delay.average 0.240476 1555681849 postfixlog.total_delay.99_percentile 0.250000 1555681849 postfixlog.total_delay.95_percentile 0.240000 1555681849 postfixlog.total_delay.90_percentile

    0.240000 1555681849 postfixlog.recving_delay.average 0.040000 1555681849 postfixlog.recving_delay.99_percentile 0.040000 1555681849 postfixlog.recving_delay.95_percentile 0.040000 1555681849 postfixlog.recving_delay.90_percentile 0.040000 1555681849 postfixlog.queuing_delay.average 0.000476 1555681849 postfixlog.queuing_delay.99_percentile 0.010000 1555681849 postfixlog.queuing_delay.95_percentile 0.000000 1555681849 postfixlog.queuing_delay.90_percentile 0.000000 1555681849 postfixlog.connection_delay.average 0.090000 1555681849 postfixlog.connection_delay.99_percentile 0.090000 1555681849 postfixlog.connection_delay.95_percentile 0.090000 1555681849 postfixlog.connection_delay.90_percentile 0.090000 1555681849 postfixlog.transmission_delay.average 0.090000 1555681849 postfixlog.transmission_delay.99_percentile 0.090000 1555681849 postfixlog.transmission_delay.95_percentile 0.090000 1555681849 postfixlog.transmission_delay.90_percentile 0.090000 1555681849 postfixlog.transfer_num.2xx_count 1.615385 1555681849 postfixlog.transfer_num.4xx_count 0.000000 1555681849 postfixlog.transfer_num.5xx_count 0.000000 1555681849 postfixlog.transfer_total.count 1.615385 1555681849 postfixlog.transfer_ratio.2xx_percentage 100.000000 1555681849 postfixlog.transfer_ratio.4xx_percentage 0.000000 1555681849 postfixlog.transfer_ratio.5xx_percentage 0.000000 1555681849 ड৴࣌ͷ஗Ԇ Ωϡʔʹೖ͍ͬͯΔ࣌ؒ ૹ৴ઌ΁ͷ઀ଓʹ͔͔Δ࣌ؒ ૹ৴ͷ஗Ԇ

15. • mackerel-plugin-log-incr-rate • https://github.com/kazeburo/mackerel-plugin-log-incr-rate • ͋Δϩάʹର͢Δผͷϩάͷߦ਺ͷ૿Ճׂ߹Λ؂ࢹ͢Δ • accesslogʹର͢Δapplication_log, errorlogͷ݅਺ͳͲ •

    application_log΁ͷॻ͖ࠐΈ͕രൃͯ͠ϩάܭଌʹӨڹͨ͠ো֐ʹର͢Δ؂ࢹ
16. None

17. • mackerel-plugin-gzip-response • Not OSS • ϨεϙϯεͷGzipѹॖ͕ߦΘΕ͍ͯΔ͔؂ࢹ • Ұ෦ΫϥΠΞϯτͷ࣮૷ͰGzip͕ແޮʹͳͬͨ··ɺେ͖ͳ഑৴ͱͳΔϚε λʔߋ৽͕ߦΘΕɺωοτϫʔΫʹෛՙ͕͔͔Δࣄଶ͕͋ͬͨ

    • ϩάʹϨεϙϯεͷ Conent-Encoding ϔομΛه࿥͠ूܭ
18. None

19. • Mackerel Plugin ࡞੒ύλʔϯ • ো֐/ো֐ʹܨ͕Δ໰୊ͷൃݟ • Metrics Plugin ͷ࡞੒ͱαʔό΁ͷ൓ө

    • ਺೔MetricsΛΈͯͷɺᮢ஋ઃఆɾਵ࣌ݟ௚͠

20. DB ӡ༻ͱ؂ࢹ

21. • ϗετͷεςʔλεΛ؂ࢹ͢Δ • ϗετεςʔλε Standby ͷDB(slave)αʔόΛ production ʹ౤ೖ • ϝϯςφϯε࡞ۀͷͨΊϨϓϦέʔγϣϯ͕ࢭ·͍͕ͬͯͨɺ௨஌͸དྷͳ͍

22. App App consul consul DNS consul consul 1) look up

    DNS 2) connect DB check mysql uptime before slave slave slave

23. App App consul consul DNS consul consul 1) look up

    DNS 2) connect DB A) check mysql uptime After Mackerel batch batch batch B) check host status file B’ ) update host status file by cron slave slave slave

24. • Consul ͷαʔϏε͕༗ޮʹͳΔͨΊʹ͸ɺmackerelͷεςʔλεͷมߋ͕ ඞཁɻΞϥʔτʹؾ͕ͭ͘Մೳੑˢ • ϗετεςʔλεΛϩʔΧϧͷͲ͔͜ʹॻ͖ࠐΉػೳɺ
 mackerel-agent ʹ͋Δͱศར?

25. • Slave Ͱͷ Trigger ࣮ߦΛकΔ • ProductionͷҰ෦ͷslaveΛɺௐࠪɾूܭ༻ͱͯ͠࢖͑ΔΑ͏ʹ͍ͯ͠Δ • ηϯγςΟϒͳ৘ใʹ͍ͭͯ͸Slave্ͷTriggerͰ࡟আ

26. • MySQLͱϨϓϦέʔγϣϯͱTrigger • MySQLͷϨϓϦέʔγϣϯͷํࣜʹ͸SQL͕సૹ͞Εͯ͘ΔStatementํࣜͱɺ σʔλ͕ૹΒΕͯ͘ΔRow Basedํࣜɺ·ͨͲͪΒ͔͕ࣗಈͰબ୒͞ΕΔMixed ํ͕ࣜ͋ΔɻMixedํࣜͰ͸جຊ͸Statementํ͕ࣜͩɺඇܾఆੑΫΤϦ͕࢖Θ ΕΔͱɺRow BasedͱͳΔ •

    ϨϓϦέʔγϣϯΛड͚Δslaveαʔό্ͰͷTrigger͸ɺStatementํࣜͷ৔߹ͷ Έ࣮ߦ͞ΕΔ • SlaveͰtriggerΛ҆શʹར༻͢ΔͨΊʹ͸ҙਤ͠ͳ͍ Row basedͳϨϓϦέʔ γϣϯΛൃݟ͢Δඞཁ͕͋Δ

27. • check-mysqlbinlog-rbr-fallbacks • No OSS • mysqlbinlog ίϚϯυ࢖͍ɺRow BasedʹͳͬͨςʔϒϧΛݕग़

28. Master Backup Anon Slave Trigger binlog binlog check-mysqlbinlog-rbr-fallbacks Detect the

    SQL and fix data and 
 applications if necessary App

29. • ௅ઓ͍ͯ͠ΔݶΓɺ؂ࢹ͸૿͑Δ • Application Server: 175 Metrics • DB Server:

    245 Metrics

30. • Ҏ্