$30 off During Our Annual Pro Sale. View Details »

最近の監視(仮)/Recent system monitoring with mackerel

kazeburo
December 23, 2019

最近の監視(仮)/Recent system monitoring with mackerel

mackerel day #2

kazeburo

December 23, 2019
Tweet

More Decks by kazeburo

Other Decks in Technology

Transcript

  1. ࠷ۙͷ؂ࢹ(Ծ)
    Mackerel Day #2
    kazeburo

    View Slide

  2. Me
    • Masahiro Nagano
    • @kazeburo
    • Mackerel Ξϯόαμʔ
    • Ұ൪޷͖ͳ mackerelϓϥάΠϯ͸ mackerel-plugin-axslog (ࣗ࡞)ͩͳɻ໾
    ʹཱͭͱ͜ΖͰ͸ diff-detector(ࣗ࡞)

    View Slide

  3. https://ihara2525.tumblr.com/
    “Ұߦͷϩάͷ޲͜͏ʹ͸ɺҰਓͷϢʔβ͕͍Δ”

    View Slide

  4. • Τϥʔ͸όδΣοτͱ͍͏ߟ͑
    • ࢒೦ʹࢥ͏ϢʔβΛগͳ͍͖͍ͯͨ͘͠
    • ؂ࢹ͸Ϣʔβͷظ଴ʹԠ͑ΔͨΊͷπʔϧ

    View Slide

  5. • ʮ؂ࢹʯʹΑΓ
    • ো֐Λະવʹ๷͙
    • ো֐ʹૉૣ͘ؾͮ͘
    • ো֐ͷݪҼಛఆͷώϯτ/MTTRͷ୹ॖ

    View Slide

  6. mackerel-plugin-axslog ͱ೿ੜ

    View Slide

  7. View Slide

  8. • mackerel-plugin-axslog
    • https://github.com/kazeburo/mackerel-plugin-axslog
    • mackerel-plugin-accesslog ͱಉ͘͡accesslogͷՄࢹԽ
    • status͝ͱͷ਺ͱׂ߹ɺresponse timeͷpercentile஋Λܭଌ
    • ूܭ͢ΔlabelΛมߋՄೳɾJSONαϙʔτ
    • ߴ଎

    View Slide

  9. 0
    2
    4
    6
    8
    accesslog(ltsv) axslog(ltsv) axslog(json)
    1.17s
    0.584s
    7.976s
    120ສߦ(20000req/sec)ͷϩάΛॲཧ͢Δ࣌ؒ

    View Slide

  10. • ߴ଎Խͷར఺
    • γεςϜෛՙΛؾʹͤͣɺʮΨϯΨϯʯϝτϦΫεΛ૿΍ͤΔ
    • ϑΟϧλػೳͷ௥Ճ

    View Slide

  11. [plugin.metrics.axslog-web-googlebot]
    command = "/usr/local/bin/mackerel-plugin-axslog --key-prefix=googlebot \

    --logfile /var/log/nginx/access.log --filter Googlebot"
    [plugin.metrics.axslog-web-adsbot]
    command = "/usr/local/bin/mackerel-plugin-axslog --key-prefix=adsbot \

    --logfile /var/log/nginx/access.log --filter AdsBot-Google"

    View Slide

  12. • mackerel-plugin-postfix-log
    • https://github.com/kazeburo/mackerel-plugin-postfix-log
    • SMTP
    • AxlogͰlogrotateΛ௥ै͢Δίʔυॻ͍ͨͷͰ͙͢Ͱ͖ͨ

    View Slide

  13. View Slide

  14. postfixlog.total_delay.average 0.240476 1555681849
    postfixlog.total_delay.99_percentile 0.250000 1555681849
    postfixlog.total_delay.95_percentile 0.240000 1555681849
    postfixlog.total_delay.90_percentile 0.240000 1555681849
    postfixlog.recving_delay.average 0.040000 1555681849
    postfixlog.recving_delay.99_percentile 0.040000 1555681849
    postfixlog.recving_delay.95_percentile 0.040000 1555681849
    postfixlog.recving_delay.90_percentile 0.040000 1555681849
    postfixlog.queuing_delay.average 0.000476 1555681849
    postfixlog.queuing_delay.99_percentile 0.010000 1555681849
    postfixlog.queuing_delay.95_percentile 0.000000 1555681849
    postfixlog.queuing_delay.90_percentile 0.000000 1555681849
    postfixlog.connection_delay.average 0.090000 1555681849
    postfixlog.connection_delay.99_percentile 0.090000 1555681849
    postfixlog.connection_delay.95_percentile 0.090000 1555681849
    postfixlog.connection_delay.90_percentile 0.090000 1555681849
    postfixlog.transmission_delay.average 0.090000 1555681849
    postfixlog.transmission_delay.99_percentile 0.090000 1555681849
    postfixlog.transmission_delay.95_percentile 0.090000 1555681849
    postfixlog.transmission_delay.90_percentile 0.090000 1555681849
    postfixlog.transfer_num.2xx_count 1.615385 1555681849
    postfixlog.transfer_num.4xx_count 0.000000 1555681849
    postfixlog.transfer_num.5xx_count 0.000000 1555681849
    postfixlog.transfer_total.count 1.615385 1555681849
    postfixlog.transfer_ratio.2xx_percentage 100.000000 1555681849
    postfixlog.transfer_ratio.4xx_percentage 0.000000 1555681849
    postfixlog.transfer_ratio.5xx_percentage 0.000000 1555681849
    ड৴࣌ͷ஗Ԇ
    Ωϡʔʹೖ͍ͬͯΔ࣌ؒ
    ૹ৴ઌ΁ͷ઀ଓʹ͔͔Δ࣌ؒ
    ૹ৴ͷ஗Ԇ

    View Slide

  15. • mackerel-plugin-log-incr-rate
    • https://github.com/kazeburo/mackerel-plugin-log-incr-rate
    • ͋Δϩάʹର͢Δผͷϩάͷߦ਺ͷ૿Ճׂ߹Λ؂ࢹ͢Δ
    • accesslogʹର͢Δapplication_log, errorlogͷ݅਺ͳͲ
    • application_log΁ͷॻ͖ࠐΈ͕രൃͯ͠ϩάܭଌʹӨڹͨ͠ো֐ʹର͢Δ؂ࢹ

    View Slide

  16. View Slide

  17. • mackerel-plugin-gzip-response
    • Not OSS
    • ϨεϙϯεͷGzipѹॖ͕ߦΘΕ͍ͯΔ͔؂ࢹ
    • Ұ෦ΫϥΠΞϯτͷ࣮૷ͰGzip͕ແޮʹͳͬͨ··ɺେ͖ͳ഑৴ͱͳΔϚε
    λʔߋ৽͕ߦΘΕɺωοτϫʔΫʹෛՙ͕͔͔Δࣄଶ͕͋ͬͨ
    • ϩάʹϨεϙϯεͷ Conent-Encoding ϔομΛه࿥͠ूܭ

    View Slide

  18. View Slide

  19. • Mackerel Plugin ࡞੒ύλʔϯ
    • ো֐/ো֐ʹܨ͕Δ໰୊ͷൃݟ
    • Metrics Plugin ͷ࡞੒ͱαʔό΁ͷ൓ө
    • ਺೔MetricsΛΈͯͷɺᮢ஋ઃఆɾਵ࣌ݟ௚͠

    View Slide

  20. DB ӡ༻ͱ؂ࢹ

    View Slide

  21. • ϗετͷεςʔλεΛ؂ࢹ͢Δ
    • ϗετεςʔλε Standby ͷDB(slave)αʔόΛ production ʹ౤ೖ
    • ϝϯςφϯε࡞ۀͷͨΊϨϓϦέʔγϣϯ͕ࢭ·͍͕ͬͯͨɺ௨஌͸དྷͳ͍

    View Slide

  22. App App
    consul
    consul DNS
    consul consul
    1) look up DNS
    2) connect DB
    check mysql uptime
    before
    slave slave slave

    View Slide

  23. App App
    consul
    consul DNS
    consul consul
    1) look up DNS
    2) connect DB
    A) check mysql uptime
    After
    Mackerel
    batch batch batch
    B) check host status file
    B’ ) update host status file by cron
    slave slave slave

    View Slide

  24. • Consul ͷαʔϏε͕༗ޮʹͳΔͨΊʹ͸ɺmackerelͷεςʔλεͷมߋ͕
    ඞཁɻΞϥʔτʹؾ͕ͭ͘Մೳੑˢ
    • ϗετεςʔλεΛϩʔΧϧͷͲ͔͜ʹॻ͖ࠐΉػೳɺ

    mackerel-agent ʹ͋Δͱศར?

    View Slide

  25. • Slave Ͱͷ Trigger ࣮ߦΛकΔ
    • ProductionͷҰ෦ͷslaveΛɺௐࠪɾूܭ༻ͱͯ͠࢖͑ΔΑ͏ʹ͍ͯ͠Δ
    • ηϯγςΟϒͳ৘ใʹ͍ͭͯ͸Slave্ͷTriggerͰ࡟আ

    View Slide

  26. • MySQLͱϨϓϦέʔγϣϯͱTrigger
    • MySQLͷϨϓϦέʔγϣϯͷํࣜʹ͸SQL͕సૹ͞Εͯ͘ΔStatementํࣜͱɺ
    σʔλ͕ૹΒΕͯ͘ΔRow Basedํࣜɺ·ͨͲͪΒ͔͕ࣗಈͰબ୒͞ΕΔMixed
    ํ͕ࣜ͋ΔɻMixedํࣜͰ͸جຊ͸Statementํ͕ࣜͩɺඇܾఆੑΫΤϦ͕࢖Θ
    ΕΔͱɺRow BasedͱͳΔ
    • ϨϓϦέʔγϣϯΛड͚Δslaveαʔό্ͰͷTrigger͸ɺStatementํࣜͷ৔߹ͷ
    Έ࣮ߦ͞ΕΔ
    • SlaveͰtriggerΛ҆શʹར༻͢ΔͨΊʹ͸ҙਤ͠ͳ͍ Row basedͳϨϓϦέʔ
    γϣϯΛൃݟ͢Δඞཁ͕͋Δ

    View Slide

  27. • check-mysqlbinlog-rbr-fallbacks
    • No OSS
    • mysqlbinlog ίϚϯυ࢖͍ɺRow BasedʹͳͬͨςʔϒϧΛݕग़

    View Slide

  28. Master
    Backup
    Anon
    Slave
    Trigger
    binlog
    binlog check-mysqlbinlog-rbr-fallbacks
    Detect the SQL and fix data and 

    applications if necessary
    App

    View Slide

  29. • ௅ઓ͍ͯ͠ΔݶΓɺ؂ࢹ͸૿͑Δ
    • Application Server: 175 Metrics
    • DB Server: 245 Metrics

    View Slide

  30. • Ҏ্

    View Slide