Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ログのメトリックを取ってみる話

AGAWA Koji
November 04, 2016

 ログのメトリックを取ってみる話

logback / Prometheus

AGAWA Koji

November 04, 2016
Tweet

More Decks by AGAWA Koji

Other Decks in Programming

Transcript

  1. ϩάͷϝτϦοΫΛऔͬͯΈΔ࿩ @atty303 / AI Studio Dev Group © 2016 CyberAgent,

    Inc. 1
  2. ELK stack ͱ͔Ͱϩάऩू͢ Δ࿩͸Α͋͘Δ © 2016 CyberAgent, Inc. 2

  3. ϩάͷϝτϦοΫΛܭଌ͢ Δ࿩͸͋Μ·Γͳͦ͞͏? © 2016 CyberAgent, Inc. 3

  4. ͜Μͳײ͡Ͱܭଌͯ͠·͢ 4 DEBUG Ϩϕϧ͔͠ग़͍ͯͳ͍ͷͰ͍·͍ͪ໘നΈ͸ ͳ͍… © 2016 CyberAgent, Inc. 4

  5. 4 ϞχλϦϯάʹ Prometheus Λར༻ 4 Prometheus ެࣜͷ Java ΫϥΠΞϯτ 4

    https://github.com/prometheus/ client_java 4 simpleclient_logback Ϟδϡʔϧ 4 Ϩϕϧผʹϝοηʔδ਺Λܭଌͯ͠ϝτϦ οΫͱͯ͠ग़ྗͯ͘͠ΕΔ Logback ͷ Appender © 2016 CyberAgent, Inc. 5
  6. logback.xml <configuration> <appender name="COUNTER" class="io.prometheus.client.logback.InstrumentedAppender"> </appender> <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender"> <filter

    class="ch.qos.logback.classic.filter.ThresholdFilter"> <level>${LOG_STDOUT_LEVEL:-INFO}</level> </filter> <encoder> <pattern>%.-1level %d{MMdd HH:mm:ss.SSS} %thread [%logger{36}] %msg%n</pattern> </encoder> </appender> <root level="DEBUG"> <appender-ref ref="COUNTER" /> <appender-ref ref="STDOUT" /> </root> </configuration> © 2016 CyberAgent, Inc. 6
  7. ϙΠϯτ 4 ϧʔτϩΨʔ͸શϩάΛ appender ʹૹΔ 4 STDOUT appender ͷ ThreasholdFilter

    Ͱ ৑௕ͳϩάΛϑΟϧλϦϯά 4 COUNTER appender ͕ϩάϨϕϧผͷϝο ηʔδ਺ΛΧ΢ϯτ © 2016 CyberAgent, Inc. 7
  8. ϝτϦοΫܭଌͯ͘͠ΕΔ $ curl http://somewhere:58165/metrics # HELP logback_appender_total Log4j log statements

    at various log levels # TYPE logback_appender_total counter logback_appender_total{level="debug",} 8.9142634E7 logback_appender_total{level="warn",} 0.0 logback_appender_total{level="trace",} 0.0 logback_appender_total{level="error",} 2728.0 logback_appender_total{level="info",} 11.0 © 2016 CyberAgent, Inc. 8
  9. ϩάϨϕϧͷఆٛ DEBUG: ։ൃऀʹ͔͠ඞཁͳ͍৘ใ INFO: ΞϓϦέʔγϣϯͷॲཧঢ়گͳͲਖ਼ৗಈ࡞ Λ͍ͯ͠Δ࣌ͷ৘ใ WARN: Τϥʔ͕ग़͕ͨճ෮ग़དྷͨ࣌ͷ৘ใ ERROR: Τϥʔ͕ग़ͯճ෮ग़དྷ͖ͳ͍࣌ͷ৘ใ

    4 ͪΌΜͱϩάϨϕϧͷҙຯ෇͚Λ͍ͯ͠ͳ͍ͱ ܭଌͯ͠΋ແҙຯ… © 2016 CyberAgent, Inc. 9
  10. Ξϥʔτϧʔϧ IF rate(logback_appender_total{job="axion-inserter-admin",level="error"}[1m]) > 1 FOR 1m LABELS { severity

    = "LOW" } ANNOTATIONS { summary = "Inserter: {{ $value }} error messages found at {{ $labels.instance }}" } 4 ͱΓ͋͑ͣ ERROR ϩά͕ग़͍ͯͨΒΞϥʔ τ 4 ϝοηʔδ͸ίϯςφΛݟʹߦ͘… nomad fs <alloc-id> alloc/logs/inserter.stdout.0 © 2016 CyberAgent, Inc. 10
  11. …ͯͳӡ༻Λͯ͠·͢ © 2016 CyberAgent, Inc. 11

  12. ྑ͍ͱ͜Ζ 4 ϩάऩूΑΓѹ౗తʹτϥϑΟοΫ͕খ͍͞ͷ ͰؾָʹಋೖͰ͖Δ 4 ϩάϨϕϧ͚ͩͰ΋ҙ֎ͱগͳ͍࿑ྗͰ؂ࢹͰ ͖Δ 4 ؂ࢹΛҙࣝͨ͠ϩάग़ྗ͸ඞཁ ©

    2016 CyberAgent, Inc. 12
  13. ѱ͍ͱ͜Ζ 4 ໰୊͕ى͖ͨͱ͖͸݁ہϝοηʔδΛݟʹߦ͔ ͳ͍ͱ͍͚ͳ͍ 4 ϩάऩूͷ࢓૊Έ͕༻ҙ͞Ε͍ͯΕ͹ͦͷ΄͏ ͕ྑ͍ © 2016 CyberAgent,

    Inc. 13
  14. GCP ͷ Stackdriver ͸͏Β΍·͍͠ © 2016 CyberAgent, Inc. 14

  15. ׬ © 2016 CyberAgent, Inc. 15