Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

ログのメトリックを取ってみる話

AGAWA Koji
November 04, 2016

 ログのメトリックを取ってみる話

logback / Prometheus

AGAWA Koji

November 04, 2016
Tweet

More Decks by AGAWA Koji

Other Decks in Programming

Transcript

  1. 4 ϞχλϦϯάʹ Prometheus Λར༻ 4 Prometheus ެࣜͷ Java ΫϥΠΞϯτ 4

    https://github.com/prometheus/ client_java 4 simpleclient_logback Ϟδϡʔϧ 4 Ϩϕϧผʹϝοηʔδ਺Λܭଌͯ͠ϝτϦ οΫͱͯ͠ग़ྗͯ͘͠ΕΔ Logback ͷ Appender © 2016 CyberAgent, Inc. 5
  2. logback.xml <configuration> <appender name="COUNTER" class="io.prometheus.client.logback.InstrumentedAppender"> </appender> <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender"> <filter

    class="ch.qos.logback.classic.filter.ThresholdFilter"> <level>${LOG_STDOUT_LEVEL:-INFO}</level> </filter> <encoder> <pattern>%.-1level %d{MMdd HH:mm:ss.SSS} %thread [%logger{36}] %msg%n</pattern> </encoder> </appender> <root level="DEBUG"> <appender-ref ref="COUNTER" /> <appender-ref ref="STDOUT" /> </root> </configuration> © 2016 CyberAgent, Inc. 6
  3. ϙΠϯτ 4 ϧʔτϩΨʔ͸શϩάΛ appender ʹૹΔ 4 STDOUT appender ͷ ThreasholdFilter

    Ͱ ৑௕ͳϩάΛϑΟϧλϦϯά 4 COUNTER appender ͕ϩάϨϕϧผͷϝο ηʔδ਺ΛΧ΢ϯτ © 2016 CyberAgent, Inc. 7
  4. ϝτϦοΫܭଌͯ͘͠ΕΔ $ curl http://somewhere:58165/metrics # HELP logback_appender_total Log4j log statements

    at various log levels # TYPE logback_appender_total counter logback_appender_total{level="debug",} 8.9142634E7 logback_appender_total{level="warn",} 0.0 logback_appender_total{level="trace",} 0.0 logback_appender_total{level="error",} 2728.0 logback_appender_total{level="info",} 11.0 © 2016 CyberAgent, Inc. 8
  5. Ξϥʔτϧʔϧ IF rate(logback_appender_total{job="axion-inserter-admin",level="error"}[1m]) > 1 FOR 1m LABELS { severity

    = "LOW" } ANNOTATIONS { summary = "Inserter: {{ $value }} error messages found at {{ $labels.instance }}" } 4 ͱΓ͋͑ͣ ERROR ϩά͕ग़͍ͯͨΒΞϥʔ τ 4 ϝοηʔδ͸ίϯςφΛݟʹߦ͘… nomad fs <alloc-id> alloc/logs/inserter.stdout.0 © 2016 CyberAgent, Inc. 10