ログのメトリックを取ってみる話

logback / Prometheus

Transcript

1. ϩάͷϝτϦοΫΛऔͬͯΈΔ࿩
   @atty303 / AI Studio Dev Group
   © 2016 CyberAgent, Inc. 1
   

   View Slide

2. ELK stack ͱ͔Ͱϩάऩू͢
   Δ࿩͸Α͋͘Δ
   © 2016 CyberAgent, Inc. 2
   

   View Slide

3. ϩάͷϝτϦοΫΛܭଌ͢
   Δ࿩͸͋Μ·Γͳͦ͞͏?
   © 2016 CyberAgent, Inc. 3
   

   View Slide

4. ͜Μͳײ͡Ͱܭଌͯ͠·͢
   4 DEBUG Ϩϕϧ͔͠ग़͍ͯͳ͍ͷͰ͍·͍ͪ໘നΈ͸
   ͳ͍…
   © 2016 CyberAgent, Inc. 4
   

   View Slide

5. 4 ϞχλϦϯάʹ Prometheus Λར༻
   4 Prometheus ެࣜͷ Java ΫϥΠΞϯτ
   4 https://github.com/prometheus/
   client_java
   4 simpleclient_logback Ϟδϡʔϧ
   4 Ϩϕϧผʹϝοηʔδ਺Λܭଌͯ͠ϝτϦ
   οΫͱͯ͠ग़ྗͯ͘͠ΕΔ Logback ͷ
   Appender
   © 2016 CyberAgent, Inc. 5
   

   View Slide

6. logback.xml
   
   
   
   
   
   ${LOG_STDOUT_LEVEL:-INFO}
   
   
   %.-1level %d{MMdd HH:mm:ss.SSS} %thread [%logger{36}] %msg%n
   
   
   
   
   
   
   
   © 2016 CyberAgent, Inc. 6
   

   View Slide

7. ϙΠϯτ
   4 ϧʔτϩΨʔ͸શϩάΛ appender ʹૹΔ
   4 STDOUT appender ͷ ThreasholdFilter Ͱ
   ৑௕ͳϩάΛϑΟϧλϦϯά
   4 COUNTER appender ͕ϩάϨϕϧผͷϝο
   ηʔδ਺ΛΧ΢ϯτ
   © 2016 CyberAgent, Inc. 7
   

   View Slide

8. ϝτϦοΫܭଌͯ͘͠ΕΔ
   $ curl http://somewhere:58165/metrics
   # HELP logback_appender_total Log4j log statements at various log levels
   # TYPE logback_appender_total counter
   logback_appender_total{level="debug",} 8.9142634E7
   logback_appender_total{level="warn",} 0.0
   logback_appender_total{level="trace",} 0.0
   logback_appender_total{level="error",} 2728.0
   logback_appender_total{level="info",} 11.0
   © 2016 CyberAgent, Inc. 8
   

   View Slide

9. ϩάϨϕϧͷఆٛ
   DEBUG: ։ൃऀʹ͔͠ඞཁͳ͍৘ใ
   INFO: ΞϓϦέʔγϣϯͷॲཧঢ়گͳͲਖ਼ৗಈ࡞
   Λ͍ͯ͠Δ࣌ͷ৘ใ
   WARN: Τϥʔ͕ग़͕ͨճ෮ग़དྷͨ࣌ͷ৘ใ
   ERROR: Τϥʔ͕ग़ͯճ෮ग़དྷ͖ͳ͍࣌ͷ৘ใ
   4 ͪΌΜͱϩάϨϕϧͷҙຯ෇͚Λ͍ͯ͠ͳ͍ͱ
   ܭଌͯ͠΋ແҙຯ…
   © 2016 CyberAgent, Inc. 9
   

   View Slide

10. Ξϥʔτϧʔϧ
    IF rate(logback_appender_total{job="axion-inserter-admin",level="error"}[1m]) > 1
    FOR 1m
    LABELS { severity = "LOW" }
    ANNOTATIONS {
    summary = "Inserter: {{ $value }} error messages found at {{ $labels.instance }}"
    }
    4 ͱΓ͋͑ͣ ERROR ϩά͕ग़͍ͯͨΒΞϥʔ
    τ
    4 ϝοηʔδ͸ίϯςφΛݟʹߦ͘…
    nomad fs alloc/logs/inserter.stdout.0
    © 2016 CyberAgent, Inc. 10
    

    View Slide

11. …ͯͳӡ༻Λͯ͠·͢
    © 2016 CyberAgent, Inc. 11
    

    View Slide

12. ྑ͍ͱ͜Ζ
    4 ϩάऩूΑΓѹ౗తʹτϥϑΟοΫ͕খ͍͞ͷ
    ͰؾָʹಋೖͰ͖Δ
    4 ϩάϨϕϧ͚ͩͰ΋ҙ֎ͱগͳ͍࿑ྗͰ؂ࢹͰ
    ͖Δ
    4 ؂ࢹΛҙࣝͨ͠ϩάग़ྗ͸ඞཁ
    © 2016 CyberAgent, Inc. 12
    

    View Slide

13. ѱ͍ͱ͜Ζ
    4 ໰୊͕ى͖ͨͱ͖͸݁ہϝοηʔδΛݟʹߦ͔
    ͳ͍ͱ͍͚ͳ͍
    4 ϩάऩूͷ࢓૊Έ͕༻ҙ͞Ε͍ͯΕ͹ͦͷ΄͏
    ͕ྑ͍
    © 2016 CyberAgent, Inc. 13
    

    View Slide

14. GCP ͷ Stackdriver
    ͸͏Β΍·͍͠
    © 2016 CyberAgent, Inc. 14
    

    View Slide

15. ׬
    © 2016 CyberAgent, Inc. 15
    

    View Slide