ϩάͷϝτϦοΫΛऔͬͯΈΔ࿩ @atty303 / AI Studio Dev Group © 2016 CyberAgent, Inc. 1

ELK stack ͱ͔Ͱϩάऩू͢ Δ࿩͸Α͋͘Δ © 2016 CyberAgent, Inc. 2

ϩάͷϝτϦοΫΛܭଌ͢ Δ࿩͸͋Μ·Γͳͦ͞͏? © 2016 CyberAgent, Inc. 3

͜Μͳײ͡Ͱܭଌͯ͠·͢ 4 DEBUG Ϩϕϧ͔͠ग़͍ͯͳ͍ͷͰ͍·͍ͪ໘നΈ͸ ͳ͍… © 2016 CyberAgent, Inc. 4

4 ϞχλϦϯάʹ Prometheus Λར༻ 4 Prometheus ެࣜͷ Java ΫϥΠΞϯτ 4 https://github.com/prometheus/ client_java 4 simpleclient_logback Ϟδϡʔϧ 4 Ϩϕϧผʹϝοηʔδ਺Λܭଌͯ͠ϝτϦ οΫͱͯ͠ग़ྗͯ͘͠ΕΔ Logback ͷ Appender © 2016 CyberAgent, Inc. 5

logback.xml ${LOG_STDOUT_LEVEL:-INFO} %.-1level %d{MMdd HH:mm:ss.SSS} %thread [%logger{36}] %msg%n © 2016 CyberAgent, Inc. 6

ϙΠϯτ 4 ϧʔτϩΨʔ͸શϩάΛ appender ʹૹΔ 4 STDOUT appender ͷ ThreasholdFilter Ͱ ৑௕ͳϩάΛϑΟϧλϦϯά 4 COUNTER appender ͕ϩάϨϕϧผͷϝο ηʔδ਺ΛΧ΢ϯτ © 2016 CyberAgent, Inc. 7

ϝτϦοΫܭଌͯ͘͠ΕΔ $ curl http://somewhere:58165/metrics # HELP logback_appender_total Log4j log statements at various log levels # TYPE logback_appender_total counter logback_appender_total{level="debug",} 8.9142634E7 logback_appender_total{level="warn",} 0.0 logback_appender_total{level="trace",} 0.0 logback_appender_total{level="error",} 2728.0 logback_appender_total{level="info",} 11.0 © 2016 CyberAgent, Inc. 8

ϩάϨϕϧͷఆٛ DEBUG: ։ൃऀʹ͔͠ඞཁͳ͍৘ใ INFO: ΞϓϦέʔγϣϯͷॲཧঢ়گͳͲਖ਼ৗಈ࡞ Λ͍ͯ͠Δ࣌ͷ৘ใ WARN: Τϥʔ͕ग़͕ͨճ෮ग़དྷͨ࣌ͷ৘ใ ERROR: Τϥʔ͕ग़ͯճ෮ग़དྷ͖ͳ͍࣌ͷ৘ใ 4 ͪΌΜͱϩάϨϕϧͷҙຯ෇͚Λ͍ͯ͠ͳ͍ͱ ܭଌͯ͠΋ແҙຯ… © 2016 CyberAgent, Inc. 9

Ξϥʔτϧʔϧ IF rate(logback_appender_total{job="axion-inserter-admin",level="error"}[1m]) > 1 FOR 1m LABELS { severity = "LOW" } ANNOTATIONS { summary = "Inserter: {{ $value }} error messages found at {{ $labels.instance }}" } 4 ͱΓ͋͑ͣ ERROR ϩά͕ग़͍ͯͨΒΞϥʔ τ 4 ϝοηʔδ͸ίϯςφΛݟʹߦ͘… nomad fs alloc/logs/inserter.stdout.0 © 2016 CyberAgent, Inc. 10

…ͯͳӡ༻Λͯ͠·͢ © 2016 CyberAgent, Inc. 11

ྑ͍ͱ͜Ζ 4 ϩάऩूΑΓѹ౗తʹτϥϑΟοΫ͕খ͍͞ͷ ͰؾָʹಋೖͰ͖Δ 4 ϩάϨϕϧ͚ͩͰ΋ҙ֎ͱগͳ͍࿑ྗͰ؂ࢹͰ ͖Δ 4 ؂ࢹΛҙࣝͨ͠ϩάग़ྗ͸ඞཁ © 2016 CyberAgent, Inc. 12

ѱ͍ͱ͜Ζ 4 ໰୊͕ى͖ͨͱ͖͸݁ہϝοηʔδΛݟʹߦ͔ ͳ͍ͱ͍͚ͳ͍ 4 ϩάऩूͷ࢓૊Έ͕༻ҙ͞Ε͍ͯΕ͹ͦͷ΄͏ ͕ྑ͍ © 2016 CyberAgent, Inc. 13

GCP ͷ Stackdriver ͸͏Β΍·͍͠ © 2016 CyberAgent, Inc. 14

׬ © 2016 CyberAgent, Inc. 15