Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ユーザ目線でのPrometheus #mackerel_ug /monitoring-prometheus

ユーザ目線でのPrometheus #mackerel_ug /monitoring-prometheus

モニタリング勉強会の発表資料です

Manabu Matsuzaki

October 27, 2017
Tweet

More Decks by Manabu Matsuzaki

Other Decks in Technology

Transcript

  1. Ϣʔβ໨ઢͰͷPrometheus
    ϞχλϦϯάษڧձ
    2017/10/27 @matsumana

    View full-size slide

  2. ࣗݾ঺հ
    • ໊લɿ দ࡚ ֶ
    • ॴଐɿ LINE Fukuokaגࣜձࣾ

    ։ൃࣨ
    • Twitterɿ @matsumana

    View full-size slide

  3. ΞδΣϯμ
    • ࣮ࡍʹͲ͏͍ͬͨϝτϦΫεΛऔ͍ͬͯΔͷ͔
    • exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε
    • Ξϥʔτϧʔϧͷॻ͖ํͷྫ
    • ·ͱΊ
    ※࣌ؒͷ౎߹্ɺPrometheusೖ໳తͳࣄ͸࿩͠·ͤΜ

    View full-size slide

  4. લఏ
    • ݱࡏ࢖༻͍ͯ͠Δόʔδϣϯ
    • Prometheus 1.8.0

    View full-size slide

  5. ݱࡏ࢖͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε

    ʢҰ෦ͷΈൈਮʣ
    • node_exporter
    • loading
    • CPU࢖༻཰
    • ϝϞϦ࢖༻཰
    • εϫοϓ
    • slab
    • context switch

    View full-size slide

  6. ݱࡏ࢖͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε

    ʢҰ෦ͷΈൈਮʣ
    • node_exporter
    • σΟεΫI/O
    • σΟεΫ࢖༻཰
    • inode਺
    • TCP઀ଓ
    • ESTABLISHED
    • TIME_WAIT
    • ωοτϫʔΫ τϥϑΟοΫ

    View full-size slide

  7. ݱࡏ࢖͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε

    ʢҰ෦ͷΈൈਮʣ
    • apache_exporter
    • mod_statusϞδϡʔϧͰऔΕΔϝτϦΫε
    • nginx_exporter
    • ngx_http_stub_status_moduleϞδϡʔϧͰऔΕΔϝτϦΫε

    View full-size slide

  8. ݱࡏ࢖͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε

    ʢҰ෦ͷΈൈਮʣ
    • plack_exporter
    • Plack::Middleware::ServerStatus::LiteͰऔΕΔϝτϦΫε

    View full-size slide

  9. ݱࡏ࢖͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε

    ʢҰ෦ͷΈൈਮʣ
    • jmx_exporter
    • JVMͷڞ௨తͳϝτϦΫε
    • Heap
    • Metaspace
    • GC
    • Thread਺
    • Code cache

    View full-size slide

  10. ݱࡏ࢖͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε

    ʢҰ෦ͷΈൈਮʣ
    • jmx_exporter
    • ϛυϧ΢ΣΞ/ϥΠϒϥϦݻ༗ͷϝτϦΫε
    • Tomcat
    • Jetty
    • HikariCP
    • Caffeine
    • etc…

    View full-size slide

  11. ݱࡏ࢖͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε

    ʢҰ෦ͷΈൈਮʣ
    • memcached_exporter
    • ΩϟογϡαΠζ
    • Ωϟογϡ਺
    • Ωϟογϡώοτ཰
    • operation(get/set)
    • Eviction

    View full-size slide

  12. ݱࡏ࢖͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε

    ʢҰ෦ͷΈൈਮʣ
    • mysqld_exporter
    • QPS
    • Slow query
    • Replicaton delay
    • binlog
    • connection਺
    • InnoDBؔ࿈

    View full-size slide

  13. ݱࡏ࢖͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε

    ʢҰ෦ͷΈൈਮʣ
    • redis_exporter
    • ࢖༻ϝϞϦ
    • Key਺
    • Eviction
    • Slow query

    View full-size slide

  14. ݱࡏ࢖͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε

    ʢҰ෦ͷΈൈਮʣ
    • elasticsearch_exporter
    • Cluster health
    • Document਺(primary shardͷΈ)
    • DocumentαΠζ(primary shardͷΈ)
    • Thread pool queue਺
    • Thread pool rejected਺
    • Circuit Breaker tripped਺

    View full-size slide

  15. ݱࡏ࢖͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε

    ʢҰ෦ͷΈൈਮʣ
    • elasticsearch_exporter
    • Fielddata cache size
    • Query cache size
    • Query cache hit,miss,total
    • Request cache size

    View full-size slide

  16. ݱࡏ࢖͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε

    ʢҰ෦ͷΈൈਮʣ
    • fluent-plugin-prometheus
    • ֤BufferedOutput pluginͷϝτϦΫε
    • buffer_queue_length
    • buffer_total_bytes
    • retry_count

    View full-size slide

  17. ݱࡏ࢖͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε

    ʢҰ෦ͷΈൈਮʣ
    • fluentd_exporter
    • fluentdϓϩηε୯ҐͷCPU࢖༻཰
    • fluentdϓϩηε୯ҐͷϝϞϦ࢖༻཰
    • td-agent_exporter
    • fluentdϓϩηε୯ҐͷCPU࢖༻཰
    • fluentdϓϩηε୯ҐͷϝϞϦ࢖༻཰
    • fluent-agent-lite_exporter
    • fluentdϓϩηε୯ҐͷCPU࢖༻཰
    • fluentdϓϩηε୯ҐͷϝϞϦ࢖༻཰

    View full-size slide

  18. exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε
    • https://prometheus.io/docs/instrumenting/exporters/
    • https://github.com/prometheus/prometheus/wiki/Default-
    port-allocations
    • άάΔ
    • ࣗ෼͕ඞཁͳϝτϦΫε͕export͞Ε͍ͯͳΕ͹PR͢Δ
    ·ͣ͸ɺଞͷਓ͕࡞ͬͨ΋ͷ͕ແ͍͔୳ͯ͠ΈΔ

    View full-size slide

  19. • ΦϑΟγϟϧͷΨΠυϥΠϯ

    https://prometheus.io/docs/instrumenting/writing_exporters/
    exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε
    ࣗ෼Ͱ࡞Δ৔߹

    View full-size slide

  20. • ΦϑΟγϟϧͷclient library

    https://prometheus.io/docs/instrumenting/clientlibs/
    • ΦϑΟγϟϧ͸golang, Java, Python, RubyͷΈ
    • Third partyʹ͸PHPͳͲ΋͋Δ
    exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε
    ࣗ෼Ͱ࡞Δ৔߹

    View full-size slide

  21. • exporter͸goͰॻ͘ͷ͕ྑͦ͞͏͚ͩͲ஫ҙ఺΋͋Δ
    • Linux Kernel 2.6.23ະຬ͸golangࣗମ͕αϙʔτͯ͠ͳ͍

    CentOS5ܥͩͱgolang੡ͷexporter͸ਖ਼͘͠ಈ͔ͳ͍͔΋

    https://github.com/golang/go/wiki/MinimumRequirements
    exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε
    ࣗ෼Ͱ࡞Δ৔߹

    View full-size slide

  22. exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε
    ΦϑΟγϟϧΨΠυϥΠϯʹهࡌ͞Ε͍ͯΔ΋ͷΛҰ෦͝঺հ
    • ଞͷexporterͰ࢖ΘΕͯͳ͍portΛ࢖͏ɻ࡞ͬͨΒهࡌ͠Α͏

    https://github.com/prometheus/prometheus/wiki/Default-port-
    allocations
    • ϝτϦΫε໊ɺϥϕϧ໊ͷϕετϓϥΫςΟεʹै͏

    https://prometheus.io/docs/practices/naming/
    • exporterͰϝτϦΫε஋Λܭࢉ͠ͳ͍ɻPromQLͰܭࢉ͢Δ
    • ྫʣΩϟογϡώοτ཰ͳͲ

    View full-size slide

  23. exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε
    • exporterͷόʔδϣϯ൪߸ΛϝτϦΫεͱͯ͠export͢Δ
    • ؂ࢹର৅ͷαʔό΍exporterͷ਺͕૿͑ΔͱɺͲͷόʔδϣ
    ϯ͕σϓϩΠ͞Ε͍ͯΔ͔Θ͔Βͳ͘ͳΔ
    • ͪͳΈʹɺPrometheusຊମ͸ҎԼͷΑ͏ͳϝτϦΫεΛ
    export͍ͯ͠Δ

    prometheus_build_info{branch=“master",goversion="go1.8.3",revision="3afb3fffa3a29
    c3de865e1172fb740442e9d0133",version="1.7.1"} 1
    ΦϑΟγϟϧΨΠυϥΠϯʹهࡌ͞Ε͍ͯΔ΋ͷΛҰ෦͝঺հ

    View full-size slide

  24. exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε
    ΦϑΟγϟϧΨΠυϥΠϯʹهࡌ͞Ε͍ͯΔ΋ͷΛҰ෦͝঺հ
    • ̋̋̋_up ϝτϦΫεΛexport͢Δ
    • ྫ͑͹ɺMySQLͷ৔߹͸mysql_upͱ͍͏ϝτϦΫε
    • ؂ࢹର৅͔ΒϝτϦΫεऔಘ੒ޭͳΒ1ɺࣦഊͳΒ0Λexport
    • ཧ༝͸ޙड़

    View full-size slide

  25. Ξϥʔτϧʔϧͷॻ͖ํͷྫ
    • ྫ1) exporter͕མ͍ͪͯͳ͍͔Ͳ͏͔؂ࢹ͢Δ




    • ྫ2) exporertͷ؂ࢹର৅͕མ͍ͪͯͳ͍͔Ͳ͏͔؂ࢹ͢Δ



    • ྫ3) dailyόον͕ਖ਼ৗʹಈ͍͍ͯΔ͔Ͳ͏͔؂ࢹ͢Δ

    View full-size slide

  26. Ξϥʔτϧʔϧͷॻ͖ํͷྫ
    • Prometheusαʔόࣗ਎͕ɺશͯͷexporterʹ͍ͭͯ

    upϝτϦΫεΛه࿥͍ͯ͠Δ

    https://prometheus.io/docs/concepts/jobs_instances/


    ྫʣnode_exporterͷ৔߹
    up{instance="host00:9100", job="node"} 1

    ྫ1ʣexporter͕མ͍ͪͯͳ͍͔Ͳ͏͔؂ࢹ͢Δ

    View full-size slide

  27. Ξϥʔτϧʔϧͷॻ͖ํͷྫ
    ALERT ExporterDown
    IF up == 0
    FOR 1m
    LABELS {severity="major"}
    ANNOTATIONS {
    description="{{ $labels.instance }} of job {{ $labels.job }} has been down”
    }
    ྫ1ʣexporter͕མ͍ͪͯͳ͍͔Ͳ͏͔؂ࢹ͢Δ
    • ͳͷͰɺҎԼͷ༷ͳΞϥʔτϧʔϧΛ1ͭొ࿥͢Ε͹

    શͯͷexporterʹରԠͰ͖Δ

    View full-size slide

  28. Ξϥʔτϧʔϧͷॻ͖ํͷྫ
    ྫ̎ʣexporertͷ؂ࢹର৅͕མ͍ͪͯͳ͍͔Ͳ͏͔؂ࢹ͢Δ
    • ҎԼͷ༷ͳΞϥʔτϧʔϧΛ1ͭొ࿥͢Ε͹ɺ

    ̋̋̋_upϝτϦΫεΛ࣋ͭશͯͷexporterʹରԠͰ͖Δ
    ALERT ExporterTargetDown
    IF {__name__=~".*_up"} == 0
    FOR 1m
    LABELS {severity="major"}
    ANNOTATIONS {
    summary="Exporter target ( {{ $labels.job }} ) down"
    }

    View full-size slide

  29. Ξϥʔτϧʔϧͷॻ͖ํͷྫ
    • node_exporterͷTextfile CollectorػೳΛ࢖ͬͯϝτϦΫεΛexport͢Δ

    (pushgatewayΛ࢖ͬͯ΋ϝτϦΫε͸exportग़དྷΔ)
    • ͜Μͳײ͡ͷshellΛjobεέδϡʔϥͰ࣮ߦ͢ΔΑ͏ʹͯ͠ɺ
    ྫ̏ʣdailyόον͕ਖ਼ৗʹಈ͍͍ͯΔ͔Ͳ͏͔؂ࢹ͢Δ
    #!/bin/bash
    # ్தͰΤϥʔ͕ൃੜͨ͠Βऴྃ͢Δ
    set -e
    UNIXTIME=$(date +%s)
    RESULT=/path/to/node_exporter_textfile_directory/my_batch_job.prom
    # execute job
    /path/to/my_batch_job.sh
    # write metrics
    echo my_batch_job_completion_time{cron="my_batch_job", period="daily"} $UNIXTIME > $RESULT.tmp
    mv $RESULT.tmp $RESULT

    View full-size slide

  30. Ξϥʔτϧʔϧͷॻ͖ํͷྫ
    • ҎԼͷ༷ͳΞϥʔτϧʔϧΛొ࿥͢Ε͹ݕ஌Մೳ
    ྫ̏ʣdailyόον͕ਖ਼ৗʹಈ͍͍ͯΔ͔Ͳ͏͔؂ࢹ͢Δ
    ALERT DailyCronLate
    IF time() - {period="daily"} > 60 * 60 * 24
    FOR 5m
    LABELS {severity="warning"}
    ANNOTATIONS {
    summary="Daily cronjob for {{ $labels.cron }} has not run in 24 hours”
    }

    View full-size slide

  31. ·ͱΊ
    • ࢖͍ͬͯΔexporterͱɺऔಘग़དྷΔϝτϦΫεͷҰ෦Λ͝঺հ
    ͠·ͨ͠
    • exporterΛ࡞Δ৔߹ͷ஫ҙ఺(golang)ʹ͍ͭͯ͝঺հ͠·ͨ͠
    • exporterΛ࡞Δ৔߹ͷϕετϓϥΫςΟεΛ͍͔ͭ͘ൈਮͯ͠
    ͝঺հ͠·ͨ͠
    • ΞϥʔτϧʔϧͷྫΛ͍͔ͭ͘͝঺հ͠·ͨ͠

    View full-size slide

  32. Thank you :)

    View full-size slide