Upgrade to Pro — share decks privately, control downloads, hide ads and more …

コンテナのメトリクスと モニタリングパターン / 190320-sakura-event

コンテナのメトリクスと モニタリングパターン / 190320-sakura-event

Hayato Imai

March 20, 2019
Tweet

More Decks by Hayato Imai

Other Decks in Technology

Transcript

  1. ίϯςφͷϝτϦΫεͱ
    ϞχλϦϯάύλʔϯ
    2019/03/20
    ͸ͯͳ×͘͞Β͕ߟ͑ΔςΫϊϩδʔͷະདྷ ʙίϯςφɾ෼ࢄܕσʔληϯλʔʙ
    #͘͞ΒͷΠϕϯτ

    View Slide

  2. ࣗݾ঺հ
    • ࠓҪ൏ਓ(id:hayajo_77)
    • MackerelνʔϜ SRE
    • ίϯςφཁૉٕज़ɺपลٕज़
    ίϯςφٕज़ೖ໳Ծ૝Խͱͷҧ͍Λ஌Γɺཁૉٕज़Λ৮ֶͬͯ΅͏
    IUUQTFNQMPZNFOUFOKBQBODPNFOHJOFFSIVCFOUSZ

    View Slide

  3. ؂ࢹͱϝτϦΫε

    View Slide

  4. ؂ࢹͷ໨త
    • γεςϜ͕݈શʹಈ͍͍ͯΔ͔ΛνΣοΫ͢Δ
    • ظ଴ͨ͠ͱ͓Γʹಈ͍͍ͯΔ͔
    • Ձ஋ΛੜΈग़͍ͯ͠Δ͔
    • Ϧιʔεʹաෆ଍͸ͳ͍͔
    • ো֐ͷ༧ஹ͸ͳ͍͔
    • →ϝτϦΫεͷऔಘ/ऩूɺ؂ࢹɺΞϥʔτͰ࣮ݱ͢Δ

    View Slide

  5. 2ͭͷλΠϓͷϝτϦΫε
    • 2ͭͷλΠϓͷϝτϦΫεʹ෼͚ͯߟ͑Δ
    • ΞϓϦέʔγϣϯϝτϦΫε
    • ϦιʔεϝτϦΫε

    View Slide

  6. ΞϓϦέʔγϣϯϝτϦΫε
    • γεςϜ͕ਖ਼ৗʹಈ͍ͯΔ౓߹͍Λࣔ͢
    • ϦΫΤετ਺ɺ੒ޭ/ࣦഊɺϨεϙϯελΠϜɺεϧʔϓοτͳͲ
    • γεςϜͷਖ਼ৗੑ΍Ձ஋ͷࢦඪͱͳΔ
    • ͜ͷϝτϦΫε͸ো֐ݪҼڀ໌ʹ͸໾ཱͪʹ͍͘

    View Slide

  7. ϦιʔεϝτϦΫε
    • γεςϜ͕ґଘ͢ΔϦιʔεͷ࢖༻ঢ়گΛࣔ͢
    • Ϧιʔε࢖༻ྔ/཰ɺ๞࿨཰ɺΤϥʔ਺ͳͲ
    • ো֐ݪҼڀ໌΍ΩϟύγςΟϓϥϯχϯάͷࢦඪͱͳΔ
    • ͜ͷϝτϦΫε͚ͩͰ͸γεςϜ͕ਖ਼ৗʹಈ͍͍ͯΔ͔෼͔Βͳ͍

    View Slide

  8. ͲͪΒ΋ॏཁ
    • ΞϓϦέʔγϣϯϝτϦΫεˠϦιʔεϝτϦΫεͷॱʹௐࠪ͢Δ
    • ଞͷΞϓϦέʔγϣϯʹґଘ͍ͯ͠Δ৔߹͸ಉ͡Α͏ʹௐࠪ͢Δ
    • ΢ΣϒαʔόɺΞϓϦέʔγϣϯαʔόɺσʔλϕʔεαʔό

    View Slide

  9. ίϯςφͷϝτϦΫε

    View Slide

  10. ίϯςφͷΞϓϦέʔγϣϯϝτϦΫε
    • ΞϓϦέʔγϣϯʹΑͬͯऔಘͰ͖ΔϝτϦΫε͸͞·͟·
    • ϦΫΤετ਺ɺ੒ޭ/ࣦഊɺϨεϙϯελΠϜɺεϧʔϓοτͳͲ

    View Slide

  11. ίϯςφͷΞϓϦέʔγϣϯϝτϦΫε
    औಘํ๏
    • ϑΝΠϧ
    • ϩά
    • ΤϯυϙΠϯτ

    View Slide

  12. ϑΝΠϧ
    • σʔλϘϦϡʔϜ
    • ίϯςφؒͰϑΝΠϧ΍σΟϨΫτϦΛڞ༗
    • Bind mounts
    • ίϯςφϗετͱϑΝΠϧ΍σΟϨΫτϦΛڞ༗

    View Slide

  13. $ docker run -v $(pwd):/data alpine /bin/sh -c
    'echo "hello world" >/data/hello'
    $ cat hello
    hello world
    e.g. Bind mounts

    View Slide

  14. ϩά
    • ΞϓϦέʔγϣϯͷSTDOUT
    • docker logs ίϚϯυ
    • kubectl logs ίϚϯυ
    • ίϯςφ಺ͷϩάϑΝΠϧ
    • σʔλϘϦϡʔϜ, Bind mountsΛར༻

    View Slide

  15. $ CONTAINER=$(docker run -d alpine /bin/sh -c
    'while true; do date; sleep 1; done')
    $ docker logs --since 3s $CONTAINER
    Tue Mar 19 10:07:57 UTC 2019
    Tue Mar 19 10:07:58 UTC 2019
    Tue Mar 19 10:07:59 UTC 2019
    e.g. docker logs

    View Slide

  16. ΤϯυϙΠϯτ
    • ωοτϫʔΫܦ༝ͰΞΫηεՄೳͳΤϯυϙΠϯτ
    • ΞϓϦέʔγϣϯͰ࣮૷͢Δඞཁ͕͋Δ
    • /metrics, /stats, /health

    View Slide

  17. $ curl http://${CONTAINER_IPADDR}/nginx_status
    Active connections: 43
    server accepts handled requests
    7368 7368 10993
    Reading: 0 Writing: 5 Waiting: 38
    e.g. Nginxͷstub_status

    View Slide

  18. ίϯςφͷϦιʔεϝτϦΫε
    • CPU
    • Memory
    • I/O
    • ωοτϫʔΫ

    View Slide

  19. ίϯςφͷϦιʔεϝτϦΫεऔಘํ๏
    • Ծ૝ϑΝΠϧ
    • API
    • CLI

    View Slide

  20. Ծ૝ϑΝΠϧ
    • cgroupͷstatϑΝΠϧ
    • cpu/cpuacct, memory, blkioαϒγεςϜͳͲ
    • /proc/PID/net/dev

    View Slide

  21. $ cat /sys/fs/cgroup/memory/docker/${CONTAINER}/memory.stat
    cache 8192
    rss 192512
    rss_huge 0
    mapped_file 0
    dirty 0
    [...]
    e.g. memory.stat

    View Slide

  22. API
    • ίϯςφϥϯλΠϜ΍ίϯςφΫϥελͷAPIͳͲ
    • Docker stats API (Docker)
    • Kubernetes/kubelet API (Kubernetes)
    • Task Metadata Endpoint (ECS)
    • cAdvisor API

    View Slide

  23. $ curl -s --unix-socket \
    /var/run/docker.sock http://localhost/${CONTAINER}/stats?stream=false | \
    jq .
    [...]
    "cpu_stats": {
    "cpu_usage": {
    "total_usage": 963481062,
    "percpu_usage": [
    431767249,
    531713813
    ],
    [...]
    e.g. Docker stats API

    View Slide

  24. CLI
    • ίϯςφϥϯλΠϜ΍ίϯςφΫϥελ͕ఏڙ͢ΔίϚϯυ
    • docker stats ίϚϯυ
    • kubectl top ίϚϯυ

    View Slide

  25. $ docker stats --no-stream
    CONTAINER ID NAME CPU %
    MEM USAGE / LIMIT MEM % NET
    I/O BLOCK I/O PIDS
    d7f9c5c0acfc app 0.00%
    1.945MiB / 1.952GiB 0.10%
    1.5kB / 0B 5.11MB / 0B 2
    8c484a94aca7 monitor 0.19%
    652KiB / 1.952GiB 0.03%
    1.41kB / 0B 1.24MB / 0B 2
    e.g. docker stats

    View Slide

  26. ωοτϫʔΫϝτϦΫεͷ஫ҙ఺
    • /proc/PID/net/dev
    • ΠϯλʔϑΣʔε୯ҐͷϝτϦΫε

    View Slide

  27. $ cat /proc/${CONTAINER_PID}/net/dev
    Inter-| Receive
    | Transmit
    face |bytes packets errs drop fifo frame compressed
    multicast|bytes packets errs drop fifo colls carrier
    compressed
    lo: 0 0 0 0 0 0 0
    0 0 0 0 0 0 0 0 0
    ecs-eth0: 467809174 273650 0 0 0 0 0
    0 27453834 328421 0 0 0 0 0 0
    eth0: 30877420 169783 0 0 0 0 0
    0 55160434 183407 0 0 0 0 0 0
    e.g. /proc/PID/net/dev

    View Slide

  28. e.g. DockerͷωοτϫʔΫϞʔυ

    View Slide

  29. e.g. DockerͷωοτϫʔΫϞʔυ
    • bridge, none
    • Private Network Mode
    • ίϯςφ͝ͱʹಠཱͨ͠ΠϯλʔϑΣʔε
    • host, container
    • Shared Network Mode
    • ϗετϓϩηε΍ଞͷίϯςφͱڞ௨ͷΠϯλʔϑΣʔε

    View Slide

  30. ίϯςφΫϥελʹ͓͚Δ
    ϝτϦΫεͷऩू

    View Slide

  31. ίϯςφΫϥελͰϝτϦΫεΛͲ͏ूΊΔ͔ʁ

    View Slide

  32. ίϯςφΫϥελͰϝτϦΫεΛͲ͏ूΊΔ͔ʁ
    • ίϯςφ͸εέδϡʔϥʹΑΓద੾ͳϗετʹσϓϩΠ͞ΕΔ
    • ϗετ͸εέʔϧ͢Δ
    • ίϯςφ಺ͰϞχλϦϯάΤʔδΣϯτΛಉډͤ͞Δͷ͸NG

    View Slide

  33. ίϯςφΫϥελʹ͓͚ΔϞχλϦϯάͷύλʔϯ
    • per-Host Monitoring Agent
    • Sidecar Monitoring Agent

    View Slide

  34. per-Host Monitoring Agent

    View Slide

  35. per-Host Monitoring Agent
    • ϗετ୯ҐͰίϯςφΛ؂ࢹ͢Δύλʔϯ
    • ֤ϗετʹϞχλϦϯάΤʔδΣϯτίϯςφΛσϓϩΠ͢Δ
    • ϗετͰಈ͘ίϯςφҰཡ͔Βಈతʹ؂ࢹઃఆΛੜ੒͢ΔͨΊɺ
    ෳࡶͳ࣮૷͕ඞཁ

    View Slide

  36. Sidecar Monitoring Agent

    View Slide

  37. Sidecar Monitoring Agent
    • ίϯςφ୯ҐͰ؂ࢹ͢Δύλʔϯ
    • ؂ࢹର৅ͷίϯςφʹԣ෇͚(Sidecar)͢ΔΑ͏ʹϞχλϦϯάΤʔ
    δΣϯτίϯςφΛσϓϩΠ
    • ؂ࢹର৅ͷίϯςφͷΈʹϑΥʔΧεͰ͖ɺίϯςφؒͰωοτϫʔ
    Ϋ΍ϑΝΠϧͳͲͷϦιʔεΛڞ༗Ͱ͖ΔͨΊ࣮૷͸γϯϓϧ
    • 1ϗετʹෳ਺ͷΤʔδΣϯτ͕σϓϩΠ͞ΕΔͷͰϦιʔε໘Ͱ೉
    ͋Γ

    View Slide

  38. MackerelίϯςφΤʔδΣϯτ
    ʢύϒϦοΫϕʔλʣ

    View Slide

  39. MackerelίϯςφΤʔδΣϯτ(ύϒϦοΫϕʔλ)

    View Slide

  40. MackerelίϯςφΤʔδΣϯτ(ύϒϦοΫϕʔλ)
    • ECS Task/Kubernetes Podઐ༻ͷϞχλϦϯάΤʔδΣϯτ
    • Sidecar Monitoring AgentΛ࠾༻
    • কདྷతʹFargateͷΑ͏ͳΠϯϑϥετϥΫνϟͷ؅ཧ͕ෆཁͳίϯςφϓϥοτ
    ϑΥʔϜ͕ओྲྀʹͳΔͱߟ͍͑ͯΔ
    • ܰྔΤʔδΣϯτͱͯ͠࠶ઃܭͯ͠ϦιʔεফඅΛܰݮ
    • ΫϥελΤʔδΣϯτͷAPI͔ΒϦιʔεϝτϦΫεΛऔಘ
    • SidecarʹΑΔϦιʔεڞ༗ʹΑΓ؂ࢹର৅ͷΞϓϦέʔγϣϯ࢓༷ʹ͍ۙ؂ࢹ͕Մೳ

    View Slide

  41. ·ͱΊ

    View Slide

  42. ·ͱΊ
    • ΞϓϦέʔγϣϯϝτϦΫεͱϦιʔεϝτϦΫεʹΘ͚ͯߟ͑Δ
    • ίϯςφͷϝτϦΫεͱͦͷऔಘํ๏Λ஌Δ
    • ίϯςφΫϥελʹ͓͚ΔϞχλϦϯάύλʔϯͱͯ͠per-Host
    Monitoring AgentͱSidecar Monitoring Agent͕͋Δ

    View Slide