Upgrade to Pro — share decks privately, control downloads, hide ads and more …

コンテナのメトリクスと モニタリングパターン / 190320-sakura-event

コンテナのメトリクスと モニタリングパターン / 190320-sakura-event

87425b9ed1c97009802d66c6aebbfcdb?s=128

Hayato Imai

March 20, 2019
Tweet

Transcript

  1. ίϯςφͷϝτϦΫεͱ ϞχλϦϯάύλʔϯ 2019/03/20 ͸ͯͳ×͘͞Β͕ߟ͑ΔςΫϊϩδʔͷະདྷ ʙίϯςφɾ෼ࢄܕσʔληϯλʔʙ #͘͞ΒͷΠϕϯτ

  2. ࣗݾ঺հ • ࠓҪ൏ਓ(id:hayajo_77) • MackerelνʔϜ SRE • ίϯςφཁૉٕज़ɺपลٕज़ ίϯςφٕज़ೖ໳Ծ૝Խͱͷҧ͍Λ஌Γɺཁૉٕज़Λ৮ֶͬͯ΅͏ IUUQTFNQMPZNFOUFOKBQBODPNFOHJOFFSIVCFOUSZ

    
  3. ؂ࢹͱϝτϦΫε

  4. ؂ࢹͷ໨త • γεςϜ͕݈શʹಈ͍͍ͯΔ͔ΛνΣοΫ͢Δ • ظ଴ͨ͠ͱ͓Γʹಈ͍͍ͯΔ͔ • Ձ஋ΛੜΈग़͍ͯ͠Δ͔ • Ϧιʔεʹաෆ଍͸ͳ͍͔ •

    ো֐ͷ༧ஹ͸ͳ͍͔ • →ϝτϦΫεͷऔಘ/ऩूɺ؂ࢹɺΞϥʔτͰ࣮ݱ͢Δ
  5. 2ͭͷλΠϓͷϝτϦΫε • 2ͭͷλΠϓͷϝτϦΫεʹ෼͚ͯߟ͑Δ • ΞϓϦέʔγϣϯϝτϦΫε • ϦιʔεϝτϦΫε

  6. ΞϓϦέʔγϣϯϝτϦΫε • γεςϜ͕ਖ਼ৗʹಈ͍ͯΔ౓߹͍Λࣔ͢ • ϦΫΤετ਺ɺ੒ޭ/ࣦഊɺϨεϙϯελΠϜɺεϧʔϓοτͳͲ • γεςϜͷਖ਼ৗੑ΍Ձ஋ͷࢦඪͱͳΔ • ͜ͷϝτϦΫε͸ো֐ݪҼڀ໌ʹ͸໾ཱͪʹ͍͘

  7. ϦιʔεϝτϦΫε • γεςϜ͕ґଘ͢ΔϦιʔεͷ࢖༻ঢ়گΛࣔ͢ • Ϧιʔε࢖༻ྔ/཰ɺ๞࿨཰ɺΤϥʔ਺ͳͲ • ো֐ݪҼڀ໌΍ΩϟύγςΟϓϥϯχϯάͷࢦඪͱͳΔ • ͜ͷϝτϦΫε͚ͩͰ͸γεςϜ͕ਖ਼ৗʹಈ͍͍ͯΔ͔෼͔Βͳ͍

  8. ͲͪΒ΋ॏཁ • ΞϓϦέʔγϣϯϝτϦΫεˠϦιʔεϝτϦΫεͷॱʹௐࠪ͢Δ • ଞͷΞϓϦέʔγϣϯʹґଘ͍ͯ͠Δ৔߹͸ಉ͡Α͏ʹௐࠪ͢Δ • ΢ΣϒαʔόɺΞϓϦέʔγϣϯαʔόɺσʔλϕʔεαʔό

  9. ίϯςφͷϝτϦΫε

  10. ίϯςφͷΞϓϦέʔγϣϯϝτϦΫε • ΞϓϦέʔγϣϯʹΑͬͯऔಘͰ͖ΔϝτϦΫε͸͞·͟· • ϦΫΤετ਺ɺ੒ޭ/ࣦഊɺϨεϙϯελΠϜɺεϧʔϓοτͳͲ

  11. ίϯςφͷΞϓϦέʔγϣϯϝτϦΫε औಘํ๏ • ϑΝΠϧ • ϩά • ΤϯυϙΠϯτ

  12. ϑΝΠϧ • σʔλϘϦϡʔϜ • ίϯςφؒͰϑΝΠϧ΍σΟϨΫτϦΛڞ༗ • Bind mounts • ίϯςφϗετͱϑΝΠϧ΍σΟϨΫτϦΛڞ༗

  13. $ docker run -v $(pwd):/data alpine /bin/sh -c 'echo "hello

    world" >/data/hello' $ cat hello hello world e.g. Bind mounts
  14. ϩά • ΞϓϦέʔγϣϯͷSTDOUT • docker logs ίϚϯυ • kubectl logs

    ίϚϯυ • ίϯςφ಺ͷϩάϑΝΠϧ • σʔλϘϦϡʔϜ, Bind mountsΛར༻
  15. $ CONTAINER=$(docker run -d alpine /bin/sh -c 'while true; do

    date; sleep 1; done') $ docker logs --since 3s $CONTAINER Tue Mar 19 10:07:57 UTC 2019 Tue Mar 19 10:07:58 UTC 2019 Tue Mar 19 10:07:59 UTC 2019 e.g. docker logs
  16. ΤϯυϙΠϯτ • ωοτϫʔΫܦ༝ͰΞΫηεՄೳͳΤϯυϙΠϯτ • ΞϓϦέʔγϣϯͰ࣮૷͢Δඞཁ͕͋Δ • /metrics, /stats, /health

  17. $ curl http://${CONTAINER_IPADDR}/nginx_status Active connections: 43 server accepts handled requests

    7368 7368 10993 Reading: 0 Writing: 5 Waiting: 38 e.g. Nginxͷstub_status
  18. ίϯςφͷϦιʔεϝτϦΫε • CPU • Memory • I/O • ωοτϫʔΫ

  19. ίϯςφͷϦιʔεϝτϦΫεऔಘํ๏ • Ծ૝ϑΝΠϧ • API • CLI

  20. Ծ૝ϑΝΠϧ • cgroupͷstatϑΝΠϧ • cpu/cpuacct, memory, blkioαϒγεςϜͳͲ • /proc/PID/net/dev

  21. $ cat /sys/fs/cgroup/memory/docker/${CONTAINER}/memory.stat cache 8192 rss 192512 rss_huge 0 mapped_file

    0 dirty 0 [...] e.g. memory.stat
  22. API • ίϯςφϥϯλΠϜ΍ίϯςφΫϥελͷAPIͳͲ • Docker stats API (Docker) • Kubernetes/kubelet

    API (Kubernetes) • Task Metadata Endpoint (ECS) • cAdvisor API
  23. $ curl -s --unix-socket \ /var/run/docker.sock http://localhost/${CONTAINER}/stats?stream=false | \ jq

    . [...] "cpu_stats": { "cpu_usage": { "total_usage": 963481062, "percpu_usage": [ 431767249, 531713813 ], [...] e.g. Docker stats API
  24. CLI • ίϯςφϥϯλΠϜ΍ίϯςφΫϥελ͕ఏڙ͢ΔίϚϯυ • docker stats ίϚϯυ • kubectl top

    ίϚϯυ
  25. $ docker stats --no-stream CONTAINER ID NAME CPU % MEM

    USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS d7f9c5c0acfc app 0.00% 1.945MiB / 1.952GiB 0.10% 1.5kB / 0B 5.11MB / 0B 2 8c484a94aca7 monitor 0.19% 652KiB / 1.952GiB 0.03% 1.41kB / 0B 1.24MB / 0B 2 e.g. docker stats
  26. ωοτϫʔΫϝτϦΫεͷ஫ҙ఺ • /proc/PID/net/dev • ΠϯλʔϑΣʔε୯ҐͷϝτϦΫε

  27. $ cat /proc/${CONTAINER_PID}/net/dev Inter-| Receive | Transmit face |bytes packets

    errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed lo: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ecs-eth0: 467809174 273650 0 0 0 0 0 0 27453834 328421 0 0 0 0 0 0 eth0: 30877420 169783 0 0 0 0 0 0 55160434 183407 0 0 0 0 0 0 e.g. /proc/PID/net/dev
  28. e.g. DockerͷωοτϫʔΫϞʔυ

  29. e.g. DockerͷωοτϫʔΫϞʔυ • bridge, none • Private Network Mode •

    ίϯςφ͝ͱʹಠཱͨ͠ΠϯλʔϑΣʔε • host, container • Shared Network Mode • ϗετϓϩηε΍ଞͷίϯςφͱڞ௨ͷΠϯλʔϑΣʔε
  30. ίϯςφΫϥελʹ͓͚Δ ϝτϦΫεͷऩू

  31. ίϯςφΫϥελͰϝτϦΫεΛͲ͏ूΊΔ͔ʁ

  32. ίϯςφΫϥελͰϝτϦΫεΛͲ͏ूΊΔ͔ʁ • ίϯςφ͸εέδϡʔϥʹΑΓద੾ͳϗετʹσϓϩΠ͞ΕΔ • ϗετ͸εέʔϧ͢Δ • ίϯςφ಺ͰϞχλϦϯάΤʔδΣϯτΛಉډͤ͞Δͷ͸NG

  33. ίϯςφΫϥελʹ͓͚ΔϞχλϦϯάͷύλʔϯ • per-Host Monitoring Agent • Sidecar Monitoring Agent

  34. per-Host Monitoring Agent

  35. per-Host Monitoring Agent • ϗετ୯ҐͰίϯςφΛ؂ࢹ͢Δύλʔϯ • ֤ϗετʹϞχλϦϯάΤʔδΣϯτίϯςφΛσϓϩΠ͢Δ • ϗετͰಈ͘ίϯςφҰཡ͔Βಈతʹ؂ࢹઃఆΛੜ੒͢ΔͨΊɺ ෳࡶͳ࣮૷͕ඞཁ

  36. Sidecar Monitoring Agent

  37. Sidecar Monitoring Agent • ίϯςφ୯ҐͰ؂ࢹ͢Δύλʔϯ • ؂ࢹର৅ͷίϯςφʹԣ෇͚(Sidecar)͢ΔΑ͏ʹϞχλϦϯάΤʔ δΣϯτίϯςφΛσϓϩΠ • ؂ࢹର৅ͷίϯςφͷΈʹϑΥʔΧεͰ͖ɺίϯςφؒͰωοτϫʔ

    Ϋ΍ϑΝΠϧͳͲͷϦιʔεΛڞ༗Ͱ͖ΔͨΊ࣮૷͸γϯϓϧ • 1ϗετʹෳ਺ͷΤʔδΣϯτ͕σϓϩΠ͞ΕΔͷͰϦιʔε໘Ͱ೉ ͋Γ
  38. MackerelίϯςφΤʔδΣϯτ ʢύϒϦοΫϕʔλʣ

  39. MackerelίϯςφΤʔδΣϯτ(ύϒϦοΫϕʔλ)

  40. MackerelίϯςφΤʔδΣϯτ(ύϒϦοΫϕʔλ) • ECS Task/Kubernetes Podઐ༻ͷϞχλϦϯάΤʔδΣϯτ • Sidecar Monitoring AgentΛ࠾༻ •

    কདྷతʹFargateͷΑ͏ͳΠϯϑϥετϥΫνϟͷ؅ཧ͕ෆཁͳίϯςφϓϥοτ ϑΥʔϜ͕ओྲྀʹͳΔͱߟ͍͑ͯΔ • ܰྔΤʔδΣϯτͱͯ͠࠶ઃܭͯ͠ϦιʔεফඅΛܰݮ • ΫϥελΤʔδΣϯτͷAPI͔ΒϦιʔεϝτϦΫεΛऔಘ • SidecarʹΑΔϦιʔεڞ༗ʹΑΓ؂ࢹର৅ͷΞϓϦέʔγϣϯ࢓༷ʹ͍ۙ؂ࢹ͕Մೳ
  41. ·ͱΊ

  42. ·ͱΊ • ΞϓϦέʔγϣϯϝτϦΫεͱϦιʔεϝτϦΫεʹΘ͚ͯߟ͑Δ • ίϯςφͷϝτϦΫεͱͦͷऔಘํ๏Λ஌Δ • ίϯςφΫϥελʹ͓͚ΔϞχλϦϯάύλʔϯͱͯ͠per-Host Monitoring AgentͱSidecar Monitoring

    Agent͕͋Δ