コンテナのメトリクスと モニタリングパターン / 190320-sakura-event

Transcript

1. ίϯςφͷϝτϦΫεͱ ϞχλϦϯάύλʔϯ 2019/03/20 ͸ͯͳ×͘͞Β͕ߟ͑ΔςΫϊϩδʔͷະདྷ ʙίϯςφɾ෼ࢄܕσʔληϯλʔʙ #͘͞ΒͷΠϕϯτ

2. ࣗݾ঺հ • ࠓҪ൏ਓ(id:hayajo_77) • MackerelνʔϜ SRE • ίϯςφཁૉٕज़ɺपลٕज़ ίϯςφٕज़ೖ໳
   
   Ծ૝Խͱͷҧ͍Λ஌Γɺཁૉٕज़Λ৮ֶͬͯ΅͏
   IUUQTFNQMPZNFOUFOKBQBODPNFOHJOFFSIVCFOUSZ

   

3. ؂ࢹͱϝτϦΫε

4. ؂ࢹͷ໨త • γεςϜ͕݈શʹಈ͍͍ͯΔ͔ΛνΣοΫ͢Δ • ظ଴ͨ͠ͱ͓Γʹಈ͍͍ͯΔ͔ • Ձ஋ΛੜΈग़͍ͯ͠Δ͔ • Ϧιʔεʹաෆ଍͸ͳ͍͔ •

   ো֐ͷ༧ஹ͸ͳ͍͔ • →ϝτϦΫεͷऔಘ/ऩूɺ؂ࢹɺΞϥʔτͰ࣮ݱ͢Δ

5. 2ͭͷλΠϓͷϝτϦΫε • 2ͭͷλΠϓͷϝτϦΫεʹ෼͚ͯߟ͑Δ • ΞϓϦέʔγϣϯϝτϦΫε • ϦιʔεϝτϦΫε

6. ΞϓϦέʔγϣϯϝτϦΫε • γεςϜ͕ਖ਼ৗʹಈ͍ͯΔ౓߹͍Λࣔ͢ • ϦΫΤετ਺ɺ੒ޭ/ࣦഊɺϨεϙϯελΠϜɺεϧʔϓοτͳͲ • γεςϜͷਖ਼ৗੑ΍Ձ஋ͷࢦඪͱͳΔ • ͜ͷϝτϦΫε͸ো֐ݪҼڀ໌ʹ͸໾ཱͪʹ͍͘

7. ϦιʔεϝτϦΫε • γεςϜ͕ґଘ͢ΔϦιʔεͷ࢖༻ঢ়گΛࣔ͢ • Ϧιʔε࢖༻ྔ/཰ɺ๞࿨཰ɺΤϥʔ਺ͳͲ • ো֐ݪҼڀ໌΍ΩϟύγςΟϓϥϯχϯάͷࢦඪͱͳΔ • ͜ͷϝτϦΫε͚ͩͰ͸γεςϜ͕ਖ਼ৗʹಈ͍͍ͯΔ͔෼͔Βͳ͍

8. ͲͪΒ΋ॏཁ • ΞϓϦέʔγϣϯϝτϦΫεˠϦιʔεϝτϦΫεͷॱʹௐࠪ͢Δ • ଞͷΞϓϦέʔγϣϯʹґଘ͍ͯ͠Δ৔߹͸ಉ͡Α͏ʹௐࠪ͢Δ • ΢ΣϒαʔόɺΞϓϦέʔγϣϯαʔόɺσʔλϕʔεαʔό

9. ίϯςφͷϝτϦΫε

10. ίϯςφͷΞϓϦέʔγϣϯϝτϦΫε • ΞϓϦέʔγϣϯʹΑͬͯऔಘͰ͖ΔϝτϦΫε͸͞·͟· • ϦΫΤετ਺ɺ੒ޭ/ࣦഊɺϨεϙϯελΠϜɺεϧʔϓοτͳͲ

11. ίϯςφͷΞϓϦέʔγϣϯϝτϦΫε औಘํ๏ • ϑΝΠϧ • ϩά • ΤϯυϙΠϯτ

12. ϑΝΠϧ • σʔλϘϦϡʔϜ • ίϯςφؒͰϑΝΠϧ΍σΟϨΫτϦΛڞ༗ • Bind mounts • ίϯςφϗετͱϑΝΠϧ΍σΟϨΫτϦΛڞ༗

13. $ docker run -v $(pwd):/data alpine /bin/sh -c 'echo "hello

    world" >/data/hello' $ cat hello hello world e.g. Bind mounts

14. ϩά • ΞϓϦέʔγϣϯͷSTDOUT • docker logs ίϚϯυ • kubectl logs

    ίϚϯυ • ίϯςφ಺ͷϩάϑΝΠϧ • σʔλϘϦϡʔϜ, Bind mountsΛར༻

15. $ CONTAINER=$(docker run -d alpine /bin/sh -c 'while true; do

    date; sleep 1; done') $ docker logs --since 3s $CONTAINER Tue Mar 19 10:07:57 UTC 2019 Tue Mar 19 10:07:58 UTC 2019 Tue Mar 19 10:07:59 UTC 2019 e.g. docker logs

16. ΤϯυϙΠϯτ • ωοτϫʔΫܦ༝ͰΞΫηεՄೳͳΤϯυϙΠϯτ • ΞϓϦέʔγϣϯͰ࣮૷͢Δඞཁ͕͋Δ • /metrics, /stats, /health

17. $ curl http://${CONTAINER_IPADDR}/nginx_status Active connections: 43 server accepts handled requests

    7368 7368 10993 Reading: 0 Writing: 5 Waiting: 38 e.g. Nginxͷstub_status

18. ίϯςφͷϦιʔεϝτϦΫε • CPU • Memory • I/O • ωοτϫʔΫ

19. ίϯςφͷϦιʔεϝτϦΫεऔಘํ๏ • Ծ૝ϑΝΠϧ • API • CLI

20. Ծ૝ϑΝΠϧ • cgroupͷstatϑΝΠϧ • cpu/cpuacct, memory, blkioαϒγεςϜͳͲ • /proc/PID/net/dev

21. $ cat /sys/fs/cgroup/memory/docker/${CONTAINER}/memory.stat cache 8192 rss 192512 rss_huge 0 mapped_file

    0 dirty 0 [...] e.g. memory.stat

22. API • ίϯςφϥϯλΠϜ΍ίϯςφΫϥελͷAPIͳͲ • Docker stats API (Docker) • Kubernetes/kubelet

    API (Kubernetes) • Task Metadata Endpoint (ECS) • cAdvisor API

23. $ curl -s --unix-socket \ /var/run/docker.sock http://localhost/${CONTAINER}/stats?stream=false | \ jq

    . [...] "cpu_stats": { "cpu_usage": { "total_usage": 963481062, "percpu_usage": [ 431767249, 531713813 ], [...] e.g. Docker stats API

24. CLI • ίϯςφϥϯλΠϜ΍ίϯςφΫϥελ͕ఏڙ͢ΔίϚϯυ • docker stats ίϚϯυ • kubectl top

    ίϚϯυ

25. $ docker stats --no-stream CONTAINER ID NAME CPU % MEM

    USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS d7f9c5c0acfc app 0.00% 1.945MiB / 1.952GiB 0.10% 1.5kB / 0B 5.11MB / 0B 2 8c484a94aca7 monitor 0.19% 652KiB / 1.952GiB 0.03% 1.41kB / 0B 1.24MB / 0B 2 e.g. docker stats

26. ωοτϫʔΫϝτϦΫεͷ஫ҙ఺ • /proc/PID/net/dev • ΠϯλʔϑΣʔε୯ҐͷϝτϦΫε

27. $ cat /proc/${CONTAINER_PID}/net/dev Inter-| Receive | Transmit face |bytes packets

    errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed lo: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ecs-eth0: 467809174 273650 0 0 0 0 0 0 27453834 328421 0 0 0 0 0 0 eth0: 30877420 169783 0 0 0 0 0 0 55160434 183407 0 0 0 0 0 0 e.g. /proc/PID/net/dev

28. e.g. DockerͷωοτϫʔΫϞʔυ

29. e.g. DockerͷωοτϫʔΫϞʔυ • bridge, none • Private Network Mode •

    ίϯςφ͝ͱʹಠཱͨ͠ΠϯλʔϑΣʔε • host, container • Shared Network Mode • ϗετϓϩηε΍ଞͷίϯςφͱڞ௨ͷΠϯλʔϑΣʔε

30. ίϯςφΫϥελʹ͓͚Δ ϝτϦΫεͷऩू

31. ίϯςφΫϥελͰϝτϦΫεΛͲ͏ूΊΔ͔ʁ

32. ίϯςφΫϥελͰϝτϦΫεΛͲ͏ूΊΔ͔ʁ • ίϯςφ͸εέδϡʔϥʹΑΓద੾ͳϗετʹσϓϩΠ͞ΕΔ • ϗετ͸εέʔϧ͢Δ • ίϯςφ಺ͰϞχλϦϯάΤʔδΣϯτΛಉډͤ͞Δͷ͸NG

33. ίϯςφΫϥελʹ͓͚ΔϞχλϦϯάͷύλʔϯ • per-Host Monitoring Agent • Sidecar Monitoring Agent

34. per-Host Monitoring Agent

35. per-Host Monitoring Agent • ϗετ୯ҐͰίϯςφΛ؂ࢹ͢Δύλʔϯ • ֤ϗετʹϞχλϦϯάΤʔδΣϯτίϯςφΛσϓϩΠ͢Δ • ϗετͰಈ͘ίϯςφҰཡ͔Βಈతʹ؂ࢹઃఆΛੜ੒͢ΔͨΊɺ ෳࡶͳ࣮૷͕ඞཁ

36. Sidecar Monitoring Agent

37. Sidecar Monitoring Agent • ίϯςφ୯ҐͰ؂ࢹ͢Δύλʔϯ • ؂ࢹର৅ͷίϯςφʹԣ෇͚(Sidecar)͢ΔΑ͏ʹϞχλϦϯάΤʔ δΣϯτίϯςφΛσϓϩΠ • ؂ࢹର৅ͷίϯςφͷΈʹϑΥʔΧεͰ͖ɺίϯςφؒͰωοτϫʔ

    Ϋ΍ϑΝΠϧͳͲͷϦιʔεΛڞ༗Ͱ͖ΔͨΊ࣮૷͸γϯϓϧ • 1ϗετʹෳ਺ͷΤʔδΣϯτ͕σϓϩΠ͞ΕΔͷͰϦιʔε໘Ͱ೉ ͋Γ

38. MackerelίϯςφΤʔδΣϯτ ʢύϒϦοΫϕʔλʣ

39. MackerelίϯςφΤʔδΣϯτ(ύϒϦοΫϕʔλ)

40. MackerelίϯςφΤʔδΣϯτ(ύϒϦοΫϕʔλ) • ECS Task/Kubernetes Podઐ༻ͷϞχλϦϯάΤʔδΣϯτ • Sidecar Monitoring AgentΛ࠾༻ •

    কདྷతʹFargateͷΑ͏ͳΠϯϑϥετϥΫνϟͷ؅ཧ͕ෆཁͳίϯςφϓϥοτ ϑΥʔϜ͕ओྲྀʹͳΔͱߟ͍͑ͯΔ • ܰྔΤʔδΣϯτͱͯ͠࠶ઃܭͯ͠ϦιʔεফඅΛܰݮ • ΫϥελΤʔδΣϯτͷAPI͔ΒϦιʔεϝτϦΫεΛऔಘ • SidecarʹΑΔϦιʔεڞ༗ʹΑΓ؂ࢹର৅ͷΞϓϦέʔγϣϯ࢓༷ʹ͍ۙ؂ࢹ͕Մೳ

41. ·ͱΊ

42. ·ͱΊ • ΞϓϦέʔγϣϯϝτϦΫεͱϦιʔεϝτϦΫεʹΘ͚ͯߟ͑Δ • ίϯςφͷϝτϦΫεͱͦͷऔಘํ๏Λ஌Δ • ίϯςφΫϥελʹ͓͚ΔϞχλϦϯάύλʔϯͱͯ͠per-Host Monitoring AgentͱSidecar Monitoring

    Agent͕͋Δ