コンテナのメトリクスとモニタリングパターン / 190320-sakura-event

Slide 1

Slide 1 text

ίϯςφͷϝτϦΫεͱ ϞχλϦϯάύλʔϯ 2019/03/20 ͸ͯͳ×͘͞Β͕ߟ͑ΔςΫϊϩδʔͷະདྷ ʙίϯςφɾ෼ࢄܕσʔληϯλʔʙ #͘͞ΒͷΠϕϯτ

Slide 2

Slide 2 text

ࣗݾ঺հ • ࠓҪ൏ਓ(id:hayajo_77) • MackerelνʔϜ SRE • ίϯςφཁૉٕज़ɺपลٕज़ ίϯςφٕज़ೖ໳Ծ૝Խͱͷҧ͍Λ஌Γɺཁૉٕज़Λ৮ֶͬͯ΅͏ IUUQTFNQMPZNFOUFOKBQBODPNFOHJOFFSIVCFOUSZ

Slide 3

Slide 3 text

؂ࢹͱϝτϦΫε

Slide 4

Slide 4 text

؂ࢹͷ໨త • γεςϜ͕݈શʹಈ͍͍ͯΔ͔ΛνΣοΫ͢Δ • ظ଴ͨ͠ͱ͓Γʹಈ͍͍ͯΔ͔ • Ձ஋ΛੜΈग़͍ͯ͠Δ͔ • Ϧιʔεʹաෆ଍͸ͳ͍͔ • ো֐ͷ༧ஹ͸ͳ͍͔ • →ϝτϦΫεͷऔಘ/ऩूɺ؂ࢹɺΞϥʔτͰ࣮ݱ͢Δ

Slide 5

Slide 5 text

2ͭͷλΠϓͷϝτϦΫε • 2ͭͷλΠϓͷϝτϦΫεʹ෼͚ͯߟ͑Δ • ΞϓϦέʔγϣϯϝτϦΫε • ϦιʔεϝτϦΫε

Slide 6

Slide 6 text

ΞϓϦέʔγϣϯϝτϦΫε • γεςϜ͕ਖ਼ৗʹಈ͍ͯΔ౓߹͍Λࣔ͢ • ϦΫΤετ਺ɺ੒ޭ/ࣦഊɺϨεϙϯελΠϜɺεϧʔϓοτͳͲ • γεςϜͷਖ਼ৗੑ΍Ձ஋ͷࢦඪͱͳΔ • ͜ͷϝτϦΫε͸ো֐ݪҼڀ໌ʹ͸໾ཱͪʹ͍͘

Slide 7

Slide 7 text

ϦιʔεϝτϦΫε • γεςϜ͕ґଘ͢ΔϦιʔεͷ࢖༻ঢ়گΛࣔ͢ • Ϧιʔε࢖༻ྔ/཰ɺ๞࿨཰ɺΤϥʔ਺ͳͲ • ো֐ݪҼڀ໌΍ΩϟύγςΟϓϥϯχϯάͷࢦඪͱͳΔ • ͜ͷϝτϦΫε͚ͩͰ͸γεςϜ͕ਖ਼ৗʹಈ͍͍ͯΔ͔෼͔Βͳ͍

Slide 8

Slide 8 text

ͲͪΒ΋ॏཁ • ΞϓϦέʔγϣϯϝτϦΫεˠϦιʔεϝτϦΫεͷॱʹௐࠪ͢Δ • ଞͷΞϓϦέʔγϣϯʹґଘ͍ͯ͠Δ৔߹͸ಉ͡Α͏ʹௐࠪ͢Δ • ΢ΣϒαʔόɺΞϓϦέʔγϣϯαʔόɺσʔλϕʔεαʔό

Slide 9

Slide 9 text

ίϯςφͷϝτϦΫε

Slide 10

Slide 10 text

ίϯςφͷΞϓϦέʔγϣϯϝτϦΫε • ΞϓϦέʔγϣϯʹΑͬͯऔಘͰ͖ΔϝτϦΫε͸͞·͟· • ϦΫΤετ਺ɺ੒ޭ/ࣦഊɺϨεϙϯελΠϜɺεϧʔϓοτͳͲ

Slide 11

Slide 11 text

ίϯςφͷΞϓϦέʔγϣϯϝτϦΫε औಘํ๏ • ϑΝΠϧ • ϩά • ΤϯυϙΠϯτ

Slide 12

Slide 12 text

ϑΝΠϧ • σʔλϘϦϡʔϜ • ίϯςφؒͰϑΝΠϧ΍σΟϨΫτϦΛڞ༗ • Bind mounts • ίϯςφϗετͱϑΝΠϧ΍σΟϨΫτϦΛڞ༗

Slide 13

Slide 13 text

$ docker run -v $(pwd):/data alpine /bin/sh -c 'echo "hello world" >/data/hello' $ cat hello hello world e.g. Bind mounts

Slide 14

Slide 14 text

ϩά • ΞϓϦέʔγϣϯͷSTDOUT • docker logs ίϚϯυ • kubectl logs ίϚϯυ • ίϯςφ಺ͷϩάϑΝΠϧ • σʔλϘϦϡʔϜ, Bind mountsΛར༻

Slide 15

Slide 15 text

$ CONTAINER=$(docker run -d alpine /bin/sh -c 'while true; do date; sleep 1; done') $ docker logs --since 3s $CONTAINER Tue Mar 19 10:07:57 UTC 2019 Tue Mar 19 10:07:58 UTC 2019 Tue Mar 19 10:07:59 UTC 2019 e.g. docker logs

Slide 16

Slide 16 text

ΤϯυϙΠϯτ • ωοτϫʔΫܦ༝ͰΞΫηεՄೳͳΤϯυϙΠϯτ • ΞϓϦέʔγϣϯͰ࣮૷͢Δඞཁ͕͋Δ • /metrics, /stats, /health

Slide 17

Slide 17 text

$ curl http://${CONTAINER_IPADDR}/nginx_status Active connections: 43 server accepts handled requests 7368 7368 10993 Reading: 0 Writing: 5 Waiting: 38 e.g. Nginxͷstub_status

Slide 18

Slide 18 text

ίϯςφͷϦιʔεϝτϦΫε • CPU • Memory • I/O • ωοτϫʔΫ

Slide 19

Slide 19 text

ίϯςφͷϦιʔεϝτϦΫεऔಘํ๏ • Ծ૝ϑΝΠϧ • API • CLI

Slide 20

Slide 20 text

Ծ૝ϑΝΠϧ • cgroupͷstatϑΝΠϧ • cpu/cpuacct, memory, blkioαϒγεςϜͳͲ • /proc/PID/net/dev

Slide 21

Slide 21 text

$ cat /sys/fs/cgroup/memory/docker/${CONTAINER}/memory.stat cache 8192 rss 192512 rss_huge 0 mapped_file 0 dirty 0 [...] e.g. memory.stat

Slide 22

Slide 22 text

API • ίϯςφϥϯλΠϜ΍ίϯςφΫϥελͷAPIͳͲ • Docker stats API (Docker) • Kubernetes/kubelet API (Kubernetes) • Task Metadata Endpoint (ECS) • cAdvisor API

Slide 23

Slide 23 text

$ curl -s --unix-socket \ /var/run/docker.sock http://localhost/${CONTAINER}/stats?stream=false | \ jq . [...] "cpu_stats": { "cpu_usage": { "total_usage": 963481062, "percpu_usage": [ 431767249, 531713813 ], [...] e.g. Docker stats API

Slide 24

Slide 24 text

CLI • ίϯςφϥϯλΠϜ΍ίϯςφΫϥελ͕ఏڙ͢ΔίϚϯυ • docker stats ίϚϯυ • kubectl top ίϚϯυ

Slide 25

Slide 25 text

$ docker stats --no-stream CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS d7f9c5c0acfc app 0.00% 1.945MiB / 1.952GiB 0.10% 1.5kB / 0B 5.11MB / 0B 2 8c484a94aca7 monitor 0.19% 652KiB / 1.952GiB 0.03% 1.41kB / 0B 1.24MB / 0B 2 e.g. docker stats

Slide 26

Slide 26 text

ωοτϫʔΫϝτϦΫεͷ஫ҙ఺ • /proc/PID/net/dev • ΠϯλʔϑΣʔε୯ҐͷϝτϦΫε

Slide 27

Slide 27 text

$ cat /proc/${CONTAINER_PID}/net/dev Inter-| Receive | Transmit face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed lo: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ecs-eth0: 467809174 273650 0 0 0 0 0 0 27453834 328421 0 0 0 0 0 0 eth0: 30877420 169783 0 0 0 0 0 0 55160434 183407 0 0 0 0 0 0 e.g. /proc/PID/net/dev

Slide 28

Slide 28 text

e.g. DockerͷωοτϫʔΫϞʔυ

Slide 29

Slide 29 text

e.g. DockerͷωοτϫʔΫϞʔυ • bridge, none • Private Network Mode • ίϯςφ͝ͱʹಠཱͨ͠ΠϯλʔϑΣʔε • host, container • Shared Network Mode • ϗετϓϩηε΍ଞͷίϯςφͱڞ௨ͷΠϯλʔϑΣʔε

Slide 30

Slide 30 text

ίϯςφΫϥελʹ͓͚Δ ϝτϦΫεͷऩू

Slide 31

Slide 31 text

ίϯςφΫϥελͰϝτϦΫεΛͲ͏ूΊΔ͔ʁ

Slide 32

Slide 32 text

ίϯςφΫϥελͰϝτϦΫεΛͲ͏ूΊΔ͔ʁ • ίϯςφ͸εέδϡʔϥʹΑΓద੾ͳϗετʹσϓϩΠ͞ΕΔ • ϗετ͸εέʔϧ͢Δ • ίϯςφ಺ͰϞχλϦϯάΤʔδΣϯτΛಉډͤ͞Δͷ͸NG

Slide 33

Slide 33 text

ίϯςφΫϥελʹ͓͚ΔϞχλϦϯάͷύλʔϯ • per-Host Monitoring Agent • Sidecar Monitoring Agent

Slide 34

Slide 34 text

per-Host Monitoring Agent

Slide 35

Slide 35 text

per-Host Monitoring Agent • ϗετ୯ҐͰίϯςφΛ؂ࢹ͢Δύλʔϯ • ֤ϗετʹϞχλϦϯάΤʔδΣϯτίϯςφΛσϓϩΠ͢Δ • ϗετͰಈ͘ίϯςφҰཡ͔Βಈతʹ؂ࢹઃఆΛੜ੒͢ΔͨΊɺ ෳࡶͳ࣮૷͕ඞཁ

Slide 36

Slide 36 text

Sidecar Monitoring Agent

Slide 37

Slide 37 text

Sidecar Monitoring Agent • ίϯςφ୯ҐͰ؂ࢹ͢Δύλʔϯ • ؂ࢹର৅ͷίϯςφʹԣ෇͚(Sidecar)͢ΔΑ͏ʹϞχλϦϯάΤʔ δΣϯτίϯςφΛσϓϩΠ • ؂ࢹର৅ͷίϯςφͷΈʹϑΥʔΧεͰ͖ɺίϯςφؒͰωοτϫʔ Ϋ΍ϑΝΠϧͳͲͷϦιʔεΛڞ༗Ͱ͖ΔͨΊ࣮૷͸γϯϓϧ • 1ϗετʹෳ਺ͷΤʔδΣϯτ͕σϓϩΠ͞ΕΔͷͰϦιʔε໘Ͱ೉ ͋Γ

Slide 38

Slide 38 text

MackerelίϯςφΤʔδΣϯτ ʢύϒϦοΫϕʔλʣ

Slide 39

Slide 39 text

MackerelίϯςφΤʔδΣϯτ(ύϒϦοΫϕʔλ)

Slide 40

Slide 40 text

MackerelίϯςφΤʔδΣϯτ(ύϒϦοΫϕʔλ) • ECS Task/Kubernetes Podઐ༻ͷϞχλϦϯάΤʔδΣϯτ • Sidecar Monitoring AgentΛ࠾༻ • কདྷతʹFargateͷΑ͏ͳΠϯϑϥετϥΫνϟͷ؅ཧ͕ෆཁͳίϯςφϓϥοτ ϑΥʔϜ͕ओྲྀʹͳΔͱߟ͍͑ͯΔ • ܰྔΤʔδΣϯτͱͯ͠࠶ઃܭͯ͠ϦιʔεফඅΛܰݮ • ΫϥελΤʔδΣϯτͷAPI͔ΒϦιʔεϝτϦΫεΛऔಘ • SidecarʹΑΔϦιʔεڞ༗ʹΑΓ؂ࢹର৅ͷΞϓϦέʔγϣϯ࢓༷ʹ͍ۙ؂ࢹ͕Մೳ

Slide 41

Slide 41 text

·ͱΊ

Slide 42

Slide 42 text

·ͱΊ • ΞϓϦέʔγϣϯϝτϦΫεͱϦιʔεϝτϦΫεʹΘ͚ͯߟ͑Δ • ίϯςφͷϝτϦΫεͱͦͷऔಘํ๏Λ஌Δ • ίϯςφΫϥελʹ͓͚ΔϞχλϦϯάύλʔϯͱͯ͠per-Host Monitoring AgentͱSidecar Monitoring Agent͕͋Δ