ίϯςφͷϝτϦΫεͱϞχλϦϯάύλʔϯ2019/03/20ͯͳ×͘͞Β͕ߟ͑ΔςΫϊϩδʔͷະདྷ ʙίϯςφɾࢄܕσʔληϯλʔʙ#͘͞ΒͷΠϕϯτ
View Slide
ࣗݾհ• ࠓҪ൏ਓ(id:hayajo_77)• MackerelνʔϜ SRE• ίϯςφཁૉٕज़ɺपลٕज़ίϯςφٕज़ೖԾԽͱͷҧ͍ΛΓɺཁૉٕज़Λ৮ֶͬͯ΅͏IUUQTFNQMPZNFOUFOKBQBODPNFOHJOFFSIVCFOUSZ
ࢹͱϝτϦΫε
ࢹͷత• γεςϜ͕݈શʹಈ͍͍ͯΔ͔ΛνΣοΫ͢Δ• ظͨ͠ͱ͓Γʹಈ͍͍ͯΔ͔• ՁΛੜΈग़͍ͯ͠Δ͔• Ϧιʔεʹաෆͳ͍͔• োͷ༧ஹͳ͍͔• →ϝτϦΫεͷऔಘ/ऩूɺࢹɺΞϥʔτͰ࣮ݱ͢Δ
2ͭͷλΠϓͷϝτϦΫε• 2ͭͷλΠϓͷϝτϦΫεʹ͚ͯߟ͑Δ• ΞϓϦέʔγϣϯϝτϦΫε• ϦιʔεϝτϦΫε
ΞϓϦέʔγϣϯϝτϦΫε• γεςϜ͕ਖ਼ৗʹಈ͍ͯΔ߹͍Λࣔ͢• ϦΫΤετɺޭ/ࣦഊɺϨεϙϯελΠϜɺεϧʔϓοτͳͲ• γεςϜͷਖ਼ৗੑՁͷࢦඪͱͳΔ• ͜ͷϝτϦΫεোݪҼڀ໌ʹཱͪʹ͍͘
ϦιʔεϝτϦΫε• γεςϜ͕ґଘ͢ΔϦιʔεͷ༻ঢ়گΛࣔ͢• Ϧιʔε༻ྔ/ɺɺΤϥʔͳͲ• োݪҼڀ໌ΩϟύγςΟϓϥϯχϯάͷࢦඪͱͳΔ• ͜ͷϝτϦΫε͚ͩͰγεςϜ͕ਖ਼ৗʹಈ͍͍ͯΔ͔͔Βͳ͍
ͲͪΒॏཁ• ΞϓϦέʔγϣϯϝτϦΫεˠϦιʔεϝτϦΫεͷॱʹௐࠪ͢Δ• ଞͷΞϓϦέʔγϣϯʹґଘ͍ͯ͠Δ߹ಉ͡Α͏ʹௐࠪ͢Δ• ΣϒαʔόɺΞϓϦέʔγϣϯαʔόɺσʔλϕʔεαʔό
ίϯςφͷϝτϦΫε
ίϯςφͷΞϓϦέʔγϣϯϝτϦΫε• ΞϓϦέʔγϣϯʹΑͬͯऔಘͰ͖ΔϝτϦΫε͞·͟·• ϦΫΤετɺޭ/ࣦഊɺϨεϙϯελΠϜɺεϧʔϓοτͳͲ
ίϯςφͷΞϓϦέʔγϣϯϝτϦΫεऔಘํ๏• ϑΝΠϧ• ϩά• ΤϯυϙΠϯτ
ϑΝΠϧ• σʔλϘϦϡʔϜ• ίϯςφؒͰϑΝΠϧσΟϨΫτϦΛڞ༗• Bind mounts• ίϯςφϗετͱϑΝΠϧσΟϨΫτϦΛڞ༗
$ docker run -v $(pwd):/data alpine /bin/sh -c'echo "hello world" >/data/hello'$ cat hellohello worlde.g. Bind mounts
ϩά• ΞϓϦέʔγϣϯͷSTDOUT• docker logs ίϚϯυ• kubectl logs ίϚϯυ• ίϯςφͷϩάϑΝΠϧ• σʔλϘϦϡʔϜ, Bind mountsΛར༻
$ CONTAINER=$(docker run -d alpine /bin/sh -c'while true; do date; sleep 1; done')$ docker logs --since 3s $CONTAINERTue Mar 19 10:07:57 UTC 2019Tue Mar 19 10:07:58 UTC 2019Tue Mar 19 10:07:59 UTC 2019e.g. docker logs
ΤϯυϙΠϯτ• ωοτϫʔΫܦ༝ͰΞΫηεՄೳͳΤϯυϙΠϯτ• ΞϓϦέʔγϣϯͰ࣮͢Δඞཁ͕͋Δ• /metrics, /stats, /health
$ curl http://${CONTAINER_IPADDR}/nginx_statusActive connections: 43server accepts handled requests7368 7368 10993Reading: 0 Writing: 5 Waiting: 38e.g. Nginxͷstub_status
ίϯςφͷϦιʔεϝτϦΫε• CPU• Memory• I/O• ωοτϫʔΫ
ίϯςφͷϦιʔεϝτϦΫεऔಘํ๏• ԾϑΝΠϧ• API• CLI
ԾϑΝΠϧ• cgroupͷstatϑΝΠϧ• cpu/cpuacct, memory, blkioαϒγεςϜͳͲ• /proc/PID/net/dev
$ cat /sys/fs/cgroup/memory/docker/${CONTAINER}/memory.statcache 8192rss 192512rss_huge 0mapped_file 0dirty 0[...]e.g. memory.stat
API• ίϯςφϥϯλΠϜίϯςφΫϥελͷAPIͳͲ• Docker stats API (Docker)• Kubernetes/kubelet API (Kubernetes)• Task Metadata Endpoint (ECS)• cAdvisor API
$ curl -s --unix-socket \/var/run/docker.sock http://localhost/${CONTAINER}/stats?stream=false | \jq .[...]"cpu_stats": {"cpu_usage": {"total_usage": 963481062,"percpu_usage": [431767249,531713813],[...]e.g. Docker stats API
CLI• ίϯςφϥϯλΠϜίϯςφΫϥελ͕ఏڙ͢ΔίϚϯυ• docker stats ίϚϯυ• kubectl top ίϚϯυ
$ docker stats --no-streamCONTAINER ID NAME CPU %MEM USAGE / LIMIT MEM % NETI/O BLOCK I/O PIDSd7f9c5c0acfc app 0.00%1.945MiB / 1.952GiB 0.10%1.5kB / 0B 5.11MB / 0B 28c484a94aca7 monitor 0.19%652KiB / 1.952GiB 0.03%1.41kB / 0B 1.24MB / 0B 2e.g. docker stats
ωοτϫʔΫϝτϦΫεͷҙ• /proc/PID/net/dev• ΠϯλʔϑΣʔε୯ҐͷϝτϦΫε
$ cat /proc/${CONTAINER_PID}/net/devInter-| Receive| Transmitface |bytes packets errs drop fifo frame compressedmulticast|bytes packets errs drop fifo colls carriercompressedlo: 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0ecs-eth0: 467809174 273650 0 0 0 0 00 27453834 328421 0 0 0 0 0 0eth0: 30877420 169783 0 0 0 0 00 55160434 183407 0 0 0 0 0 0e.g. /proc/PID/net/dev
e.g. DockerͷωοτϫʔΫϞʔυ
e.g. DockerͷωοτϫʔΫϞʔυ• bridge, none• Private Network Mode• ίϯςφ͝ͱʹಠཱͨ͠ΠϯλʔϑΣʔε• host, container• Shared Network Mode• ϗετϓϩηεଞͷίϯςφͱڞ௨ͷΠϯλʔϑΣʔε
ίϯςφΫϥελʹ͓͚ΔϝτϦΫεͷऩू
ίϯςφΫϥελͰϝτϦΫεΛͲ͏ूΊΔ͔ʁ
ίϯςφΫϥελͰϝτϦΫεΛͲ͏ूΊΔ͔ʁ• ίϯςφεέδϡʔϥʹΑΓదͳϗετʹσϓϩΠ͞ΕΔ• ϗετεέʔϧ͢Δ• ίϯςφͰϞχλϦϯάΤʔδΣϯτΛಉډͤ͞ΔͷNG
ίϯςφΫϥελʹ͓͚ΔϞχλϦϯάͷύλʔϯ• per-Host Monitoring Agent• Sidecar Monitoring Agent
per-Host Monitoring Agent
per-Host Monitoring Agent• ϗετ୯ҐͰίϯςφΛࢹ͢Δύλʔϯ• ֤ϗετʹϞχλϦϯάΤʔδΣϯτίϯςφΛσϓϩΠ͢Δ• ϗετͰಈ͘ίϯςφҰཡ͔ΒಈతʹࢹઃఆΛੜ͢ΔͨΊɺෳࡶͳ࣮͕ඞཁ
Sidecar Monitoring Agent
Sidecar Monitoring Agent• ίϯςφ୯ҐͰࢹ͢Δύλʔϯ• ࢹରͷίϯςφʹԣ͚(Sidecar)͢ΔΑ͏ʹϞχλϦϯάΤʔδΣϯτίϯςφΛσϓϩΠ• ࢹରͷίϯςφͷΈʹϑΥʔΧεͰ͖ɺίϯςφؒͰωοτϫʔΫϑΝΠϧͳͲͷϦιʔεΛڞ༗Ͱ͖ΔͨΊ࣮γϯϓϧ• 1ϗετʹෳͷΤʔδΣϯτ͕σϓϩΠ͞ΕΔͷͰϦιʔε໘Ͱ͋Γ
MackerelίϯςφΤʔδΣϯτʢύϒϦοΫϕʔλʣ
MackerelίϯςφΤʔδΣϯτ(ύϒϦοΫϕʔλ)
MackerelίϯςφΤʔδΣϯτ(ύϒϦοΫϕʔλ)• ECS Task/Kubernetes Podઐ༻ͷϞχλϦϯάΤʔδΣϯτ• Sidecar Monitoring AgentΛ࠾༻• কདྷతʹFargateͷΑ͏ͳΠϯϑϥετϥΫνϟͷཧ͕ෆཁͳίϯςφϓϥοτϑΥʔϜ͕ओྲྀʹͳΔͱߟ͍͑ͯΔ• ܰྔΤʔδΣϯτͱͯ͠࠶ઃܭͯ͠ϦιʔεফඅΛܰݮ• ΫϥελΤʔδΣϯτͷAPI͔ΒϦιʔεϝτϦΫεΛऔಘ• SidecarʹΑΔϦιʔεڞ༗ʹΑΓࢹରͷΞϓϦέʔγϣϯ༷ʹ͍ۙࢹ͕Մೳ
·ͱΊ
·ͱΊ• ΞϓϦέʔγϣϯϝτϦΫεͱϦιʔεϝτϦΫεʹΘ͚ͯߟ͑Δ• ίϯςφͷϝτϦΫεͱͦͷऔಘํ๏ΛΔ• ίϯςφΫϥελʹ͓͚ΔϞχλϦϯάύλʔϯͱͯ͠per-HostMonitoring AgentͱSidecar Monitoring Agent͕͋Δ
hayajo
yoshinagaiwnl
sbtechnight
ikuyamada
arthur1
nttcom
tatematsu_san
miura55
kazeburo
robhawkes
dougneiner
ufuk
reverentgeek
sugarenia
lauravandoore
colly
marcduiker
ammeep
dotmariusz
jonyablonski
smashingmag