Monitoring Docker with Mackerel (2015 ver.)

Slide 1

Slide 1 text

Monitoring Docker with Mackerel   2015೥ฤ Ϋϥ΢υͬͨࣽ͝ #11, 2015/03/28 Hatena Co., Ltd. Shinji Tanaka @stanaka

Slide 2

Slide 2 text

Shinji Tanaka / ాத৻࢘ @stanaka / id:stanaka גࣜձࣾ͸ͯͳ ࣥߦ໾һCTO 2

Slide 3

Slide 3 text

͋͋ ࠷ۙɺిࢠ൛΋ग़ͨΑ͏Ͱ͢

Slide 4

Slide 4 text

Agenda • Docker (cgroup) Statics • Host/Container Patterns • Monitoring Docker with Mackerel

Slide 5

Slide 5 text

1 Docker (cgroup) Statistics

Slide 6

Slide 6 text

DockerͷϞχλϦϯάͷجຊ • cgroupͷ࡞๏ʹै͏ • ίϯςφଆ͔Β͸ɺϗετશମ͔͠ ݟ͑ͳ͍ • /proc/loadavg, /proc/stat • ݟ͍ͨ৔߹͸`-‐v`ͰऔΓࠐΉ

Slide 7

Slide 7 text

ϗετଆ͔ΒͷϞχλϦϯά • cgroupͷϨϙʔτΛར༻ • /sys/fs/cgroup/  {cpuacct, memory} • ଞʹblkio, cpu, devicesͳͲͷα ϒγεςϜ͕͋Δ(֤छ੍ݶͳͲͷ੍ ޚ༻) VCVOUVͷ৔߹  LFSOFMHFOFSJDEPDLFS

Slide 8

Slide 8 text

CPU࢖༻཰ • /sys/fs/cgroup/cpuacct/docker/{ίϯςφID}/cpuacct.stat ! ! ! • user … ϢʔβʔϞʔυͷCPU ࣌ؒ • system … γεςϜϞʔυͷCPU ࣌ؒ • CPU࣌ؒͷ୯Ґ: USER_HZ (σϑΥϧτ͸10ms) user 480 system 116

Slide 9

Slide 9 text

cpuacctͷଞͷࢦඪ • cpuacct.usage: ૯CPU࣌ؒ (φϊඵ)  echo 0 > /cgroup/cpuacct/cpuacct.usage   ͰϦηοτ • cpuacct.usage_percpu: ֤CPUͷCPU ࣌ؒ (φϊඵ)

Slide 10

Slide 10 text

ϝϞϦ࢖༻཰ • /sys/fs/cgroup/memory/docker/{ίϯςφID}/memory.stat cache 761856 rss 25681920 rss_huge 0 mapped_file 16384 pgpgin 12569 … total_cache 761856 total_rss 25681920 … ͍ΘΏΔ344STTNBQQFE@pMF

Slide 11

Slide 11 text

memory.statৄࡉ • cache • rss • mapped_file • pgpgin • pgpgout • swap • active_anon • inactive_anon • active_file • inactive_file • unevictable • hierarchical_memory _limit • hierarchical_memsw • total_͸ࢠάϧʔϓΛ ؚΉ஋ ৄࡉIUUQTXXXLFSOFMPSHEPD%PDVNFOUBUJPODHSPVQTNFNPSZUYU

Slide 12

Slide 12 text

memoryؔ࿈(ͦͷଞ) memory.usage_in_bytes ϝϞϦ࢖༻ྔ memory.memsw.usage_in_bytes ϝϞϦ࢖༻ྔͱεϫοϓྖҬͷ࿨ memory.max_usage_in_bytes ϝϞϦ࠷େ࢖༻ྔ memory.memsw.max_usage_in_bytes εϫοϓ࠷େ࢖༻ྔ memory.limit_in_bytes ϝϞϦͷ੍ݶ஋ memory.memsw.limit_in_bytes ϝϞϦͱεϫοϓ੍ݶ஋ memory.failcnt ϝϞϦ੍ݶ஋ʹୡͨ͠ճ਺ memory.memsw.failcnt ϝϞϦ੍ݶ஋ʹୡͨ͠ճ਺

Slide 13

Slide 13 text

ڞ௨Ϩϙʔτ • tasks ࣮ߦதͷϓϩηεͷҰཡ(PID) • cgroup.procs ࣮ߦதͷεϨουάϧʔϓͷҰཡ(TGID) % cat /sys/fs/cgroup/memory/docker/…./cgroup.procs 31218 31307 % ps aux | egrep '(31218|31307)' root 31218 0.0 0.0 4408 164 ? Ss 13:15 0:00 /bin/sh -‐ c rackup -‐p4567 root 31307 0.0 5.2 281364 26336 ? Sl 13:15 0:06 ruby1.9.1 /usr/local/bin/rackup -‐p4567

Slide 14

Slide 14 text

2 Host/Container Patterns

Slide 15

Slide 15 text

ϗετͱίϯςφͷؔ܎ • 1:n • CIαʔόɾ։ൃαʔόͳͲ • 1:1 • ϓϩμΫγϣϯ؀ڥͳͲ

Slide 16

Slide 16 text

Pattern 1:n • CIαʔόɾ։ൃαʔό • ίϯςφԾ૝Խ؀ڥΛଟ਺ಈ࡞ • ಠཱͨ͠؀ڥ • ىಈఀࢭ͕ߴ଎ • Ϧιʔεͷڞ༗ʹΑΔޮ཰Խ

Slide 17

Slide 17 text

Pattern 1:n • CIαʔόɾ։ൃαʔό )PTU "QQ qVFOUE NBDLFSFM BHFOU "QQ +FOLJOT

Slide 18

Slide 18 text

Pattern 1:1 • ίϯςφԾ૝؀ڥΛҰͭͷΈಈ࡞ • ϦιʔεΛಠ઎ • ؀ڥߏங໘ʹಛԽ • ϓϩμΫγϣϯ؀ڥʹ߹͏ • AWS BeanstalkͷDocker support

Slide 19

Slide 19 text

Pattern 1:1 • ϓϩμΫγϣϯαʔό )PTU qVFOUE NBDLFSFM BHFOU "QQ

Slide 20

Slide 20 text

3 Monitoring Docker with Mackerel

Slide 21

Slide 21 text

Mackerel https://mackerel.io/

Slide 22

Slide 22 text

ಛ௃ • ʮ໾ׂʯʹΑΔϗετ܈ͷ؅ཧ • ΤʔδΣϯτʹΑΔϦιʔε؂ࢹ ϩ ʔ ϧ

Slide 23

Slide 23 text

ϩʔϧຖͷάϥϑͰՄࢹԽ

Slide 24

Slide 24 text

mackerel-agent • Go Ͱॻ͔ΕͨެࣜΤʔδΣϯτ • Linux αϙʔτ (CentOS 5/6, Debian 6/7) • ϗετεϖοΫͱϝτϦοΫΛऩू mackerelio/mackerel-agent

Slide 25

Slide 25 text

Blue-green DeploymentରԠ • ϩʔϧ୯ҐͰͷՄࢹԽ • ୀ໾ࡁΈίϯςφͷάϥϑ΋อ࣋ Deployment

Slide 26

Slide 26 text

ΦʔτεέʔϧରԠ • αʔό͕ॖୀͯ͠΋άϥϑ͸࢒Δ

Slide 27

Slide 27 text

Raspberry Pi Ͱ΋ಈ͖·͢ • Թ౓ɾ࣪౓ηϯα → Slack௨஌

Slide 28

Slide 28 text

Docker  with Mackerel

Slide 29

Slide 29 text

mackerel/mackerel-agent on Docker Hub https://registry.hub.docker.com/u/stanaka/mackerel-‐agent/

Slide 30

Slide 30 text

How to Use • ίϚϯυҰൃ • ඞཁͳύεɾAPIKEYΛ౉͢ docker run -‐h `hostname` \ -‐v /var/run/docker.sock:/var/run/docker.sock \ -‐v /var/lib/mackerel-‐agent/:/var/lib/mackerel-‐agent/ \ -‐v /proc/mounts:/host/proc/mounts:ro \ -‐v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -‐e 'apikey=‘ \ mackerel/mackerel-‐agent

Slide 31

Slide 31 text

DockerFile(ൈਮ) FROM debian:jessie ! # setup mackerel-‐agent RUN apt-‐get update \ .. && apt-‐get -‐y install mackerel-‐agent \ .. ! # setup docker plugin COPY mackerel-‐plugin-‐docker.rb /mackerel-‐plugin-‐docker.rb RUN echo [plugin.metrics.docker] \ >> /etc/mackerel-‐agent/mackerel-‐agent.conf RUN echo command = \"ruby /mackerel-‐plugin-‐docker.rb\" \ >> /etc/mackerel-‐agent/mackerel-‐agent.conf ! # boot mackerel-‐agent CMD /startup.sh

Slide 32

Slide 32 text

֤ίϯςφͷ৘ใΛऩू )PTU qVFOUE NBDLFSFM BHFOU "QQ

Slide 33

Slide 33 text

༷ࢠ(ίϯςφຖͷCPU)

Slide 34

Slide 34 text

༷ࢠ(ίϯςφຖͷϝϞϦ)

Slide 35

Slide 35 text

༷ࢠ(·ͱΊͯ)

Slide 36

Slide 36 text

ίϯςφͷத਎ͷ ৘ใΛऩू͍ͨ͠

Slide 37

Slide 37 text

ίϯςφͷத਎ͷ৘ใΛऩू )PTU NBDLFSFM BHFOU "QQ

Slide 38

Slide 38 text

Memcachedͷྫ % docker run -‐d -‐P \ -‐-‐name memcached -‐p 11211:11211 \ sylvainlasnier/memcached )PTU NBDLFSFM BHFOU NFNDBDIFE

Slide 39

Slide 39 text

linkΦϓγϣϯͰ઀ଓ % docker run -‐i -‐t -‐h `hostname` \ -‐v /var/run/docker.sock:/var/run/docker.sock \ -‐v /proc/mounts:/host/proc/mounts:ro \ -‐v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \ -‐e 'apikey=' -‐-‐link memcached:memcached \ mackerel-‐agent /bin/bash ! root@dev:/# env HOSTNAME=develop04.host.h MEMCACHED_PORT_11211_TCP_PORT=11211 MEMCACHED_PORT_11211_TCP_PROTO=tcp MEMCACHED_NAME=/tender_curie/memcached MEMCACHED_PORT=tcp://172.17.0.88:11211 MEMCACHED_PORT_11211_TCP_ADDR=172.17.0.88 MEMCACHED_PORT_11211_TCP=tcp://172.17.0.88:11211

Slide 40

Slide 40 text

؀ڥม਺ͰΞυϨεࢦఆ root@dev:/# /usr/local/bin/mackerel-‐plugin-‐memcached \ -‐host=$MEMCACHED_PORT_11211_TCP_ADDR memcached.connections.curr_connections 10 1427507503 memcached.cmd.cmd_get 0 1427507503 memcached.cmd.cmd_set 0 1427507503

Slide 41

Slide 41 text

ϓϥάΠϯઃఆΛ࢓ࠐΉ % docker run -‐h `hostname` \ -‐v /var/run/docker.sock:/var/run/docker.sock \ -‐v /var/lib/mackerel-‐agent/:/var/lib/mackerel-‐agent/ \ -‐v /proc/mounts:/host/proc/mounts:ro \ -‐v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -‐e 'apikey=‘ \ -‐-‐link memcached:memcached \ -‐v /etc/mackerel-‐agent/conf.d:/etc/mackerel-‐agent/conf.d:ro -‐e ‘include=/etc/mackerel-‐agent/conf.d/*' \ mackerel/mackerel-‐agent % cat /etc/mackerel-‐agent/conf.d/memcached.conf [plugin.metrics.memcached] command = "/usr/local/bin/mackerel-‐plugin-‐memcached -‐ host=$MEMCACHED_PORT_11211_TCP_ADDR"

Slide 42

Slide 42 text

MemcachedͷϝτϦοΫ

Slide 43

Slide 43 text

·ͱΊ • AgentΛίϯςφͰཱͯΔͱ؆୯ • ॏ͍ίϯςφ͕͙͢ʹൃݟͰ͖Δ • ֤ίϯςφͷத਎΋ݟΕΔ

Slide 44

Slide 44 text

Q & A