Monitoring Docker with Mackerel

by Shinji Tanaka

Slide 1

Slide 1 text

Monitoring Docker with Mackerel Docker meetup Tokyo #3, 2014-07-04 Hatena Co., Ltd. Shinji Tanaka @stanaka

Slide 2

Slide 2 text

Agenda • Monitoring Docker • Docker Patterns & Deployment • Monitoring Docker with Mackerel

Slide 3

Slide 3 text

1 Monitoring Docker

Slide 4

Slide 4 text

DockerͷϞχλϦϯάͷجຊ • cgroupͷ࡞๏ʹै͏ • ίϯςφଆ͔Β͸ɺϗετશମ͔͠ ݟ͑ͳ͍ • /proc/loadavg, /proc/stat

Slide 5

Slide 5 text

ϗετଆ͔ΒͷϞχλϦϯά • cgroupͷϨϙʔτΛར༻ • /sys/fs/cgroup/  {cpuacct, memory} • ଞʹblkio, cpu, devicesͳͲͷα ϒγεςϜ͕͋Δ(֤छ੍ݶͳͲͷ੍ ޚ༻) VCVOUVͷ৔߹  LFSOFMHFOFSJDEPDLFS

Slide 6

Slide 6 text

CPU࢖༻཰ • /sys/fs/cgroup/cpuacct/docker/{ίϯςφID}/cpuacct.stat ! ! ! • user … ϢʔβʔϞʔυͷCPU ࣌ؒ • system … γεςϜϞʔυͷCPU ࣌ؒ • CPU࣌ؒͷ୯Ґ: USER_HZ (σϑΥϧτ͸10ms) user 480 system 116

Slide 7

Slide 7 text

cpuacctͷଞͷࢦඪ • cpuacct.usage: ૯CPU࣌ؒ (φϊඵ)  echo 0 > /cgroup/cpuacct/cpuacct.usage   ͰϦηοτ • cpuacct.usage_percpu: ֤CPUͷCPU ࣌ؒ (φϊඵ)

Slide 8

Slide 8 text

ϝϞϦ࢖༻཰ • /sys/fs/cgroup/memory/docker/{ίϯςφID}/memory.stat cache 761856 rss 25681920 rss_huge 0 mapped_file 16384 pgpgin 12569 … total_cache 761856 total_rss 25681920 … ͍ΘΏΔ344STTNBQQFE@pMF

Slide 9

Slide 9 text

memory.statৄࡉ • cache • rss • mapped_file • pgpgin • pgpgout • swap • active_anon • inactive_anon • active_file • inactive_file • unevictable • hierarchical_memory _limit • hierarchical_memsw • total_͸ࢠάϧʔϓΛ ؚΉ஋ ৄࡉIUUQTXXXLFSOFMPSHEPD%PDVNFOUBUJPODHSPVQTNFNPSZUYU

Slide 10

Slide 10 text

memoryؔ࿈(ͦͷଞ) memory.usage_in_bytes ϝϞϦ࢖༻ྔ memory.memsw.usage_in_bytes ϝϞϦ࢖༻ྔͱεϫοϓྖҬͷ࿨ memory.max_usage_in_bytes ϝϞϦ࠷େ࢖༻ྔ memory.memsw.max_usage_in_bytes εϫοϓ࠷େ࢖༻ྔ memory.limit_in_bytes ϝϞϦͷ੍ݶ஋ memory.memsw.limit_in_bytes ϝϞϦͱεϫοϓ੍ݶ஋ memory.failcnt ϝϞϦ੍ݶ஋ʹୡͨ͠ճ਺ memory.memsw.failcnt ϝϞϦ੍ݶ஋ʹୡͨ͠ճ਺

Slide 11

Slide 11 text

ڞ௨Ϩϙʔτ • tasks ࣮ߦதͷϓϩηεͷҰཡ(PID) • cgroup.procs ࣮ߦதͷεϨουάϧʔϓͷҰཡ(TGID) % cat /sys/fs/cgroup/memory/docker/…./cgroup.procs 31218 31307 % ps aux | egrep '(31218|31307)' root 31218 0.0 0.0 4408 164 ? Ss 13:15 0:00 /bin/sh -‐ c rackup -‐p4567 root 31307 0.0 5.2 281364 26336 ? Sl 13:15 0:06 ruby1.9.1 /usr/local/bin/rackup -‐p4567

Slide 12

Slide 12 text

2 Docker Patterns (with Docker Hub) & Deployment

Slide 13

Slide 13 text

ϗετͱίϯςφͷؔ܎ • 1:n • CIαʔόɾ։ൃαʔόͳͲ • 1:1 • ϓϩμΫγϣϯ؀ڥͳͲ

Slide 14

Slide 14 text

Pattern 1:n • CIαʔόɾ։ൃαʔό • ίϯςφԾ૝Խ؀ڥΛଟ਺ಈ࡞ • ಠཱͨ͠؀ڥ • ىಈఀࢭ͕ߴ଎ • Ϧιʔεͷڞ༗ʹΑΔޮ཰Խ

Slide 15

Slide 15 text

Pattern 1:1 • ίϯςφԾ૝؀ڥΛҰͭͷΈಈ࡞ • ϦιʔεΛಠ઎ • ؀ڥߏங໘ʹಛԽ • ϓϩμΫγϣϯ؀ڥʹ߹͏ • AWS BeanstalkͷDocker support

Slide 16

Slide 16 text

Deployment with Docker Hub

Slide 17

Slide 17 text

Deployment (JU)VC %PDLFS)VC IVCPU

Slide 18

Slide 18 text

GitHub

Slide 19

Slide 19 text

Docker Hub

Slide 20

Slide 20 text

Automated Build

Slide 21

Slide 21 text

Build Success

Slide 22

Slide 22 text

Build Failed

Slide 23

Slide 23 text

Webhook from Docker Hub { "push_data":{ "pushed_at":1385141110, "images":[ "imagehash1" ], "pusher":"username" }, "repository":{ "status":"Active", "description":"my docker repo that does cool things", "is_trusted":false, "full_description":"This is my full description", "repo_url":"https://registry.hub.docker.com/u/username/reponame/",

Slide 24

Slide 24 text

hubot • νϟοτπʔϧͰσϓϩΠ • Build௨஌ ! • σϓϩΠ࣮ߦ

Slide 25

Slide 25 text

3 Monitoring Docker with Mackerel

Slide 26

Slide 26 text

Mackerel https://mackerel.io/

Slide 27

Slide 27 text

Mackerelʢ೔ຊޠͰḉʣ • αʔό؅ཧπʔϧ as a Service • લ਎: ͸ͯͳͷࣾ಺πʔϧ

Slide 28

Slide 28 text

ಛ௃ • ʮ໾ׂʯʹΑΔϗετ܈ͷ؅ཧ • ΤʔδΣϯτʹΑΔϦιʔε؂ࢹ • API ʹΑΔ؅ཧ ϩ ʔ ϧ

Slide 29

Slide 29 text

αʔϏεͱϩʔϧ • αʔϏε • ͞·͟·ͳϗετ͕ڠௐ • Hatena-Bookmark, Hatena-Blog, … • ϗετ܈Λ໾ׂͰ·ͱΊΔ = ϩʔϧ • app, db, …

Slide 30

Slide 30 text

ϩʔϧຖͷάϥϑͰՄࢹԽ

Slide 31

Slide 31 text

mackerel-agent • Go Ͱॻ͔ΕͨެࣜΤʔδΣϯτ • Linux αϙʔτ (CentOS 5/6, Debian 6/7) • ϗετεϖοΫͱϝτϦοΫΛऩू mackerelio/mackerel-agent

Slide 32

Slide 32 text

αʔϏεɾϩʔϧͷࢦఆ • agentͷઃఆ͔ΒαʔϏεɾϩʔϧΛ ࢦఆ # mackerel-‐agent.conf roles = [ “Mackerel:db”, … ]

Slide 33

Slide 33 text

agent: ΧελϜϝτϦοΫ • Sensu ϓϥάΠϯޓ׵ͷग़ྗ  ʢλϒ۠੾Γʣ % ~/.mackerel-‐agent/plugins/chrome-‐tabs-‐count.sh browser.tabs.count 9 1403153173 browser.tabs.works 7 1403153173

Slide 34

Slide 34 text

mackerel-docker • ϗετଆ͔ΒDockerίϯςφΛϞχλϦϯά ! • mackerel-agentͷઃఆ IUUQTHJUIVCDPNTUBOBLBNBDLFSFMEPDLFS % cat mackerel-‐agent.conf  [plugin.metrics.docker]  command = "ruby /home/stanaka/git/mackerel-‐docker/mackerel-‐ docker.rb"

Slide 35

Slide 35 text

Dockerίϯςφ΁ͷ૊ࠐΈ • Dockerﬁle ! ! ! ! • Dockerίϯςφͷىಈ CMD echo APIKEY=\"$APIKEY\" > /etc/default/mackerel-‐agent   && echo OTHER_OPTS="-‐role=sample:${ROLE:-‐app2}" >> /etc/default/mackerel-‐ agent   && /etc/init.d/mackerel-‐agent start   && env MACKEREL_APIKEY=$APIKEY mkr host status -‐-‐host-‐id `cat /var/lib/ mackerel-‐agent/id` -‐-‐status working   && rackup -‐p4567 docker run -‐e “APIKEY=……” stanaka/meetup

Slide 36

Slide 36 text

Docker pluginʹΑΔϞχλϦϯά

Slide 37

Slide 37 text

DockerίϯςφͷCPUάϥϑ

Slide 38

Slide 38 text

Blue-green DeploymentରԠ • ϩʔϧ୯ҐͰͷՄࢹԽ • ୀ໾ࡁΈίϯςφͷάϥϑ΋อ࣋ Deployment

Slide 39

Slide 39 text

Q & A