Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Monitoring Docker with Mackerel

Monitoring Docker with Mackerel

Docker meetup Tokyo #3 2014-07-04

A76985bd77fecba82109dee506b1fec2?s=128

Shinji Tanaka

July 13, 2014
Tweet

Transcript

  1. Monitoring Docker with Mackerel Docker meetup Tokyo #3, 2014-07-04 Hatena

    Co., Ltd. Shinji Tanaka @stanaka
  2. Agenda • Monitoring Docker • Docker Patterns & Deployment •

    Monitoring Docker with Mackerel
  3. 1 Monitoring Docker

  4. DockerͷϞχλϦϯάͷجຊ • cgroupͷ࡞๏ʹै͏ • ίϯςφଆ͔Β͸ɺϗετશମ͔͠ ݟ͑ͳ͍ • /proc/loadavg,  /proc/stat

  5. ϗετଆ͔ΒͷϞχλϦϯά • cgroupͷϨϙʔτΛར༻ • /sys/fs/cgroup/
          {cpuacct,

     memory}   • ଞʹblkio,  cpu,  devicesͳͲͷα ϒγεςϜ͕͋Δ(֤छ੍ݶͳͲͷ੍ ޚ༻) VCVOUVͷ৔߹
 LFSOFMHFOFSJDEPDLFS
  6. CPU࢖༻཰ • /sys/fs/cgroup/cpuacct/docker/{ίϯςφID}/cpuacct.stat ! ! ! • user … ϢʔβʔϞʔυͷCPU

    ࣌ؒ • system … γεςϜϞʔυͷCPU ࣌ؒ • CPU࣌ؒͷ୯Ґ: USER_HZ (σϑΥϧτ͸10ms) user  480   system  116
  7. cpuacctͷଞͷࢦඪ • cpuacct.usage: ૯CPU࣌ؒ (φϊඵ)
 echo 0 > /cgroup/cpuacct/cpuacct.usage 


    ͰϦηοτ • cpuacct.usage_percpu: ֤CPUͷCPU ࣌ؒ (φϊඵ)
  8. ϝϞϦ࢖༻཰ • /sys/fs/cgroup/memory/docker/{ίϯςφID}/memory.stat cache  761856   rss  25681920   rss_huge

     0   mapped_file  16384   pgpgin  12569   …   total_cache  761856   total_rss  25681920   …   ͍ΘΏΔ344STT NBQQFE@pMF
  9. memory.statৄࡉ • cache   • rss   • mapped_file  

    • pgpgin   • pgpgout   • swap   • active_anon   • inactive_anon   • active_file   • inactive_file   • unevictable   • hierarchical_memory _limit   • hierarchical_memsw   • total_͸ࢠάϧʔϓΛ ؚΉ஋ ৄࡉIUUQTXXXLFSOFMPSHEPD%PDVNFOUBUJPODHSPVQTNFNPSZUYU
  10. memoryؔ࿈(ͦͷଞ) memory.usage_in_bytes ϝϞϦ࢖༻ྔ memory.memsw.usage_in_bytes ϝϞϦ࢖༻ྔͱεϫοϓྖҬͷ࿨ memory.max_usage_in_bytes ϝϞϦ࠷େ࢖༻ྔ memory.memsw.max_usage_in_bytes εϫοϓ࠷େ࢖༻ྔ memory.limit_in_bytes

    ϝϞϦͷ੍ݶ஋ memory.memsw.limit_in_bytes ϝϞϦͱεϫοϓ੍ݶ஋ memory.failcnt ϝϞϦ੍ݶ஋ʹୡͨ͠ճ਺ memory.memsw.failcnt ϝϞϦ੍ݶ஋ʹୡͨ͠ճ਺
  11. ڞ௨Ϩϙʔτ • tasks ࣮ߦதͷϓϩηεͷҰཡ(PID) • cgroup.procs ࣮ߦதͷεϨουάϧʔϓͷҰཡ(TGID) %  cat  /sys/fs/cgroup/memory/docker/…./cgroup.procs

      31218   31307   %  ps  aux  |  egrep  '(31218|31307)'   root          31218    0.0    0.0      4408      164  ?                Ss      13:15      0:00  /bin/sh  -­‐ c  rackup  -­‐p4567   root          31307    0.0    5.2  281364  26336  ?                Sl      13:15      0:06  ruby1.9.1   /usr/local/bin/rackup  -­‐p4567
  12. 2 Docker Patterns (with Docker Hub) & Deployment

  13. ϗετͱίϯςφͷؔ܎ • 1:n • CIαʔόɾ։ൃαʔόͳͲ • 1:1 • ϓϩμΫγϣϯ؀ڥͳͲ

  14. Pattern 1:n • CIαʔόɾ։ൃαʔό • ίϯςφԾ૝Խ؀ڥΛଟ਺ಈ࡞ • ಠཱͨ͠؀ڥ • ىಈఀࢭ͕ߴ଎

    • Ϧιʔεͷڞ༗ʹΑΔޮ཰Խ
  15. Pattern 1:1 • ίϯςφԾ૝؀ڥΛҰͭͷΈಈ࡞ • ϦιʔεΛಠ઎ • ؀ڥߏங໘ʹಛԽ • ϓϩμΫγϣϯ؀ڥʹ߹͏

    • AWS BeanstalkͷDocker support
  16. Deployment with Docker Hub

  17. Deployment (JU)VC %PDLFS)VC IVCPU

  18. GitHub

  19. Docker Hub

  20. Automated Build

  21. Build Success

  22. Build Failed

  23. Webhook from Docker Hub {        "push_data":{  

               "pushed_at":1385141110,              "images":[                    "imagehash1"            ],              "pusher":"username"        },        "repository":{              "status":"Active",              "description":"my  docker  repo  that  does  cool  things",              "is_trusted":false,              "full_description":"This  is  my  full  description",              "repo_url":"https://registry.hub.docker.com/u/username/reponame/",
  24. hubot • νϟοτπʔϧͰσϓϩΠ • Build௨஌ ! • σϓϩΠ࣮ߦ

  25. 3 Monitoring Docker with Mackerel

  26. Mackerel https://mackerel.io/

  27. Mackerelʢ೔ຊޠͰḉʣ • αʔό؅ཧπʔϧ as a Service • લ਎: ͸ͯͳͷࣾ಺πʔϧ

  28. ಛ௃ • ʮ໾ׂʯʹΑΔϗετ܈ͷ؅ཧ • ΤʔδΣϯτʹΑΔϦιʔε؂ࢹ • API ʹΑΔ؅ཧ ϩ ʔ

    ϧ
  29. αʔϏεͱϩʔϧ • αʔϏε • ͞·͟·ͳϗετ͕ڠௐ • Hatena-Bookmark, Hatena-Blog, … •

    ϗετ܈Λ໾ׂͰ·ͱΊΔ = ϩʔϧ • app, db, …
  30. ϩʔϧຖͷάϥϑͰՄࢹԽ

  31. mackerel-agent • Go Ͱॻ͔ΕͨެࣜΤʔδΣϯτ • Linux αϙʔτ (CentOS 5/6, Debian

    6/7) • ϗετεϖοΫͱϝτϦοΫΛऩू mackerelio/mackerel-agent
  32. αʔϏεɾϩʔϧͷࢦఆ • agentͷઃఆ͔ΒαʔϏεɾϩʔϧΛ ࢦఆ #  mackerel-­‐agent.conf   roles  =  [

     “Mackerel:db”,  …  ]
  33. agent: ΧελϜϝτϦοΫ • Sensu ϓϥάΠϯޓ׵ͷग़ྗ
 ʢλϒ۠੾Γʣ %  ~/.mackerel-­‐agent/plugins/chrome-­‐tabs-­‐count.sh   browser.tabs.count

               9              1403153173   browser.tabs.works            7              1403153173
  34. mackerel-docker • ϗετଆ͔ΒDockerίϯςφΛϞχλϦϯά ! • mackerel-agentͷઃఆ IUUQTHJUIVCDPNTUBOBLBNBDLFSFMEPDLFS %  cat  mackerel-­‐agent.conf


    [plugin.metrics.docker]
 command  =  "ruby  /home/stanaka/git/mackerel-­‐docker/mackerel-­‐ docker.rb"
  35. Dockerίϯςφ΁ͷ૊ࠐΈ • Dockerfile ! ! ! ! • Dockerίϯςφͷىಈ CMD

     echo  APIKEY=\"$APIKEY\"  >  /etc/default/mackerel-­‐agent  
 &&  echo  OTHER_OPTS="-­‐role=sample:${ROLE:-­‐app2}"  >>  /etc/default/mackerel-­‐ agent  
 &&  /etc/init.d/mackerel-­‐agent  start  
 &&  env  MACKEREL_APIKEY=$APIKEY  mkr  host  status  -­‐-­‐host-­‐id  `cat  /var/lib/ mackerel-­‐agent/id`  -­‐-­‐status  working  
 &&  rackup  -­‐p4567 docker  run  -­‐e  “APIKEY=……”  stanaka/meetup
  36. Docker pluginʹΑΔϞχλϦϯά

  37. DockerίϯςφͷCPUάϥϑ

  38. Blue-green DeploymentରԠ • ϩʔϧ୯ҐͰͷՄࢹԽ • ୀ໾ࡁΈίϯςφͷάϥϑ΋อ࣋ Deployment

  39. Q & A