Blue-Green Deployment with ECS and monitoring

A76985bd77fecba82109dee506b1fec2?s=47 Shinji Tanaka
September 14, 2015

Blue-Green Deployment with ECS and monitoring

JAWS コンテナ支部 #2

A76985bd77fecba82109dee506b1fec2?s=128

Shinji Tanaka

September 14, 2015
Tweet

Transcript

  1. Blue-Green Deployment with ECS and monitoring Hatena Co., Ltd. Shinji

    Tanaka @stanaka
  2. Shinji Tanaka / ాத৻࢘ @stanaka / id:stanaka גࣜձࣾ͸ͯͳ ࣥߦ໾һCTO 2

  3.  ͋͋ ࠷ۙɺిࢠ൛΋ग़ͨΑ͏Ͱ͢

  4. Agenda • Blue-Green Deployment with AWS ECS • Docker (cgroup)

    Statics • Monitoring Docker with Mackerel
  5. 1 Blue-Green Deployment with AWS ECS

  6. Immutable Infrastructure http://chadfowler.com/blog/2013/06/23/immutable-­‐deployments/

  7. Immutable Infrastructure • Immutable server • Stateless / Disposable •

    ઃఆΛมߋ͢Δࡍ͸࠶࡞੒͢Δ • ίʔυσϓϩΠ΋ؚΉ
  8. Blue-Green Deployment Host Host Host AWS Host Host Host Load

    Balancer Dispose Old hosts Create New hosts
  9. Pattern 1 • 2 Autoscaling Group • as ECS Cluster

    • Blue / Green • TaskΛ֤άϧʔϓʹσϓϩΠ
  10. Pattern 1 • ελϯόΠʹ৽͍͠AppΛσϓϩΠ • ELBͰάϧʔϓΛ੾Γସ͑ • Ұ࣌తʹ྆ํͷάϧʔϓʹ઀ଓ͢Δ

  11. Pattern 2 • 1 Autoscaling Group • 2 ECS Tasks

    • Blue / Green Auto Scaling Group
  12. Pattern 2 • ৽͍͠ίʔυΛ৽Taskͱͯ͠σϓϩΠ • Delete/CreateLoadBalancerListeners • Ұॠɺ Listener ͕ͳ͘ͳΔ

    {  "containerDefinitions":  [   …            "portMappings":  [  {                      "containerPort":  80,                      "hostPort":  80                  }   … ϙʔτΛͣΒ͢
  13. Pros/Cons of Pattern 1 • ҆શɾ࣮֬ͳ੾Γସ͑ • Ұ࣌తʹ྆ํͷάϧʔϓʹ઀ଓ͢Δ • Ϧιʔεޮ཰͸ѱ͍

    • Auto Scaling Group͕ཁஆؾ
  14. Pros/Cons of Pattern 2 • Ϧιʔεޮ཰͕ྑ͍ • ಉ͡ΠϯελϯεͰಈ͔ͤΔ • Delete/CreateLoadBalancerListners


    ͕ΞτϛοΫͳૢ࡞Ͱ͸ͳ͍
  15. Pattern 2’ • ֤ΠϯελϯεʹNginx(HAProxy)Λ ഑ஔ • ϙʔτϑΥϫʔμʔ • ৽چΛΞτϛοΫʹ੾Γସ͑Մ •

    શΠϯελϯεʹૢ࡞͕ඞཁ
  16. 2 Monitoring ECS with Mackerel

  17. 2-a Docker (cgroup) Statistics

  18. DockerͷϞχλϦϯάͷجຊ • cgroupͷ࡞๏ʹै͏ • ίϯςφଆ͔Β͸ɺϗετશମ͔͠ ݟ͑ͳ͍ • /proc/loadavg,  /proc/stat  

    • ݟ͍ͨ৔߹͸`-­‐v`ͰऔΓࠐΉ
  19. ϗετଆ͔ΒͷϞχλϦϯά • cgroupͷϨϙʔτΛར༻ • /sys/fs/cgroup/
          {cpuacct,

     memory}   • ଞʹblkio,  cpu,  devicesͳͲͷα ϒγεςϜ͕͋Δ(֤छ੍ݶͳͲͷ੍ ޚ༻) VCVOUVͷ৔߹
 LFSOFMHFOFSJDEPDLFS
  20. CPU࢖༻཰ • /sys/fs/cgroup/cpuacct/docker/{ίϯςφID}/cpuacct.stat • user … ϢʔβʔϞʔυͷCPU ࣌ؒ • system

    … γεςϜϞʔυͷCPU ࣌ؒ • CPU࣌ؒͷ୯Ґ: USER_HZ (σϑΥϧτ͸10ms) user  480   system  116
  21. cpuacctͷଞͷࢦඪ • cpuacct.usage: ૯CPU࣌ؒ (φϊඵ)
 echo 0 > /cgroup/cpuacct/cpuacct.usage 


    ͰϦηοτ • cpuacct.usage_percpu: ֤CPUͷCPU ࣌ؒ (φϊඵ)
  22. ϝϞϦ࢖༻཰ • /sys/fs/cgroup/memory/docker/{ίϯςφID}/memory.stat cache  761856   rss  25681920   rss_huge

     0   mapped_file  16384   pgpgin  12569   …   total_cache  761856   total_rss  25681920   …   ͍ΘΏΔ344STT NBQQFE@pMF
  23. memory.statৄࡉ • cache   • rss   • mapped_file  

    • pgpgin   • pgpgout   • swap   • active_anon   • inactive_anon   • active_file   • inactive_file   • unevictable   • hierarchical_memory _limit   • hierarchical_memsw   • total_͸ࢠάϧʔϓΛ ؚΉ஋ ৄࡉIUUQTXXXLFSOFMPSHEPD%PDVNFOUBUJPODHSPVQTNFNPSZUYU
  24. memoryؔ࿈(ͦͷଞ) memory.usage_in_bytes ϝϞϦ࢖༻ྔ memory.memsw.usage_in_bytes ϝϞϦ࢖༻ྔͱεϫοϓྖҬͷ࿨ memory.max_usage_in_bytes ϝϞϦ࠷େ࢖༻ྔ memory.memsw.max_usage_in_bytes εϫοϓ࠷େ࢖༻ྔ memory.limit_in_bytes

    ϝϞϦͷ੍ݶ஋ memory.memsw.limit_in_bytes ϝϞϦͱεϫοϓ੍ݶ஋ memory.failcnt ϝϞϦ੍ݶ஋ʹୡͨ͠ճ਺ memory.memsw.failcnt ϝϞϦ੍ݶ஋ʹୡͨ͠ճ਺
  25. ڞ௨Ϩϙʔτ • tasks ࣮ߦதͷϓϩηεͷҰཡ(PID) • cgroup.procs ࣮ߦதͷεϨουάϧʔϓͷҰཡ(TGID) %  cat  /sys/fs/cgroup/memory/docker/…./cgroup.procs

      31218   31307   %  ps  aux  |  egrep  '(31218|31307)'   root          31218    0.0    0.0      4408      164  ?                Ss      13:15      0:00  /bin/sh  -­‐ c  rackup  -­‐p4567   root          31307    0.0    5.2  281364  26336  ?                Sl      13:15      0:06  ruby1.9.1   /usr/local/bin/rackup  -­‐p4567
  26. 2-b Monitoring Docker with Mackerel

  27. Mackerel https://mackerel.io/

  28. ಛ௃ • ʮ໾ׂʯʹΑΔϗετ܈ͷ؅ཧ • ΤʔδΣϯτʹΑΔϦιʔε؂ࢹ ϩ ʔ ϧ

  29. ϩʔϧຖͷάϥϑͰՄࢹԽ

  30. mackerel-agent • Go Ͱॻ͔ΕͨެࣜΤʔδΣϯτ • Linux / Windows αϙʔτ •

    ϗετεϖοΫͱϝτϦοΫΛऩू mackerelio/mackerel-agent
  31. Blue-green DeploymentରԠ • ϩʔϧ୯ҐͰͷՄࢹԽ • ୀ໾ࡁΈίϯςφͷάϥϑ΋อ࣋ Deployment

  32. ΦʔτεέʔϧରԠ • αʔό͕ॖୀͯ͠΋άϥϑ͸࢒Δ

  33. Raspberry Pi Ͱ΋ಈ͖·͢ • Թ౓ɾ࣪౓ηϯα → Slack௨஌

  34. Docker
 with Mackerel

  35. mackerel/mackerel-agent on Docker Hub https://registry.hub.docker.com/u/stanaka/mackerel-­‐agent/

  36. How to Use • ίϚϯυҰൃ • ඞཁͳύεɾAPIKEYΛ౉͢ docker  run  -­‐h

     `hostname`  \      -­‐v  /var/run/docker.sock:/var/run/docker.sock  \      -­‐v  /var/lib/mackerel-­‐agent/:/var/lib/mackerel-­‐agent/  \      -­‐v  /proc/mounts:/host/proc/mounts:ro  \      -­‐v  /sys/fs/cgroup/:/host/sys/fs/cgroup:ro        -­‐e  'apikey=<APIKEY>‘  \      mackerel/mackerel-­‐agent  
  37. DockerFile(ൈਮ) FROM  debian:jessie   #  setup  mackerel-­‐agent   RUN  apt-­‐get

     update  \   ..      &&  apt-­‐get  -­‐y  install  mackerel-­‐agent  \   ..   #  setup  docker  plugin   COPY  mackerel-­‐plugin-­‐docker.rb  /mackerel-­‐plugin-­‐docker.rb   RUN  echo  [plugin.metrics.docker]  \      >>  /etc/mackerel-­‐agent/mackerel-­‐agent.conf   RUN  echo  command  =  \"ruby  /mackerel-­‐plugin-­‐docker.rb\"  \      >>  /etc/mackerel-­‐agent/mackerel-­‐agent.conf   #  boot  mackerel-­‐agent   CMD  /startup.sh
  38. ֤ίϯςφͷ৘ใΛऩू )PTU qVFOUE NBDLFSFM BHFOU "QQ

  39. ༷ࢠ(ίϯςφຖͷCPU)

  40. ίϯςφͷத਎ͷ ৘ใΛऩू͍ͨ͠

  41. ίϯςφͷத਎ͷ৘ใΛऩू )PTU NBDLFSFM BHFOU "QQ

  42. Memcachedͷྫ %  docker  run  -­‐d  -­‐P  \      

     -­‐-­‐name  memcached  -­‐p  11211:11211  \      sylvainlasnier/memcached )PTU NBDLFSFM BHFOU NFNDBDIFE
  43. linkΦϓγϣϯͰ઀ଓ %  docker  run  -­‐i  -­‐t  -­‐h  `hostname`  \  

       -­‐v  /var/run/docker.sock:/var/run/docker.sock  \      -­‐v  /proc/mounts:/host/proc/mounts:ro  \      -­‐v  /sys/fs/cgroup/:/host/sys/fs/cgroup:ro  \      -­‐e  'apikey=<APIKEY>'  -­‐-­‐link  memcached:memcached  \      mackerel-­‐agent  /bin/bash   root@dev:/#  env   HOSTNAME=develop04.host.h   MEMCACHED_PORT_11211_TCP_PORT=11211   MEMCACHED_PORT_11211_TCP_PROTO=tcp   MEMCACHED_NAME=/tender_curie/memcached   MEMCACHED_PORT=tcp://172.17.0.88:11211   MEMCACHED_PORT_11211_TCP_ADDR=172.17.0.88   MEMCACHED_PORT_11211_TCP=tcp://172.17.0.88:11211 ݱঢ়͸Pattern2’ͰରԠ
  44. ؀ڥม਺ͰΞυϨεࢦఆ root@dev:/#  /usr/local/bin/mackerel-­‐plugin-­‐memcached  \      -­‐host=$MEMCACHED_PORT_11211_TCP_ADDR   memcached.connections.curr_connections  

     10            1427507503   memcached.cmd.cmd_get      0              1427507503   memcached.cmd.cmd_set      0              1427507503
  45. ϓϥάΠϯઃఆΛ࢓ࠐΉ %  docker  run  -­‐h  `hostname`  \      -­‐v

     /var/run/docker.sock:/var/run/docker.sock  \      -­‐v  /var/lib/mackerel-­‐agent/:/var/lib/mackerel-­‐agent/  \      -­‐v  /proc/mounts:/host/proc/mounts:ro  \      -­‐v  /sys/fs/cgroup/:/host/sys/fs/cgroup:ro        -­‐e  'apikey=<APIKEY>‘  \      -­‐-­‐link  memcached:memcached  \      -­‐v  /etc/mackerel-­‐agent/conf.d:/etc/mackerel-­‐agent/conf.d:ro      -­‐e  ‘include=/etc/mackerel-­‐agent/conf.d/*'  \      mackerel/mackerel-­‐agent %  cat  /etc/mackerel-­‐agent/conf.d/memcached.conf   [plugin.metrics.memcached]   command  =  "/usr/local/bin/mackerel-­‐plugin-­‐memcached  -­‐ host=$MEMCACHED_PORT_11211_TCP_ADDR"
  46. MemcachedͷϝτϦοΫ

  47. ·ͱΊ • ECSͰBlue-Green Deployment • Patten 1, 2, 2’ •

    Monitoring Docker
 with Mackerel