Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Blue-Green Deployment with ECS and monitoring

Shinji Tanaka
September 14, 2015

Blue-Green Deployment with ECS and monitoring

JAWS コンテナ支部 #2

Shinji Tanaka

September 14, 2015
Tweet

More Decks by Shinji Tanaka

Other Decks in Technology

Transcript

  1. Blue-Green
    Deployment with ECS
    and monitoring
    Hatena Co., Ltd.
    Shinji Tanaka @stanaka

    View Slide

  2. Shinji Tanaka / ాத৻࢘
    @stanaka / id:stanaka
    גࣜձࣾ͸ͯͳ ࣥߦ໾һCTO
    2

    View Slide


  3. ͋͋
    ࠷ۙɺిࢠ൛΋ग़ͨΑ͏Ͱ͢

    View Slide

  4. Agenda
    • Blue-Green Deployment with AWS ECS
    • Docker (cgroup) Statics
    • Monitoring Docker with Mackerel

    View Slide

  5. 1
    Blue-Green Deployment
    with AWS ECS

    View Slide

  6. Immutable Infrastructure
    http://chadfowler.com/blog/2013/06/23/immutable-­‐deployments/

    View Slide

  7. Immutable Infrastructure
    • Immutable server
    • Stateless / Disposable
    • ઃఆΛมߋ͢Δࡍ͸࠶࡞੒͢Δ
    • ίʔυσϓϩΠ΋ؚΉ

    View Slide

  8. Blue-Green Deployment
    Host Host Host
    AWS
    Host Host Host
    Load Balancer
    Dispose
    Old hosts
    Create
    New hosts

    View Slide

  9. Pattern 1
    • 2 Autoscaling Group
    • as ECS Cluster
    • Blue / Green
    • TaskΛ֤άϧʔϓʹσϓϩΠ

    View Slide

  10. Pattern 1
    • ελϯόΠʹ৽͍͠AppΛσϓϩΠ
    • ELBͰάϧʔϓΛ੾Γସ͑
    • Ұ࣌తʹ྆ํͷάϧʔϓʹ઀ଓ͢Δ

    View Slide

  11. Pattern 2
    • 1 Autoscaling Group
    • 2 ECS Tasks
    • Blue / Green
    Auto Scaling Group

    View Slide

  12. Pattern 2
    • ৽͍͠ίʔυΛ৽Taskͱͯ͠σϓϩΠ
    • Delete/CreateLoadBalancerListeners
    • Ұॠɺ Listener ͕ͳ͘ͳΔ
    {  "containerDefinitions":  [  
    …  
             "portMappings":  [  {  
                       "containerPort":  80,  
                       "hostPort":  80  
                   }  

    ϙʔτΛͣΒ͢

    View Slide

  13. Pros/Cons of Pattern 1
    • ҆શɾ࣮֬ͳ੾Γସ͑
    • Ұ࣌తʹ྆ํͷάϧʔϓʹ઀ଓ͢Δ
    • Ϧιʔεޮ཰͸ѱ͍
    • Auto Scaling Group͕ཁஆؾ

    View Slide

  14. Pros/Cons of Pattern 2
    • Ϧιʔεޮ཰͕ྑ͍
    • ಉ͡ΠϯελϯεͰಈ͔ͤΔ
    • Delete/CreateLoadBalancerListners

    ͕ΞτϛοΫͳૢ࡞Ͱ͸ͳ͍

    View Slide

  15. Pattern 2’
    • ֤ΠϯελϯεʹNginx(HAProxy)Λ
    ഑ஔ
    • ϙʔτϑΥϫʔμʔ
    • ৽چΛΞτϛοΫʹ੾Γସ͑Մ
    • શΠϯελϯεʹૢ࡞͕ඞཁ

    View Slide

  16. 2
    Monitoring ECS
    with Mackerel

    View Slide

  17. 2-a
    Docker (cgroup)
    Statistics

    View Slide

  18. DockerͷϞχλϦϯάͷجຊ
    • cgroupͷ࡞๏ʹै͏
    • ίϯςφଆ͔Β͸ɺϗετશମ͔͠
    ݟ͑ͳ͍
    • /proc/loadavg,  /proc/stat  
    • ݟ͍ͨ৔߹͸`-­‐v`ͰऔΓࠐΉ

    View Slide

  19. ϗετଆ͔ΒͷϞχλϦϯά
    • cgroupͷϨϙʔτΛར༻
    • /sys/fs/cgroup/

             {cpuacct,  memory}  
    • ଞʹblkio,  cpu,  devicesͳͲͷα
    ϒγεςϜ͕͋Δ(֤छ੍ݶͳͲͷ੍
    ޚ༻)
    VCVOUVͷ৔߹

    LFSOFMHFOFSJDEPDLFS

    View Slide

  20. CPU࢖༻཰
    • /sys/fs/cgroup/cpuacct/docker/{ίϯςφID}/cpuacct.stat
    • user … ϢʔβʔϞʔυͷCPU ࣌ؒ
    • system … γεςϜϞʔυͷCPU ࣌ؒ
    • CPU࣌ؒͷ୯Ґ: USER_HZ (σϑΥϧτ͸10ms)
    user  480  
    system  116

    View Slide

  21. cpuacctͷଞͷࢦඪ
    • cpuacct.usage: ૯CPU࣌ؒ (φϊඵ)

    echo 0 > /cgroup/cpuacct/cpuacct.usage 

    ͰϦηοτ
    • cpuacct.usage_percpu: ֤CPUͷCPU ࣌ؒ (φϊඵ)

    View Slide

  22. ϝϞϦ࢖༻཰
    • /sys/fs/cgroup/memory/docker/{ίϯςφID}/memory.stat
    cache  761856  
    rss  25681920  
    rss_huge  0  
    mapped_file  16384  
    pgpgin  12569  
    …  
    total_cache  761856  
    total_rss  25681920  
    …  
    ͍ΘΏΔ[email protected]

    View Slide

  23. memory.statৄࡉ
    • cache  
    • rss  
    • mapped_file  
    • pgpgin  
    • pgpgout  
    • swap  
    • active_anon  
    • inactive_anon  
    • active_file  
    • inactive_file  
    • unevictable  
    • hierarchical_memory
    _limit  
    • hierarchical_memsw  
    • total_͸ࢠάϧʔϓΛ
    ؚΉ஋
    ৄࡉIUUQTXXXLFSOFMPSHEPD%PDVNFOUBUJPODHSPVQTNFNPSZUYU

    View Slide

  24. memoryؔ࿈(ͦͷଞ)
    memory.usage_in_bytes ϝϞϦ࢖༻ྔ
    memory.memsw.usage_in_bytes ϝϞϦ࢖༻ྔͱεϫοϓྖҬͷ࿨
    memory.max_usage_in_bytes ϝϞϦ࠷େ࢖༻ྔ
    memory.memsw.max_usage_in_bytes εϫοϓ࠷େ࢖༻ྔ
    memory.limit_in_bytes ϝϞϦͷ੍ݶ஋
    memory.memsw.limit_in_bytes ϝϞϦͱεϫοϓ੍ݶ஋
    memory.failcnt ϝϞϦ੍ݶ஋ʹୡͨ͠ճ਺
    memory.memsw.failcnt ϝϞϦ੍ݶ஋ʹୡͨ͠ճ਺

    View Slide

  25. ڞ௨Ϩϙʔτ
    • tasks ࣮ߦதͷϓϩηεͷҰཡ(PID)
    • cgroup.procs ࣮ߦதͷεϨουάϧʔϓͷҰཡ(TGID)
    %  cat  /sys/fs/cgroup/memory/docker/…./cgroup.procs  
    31218  
    31307  
    %  ps  aux  |  egrep  '(31218|31307)'  
    root          31218    0.0    0.0      4408      164  ?                Ss      13:15      0:00  /bin/sh  -­‐
    c  rackup  -­‐p4567  
    root          31307    0.0    5.2  281364  26336  ?                Sl      13:15      0:06  ruby1.9.1  
    /usr/local/bin/rackup  -­‐p4567

    View Slide

  26. 2-b
    Monitoring Docker
    with Mackerel

    View Slide

  27. Mackerel
    https://mackerel.io/

    View Slide

  28. ಛ௃
    • ʮ໾ׂʯʹΑΔϗετ܈ͷ؅ཧ
    • ΤʔδΣϯτʹΑΔϦιʔε؂ࢹ
    ϩ ʔ ϧ

    View Slide

  29. ϩʔϧຖͷάϥϑͰՄࢹԽ

    View Slide

  30. mackerel-agent
    • Go Ͱॻ͔ΕͨެࣜΤʔδΣϯτ
    • Linux / Windows αϙʔτ
    • ϗετεϖοΫͱϝτϦοΫΛऩू
    mackerelio/mackerel-agent

    View Slide

  31. Blue-green DeploymentରԠ
    • ϩʔϧ୯ҐͰͷՄࢹԽ
    • ୀ໾ࡁΈίϯςφͷάϥϑ΋อ࣋
    Deployment

    View Slide

  32. ΦʔτεέʔϧରԠ
    • αʔό͕ॖୀͯ͠΋άϥϑ͸࢒Δ

    View Slide

  33. Raspberry Pi Ͱ΋ಈ͖·͢
    • Թ౓ɾ࣪౓ηϯα → Slack௨஌

    View Slide

  34. Docker

    with Mackerel

    View Slide

  35. mackerel/mackerel-agent
    on Docker Hub
    https://registry.hub.docker.com/u/stanaka/mackerel-­‐agent/

    View Slide

  36. How to Use
    • ίϚϯυҰൃ
    • ඞཁͳύεɾAPIKEYΛ౉͢
    docker  run  -­‐h  `hostname`  \  
       -­‐v  /var/run/docker.sock:/var/run/docker.sock  \  
       -­‐v  /var/lib/mackerel-­‐agent/:/var/lib/mackerel-­‐agent/  \  
       -­‐v  /proc/mounts:/host/proc/mounts:ro  \  
       -­‐v  /sys/fs/cgroup/:/host/sys/fs/cgroup:ro    
       -­‐e  'apikey=‘  \  
       mackerel/mackerel-­‐agent  

    View Slide

  37. DockerFile(ൈਮ)
    FROM  debian:jessie  
    #  setup  mackerel-­‐agent  
    RUN  apt-­‐get  update  \  
    ..  
       &&  apt-­‐get  -­‐y  install  mackerel-­‐agent  \  
    ..  
    #  setup  docker  plugin  
    COPY  mackerel-­‐plugin-­‐docker.rb  /mackerel-­‐plugin-­‐docker.rb  
    RUN  echo  [plugin.metrics.docker]  \  
       >>  /etc/mackerel-­‐agent/mackerel-­‐agent.conf  
    RUN  echo  command  =  \"ruby  /mackerel-­‐plugin-­‐docker.rb\"  \  
       >>  /etc/mackerel-­‐agent/mackerel-­‐agent.conf  
    #  boot  mackerel-­‐agent  
    CMD  /startup.sh

    View Slide

  38. ֤ίϯςφͷ৘ใΛऩू
    )PTU
    qVFOUE
    NBDLFSFM
    BHFOU
    "QQ

    View Slide

  39. ༷ࢠ(ίϯςφຖͷCPU)

    View Slide

  40. ίϯςφͷத਎ͷ
    ৘ใΛऩू͍ͨ͠

    View Slide

  41. ίϯςφͷத਎ͷ৘ใΛऩू
    )PTU
    NBDLFSFM
    BHFOU
    "QQ

    View Slide

  42. Memcachedͷྫ
    %  docker  run  -­‐d  -­‐P  \    
       -­‐-­‐name  memcached  -­‐p  11211:11211  \  
       sylvainlasnier/memcached
    )PTU
    NBDLFSFM
    BHFOU
    NFNDBDIFE

    View Slide

  43. linkΦϓγϣϯͰ઀ଓ
    %  docker  run  -­‐i  -­‐t  -­‐h  `hostname`  \  
       -­‐v  /var/run/docker.sock:/var/run/docker.sock  \  
       -­‐v  /proc/mounts:/host/proc/mounts:ro  \  
       -­‐v  /sys/fs/cgroup/:/host/sys/fs/cgroup:ro  \  
       -­‐e  'apikey='  -­‐-­‐link  memcached:memcached  \  
       mackerel-­‐agent  /bin/bash  
    [email protected]:/#  env  
    HOSTNAME=develop04.host.h  
    MEMCACHED_PORT_11211_TCP_PORT=11211  
    MEMCACHED_PORT_11211_TCP_PROTO=tcp  
    MEMCACHED_NAME=/tender_curie/memcached  
    MEMCACHED_PORT=tcp://172.17.0.88:11211  
    MEMCACHED_PORT_11211_TCP_ADDR=172.17.0.88  
    MEMCACHED_PORT_11211_TCP=tcp://172.17.0.88:11211
    ݱঢ়͸Pattern2’ͰରԠ

    View Slide

  44. ؀ڥม਺ͰΞυϨεࢦఆ
    [email protected]:/#  /usr/local/bin/mackerel-­‐plugin-­‐memcached  \  
       -­‐host=$MEMCACHED_PORT_11211_TCP_ADDR  
    memcached.connections.curr_connections    10            1427507503  
    memcached.cmd.cmd_get      0              1427507503  
    memcached.cmd.cmd_set      0              1427507503

    View Slide

  45. ϓϥάΠϯઃఆΛ࢓ࠐΉ
    %  docker  run  -­‐h  `hostname`  \  
       -­‐v  /var/run/docker.sock:/var/run/docker.sock  \  
       -­‐v  /var/lib/mackerel-­‐agent/:/var/lib/mackerel-­‐agent/  \  
       -­‐v  /proc/mounts:/host/proc/mounts:ro  \  
       -­‐v  /sys/fs/cgroup/:/host/sys/fs/cgroup:ro    
       -­‐e  'apikey=‘  \  
       -­‐-­‐link  memcached:memcached  \  
       -­‐v  /etc/mackerel-­‐agent/conf.d:/etc/mackerel-­‐agent/conf.d:ro  
       -­‐e  ‘include=/etc/mackerel-­‐agent/conf.d/*'  \  
       mackerel/mackerel-­‐agent
    %  cat  /etc/mackerel-­‐agent/conf.d/memcached.conf  
    [plugin.metrics.memcached]  
    command  =  "/usr/local/bin/mackerel-­‐plugin-­‐memcached  -­‐
    host=$MEMCACHED_PORT_11211_TCP_ADDR"

    View Slide

  46. MemcachedͷϝτϦοΫ

    View Slide

  47. ·ͱΊ
    • ECSͰBlue-Green Deployment
    • Patten 1, 2, 2’
    • Monitoring Docker

    with Mackerel

    View Slide