Blue-Green Deployment with ECS and monitoring

by Shinji Tanaka

Slide 1

Slide 1 text

Blue-Green Deployment with ECS and monitoring Hatena Co., Ltd. Shinji Tanaka @stanaka

Slide 2

Slide 2 text

Shinji Tanaka / ాத৻࢘ @stanaka / id:stanaka גࣜձࣾ͸ͯͳ ࣥߦ໾һCTO 2

Slide 3

Slide 3 text

͋͋ ࠷ۙɺిࢠ൛΋ग़ͨΑ͏Ͱ͢

Slide 4

Slide 4 text

Agenda • Blue-Green Deployment with AWS ECS • Docker (cgroup) Statics • Monitoring Docker with Mackerel

Slide 5

Slide 5 text

1 Blue-Green Deployment with AWS ECS

Slide 6

Slide 6 text

Immutable Infrastructure http://chadfowler.com/blog/2013/06/23/immutable-‐deployments/

Slide 7

Slide 7 text

Immutable Infrastructure • Immutable server • Stateless / Disposable • ઃఆΛมߋ͢Δࡍ͸࠶࡞੒͢Δ • ίʔυσϓϩΠ΋ؚΉ

Slide 8

Slide 8 text

Blue-Green Deployment Host Host Host AWS Host Host Host Load Balancer Dispose Old hosts Create New hosts

Slide 9

Slide 9 text

Pattern 1 • 2 Autoscaling Group • as ECS Cluster • Blue / Green • TaskΛ֤άϧʔϓʹσϓϩΠ

Slide 10

Slide 10 text

Pattern 1 • ελϯόΠʹ৽͍͠AppΛσϓϩΠ • ELBͰάϧʔϓΛ੾Γସ͑ • Ұ࣌తʹ྆ํͷάϧʔϓʹ઀ଓ͢Δ

Slide 11

Slide 11 text

Pattern 2 • 1 Autoscaling Group • 2 ECS Tasks • Blue / Green Auto Scaling Group

Slide 12

Slide 12 text

Pattern 2 • ৽͍͠ίʔυΛ৽Taskͱͯ͠σϓϩΠ • Delete/CreateLoadBalancerListeners • Ұॠɺ Listener ͕ͳ͘ͳΔ { "containerDefinitions": [ … "portMappings": [ { "containerPort": 80, "hostPort": 80 } … ϙʔτΛͣΒ͢

Slide 13

Slide 13 text

Pros/Cons of Pattern 1 • ҆શɾ࣮֬ͳ੾Γସ͑ • Ұ࣌తʹ྆ํͷάϧʔϓʹ઀ଓ͢Δ • Ϧιʔεޮ཰͸ѱ͍ • Auto Scaling Group͕ཁஆؾ

Slide 14

Slide 14 text

Pros/Cons of Pattern 2 • Ϧιʔεޮ཰͕ྑ͍ • ಉ͡ΠϯελϯεͰಈ͔ͤΔ • Delete/CreateLoadBalancerListners  ͕ΞτϛοΫͳૢ࡞Ͱ͸ͳ͍

Slide 15

Slide 15 text

Pattern 2’ • ֤ΠϯελϯεʹNginx(HAProxy)Λ ഑ஔ • ϙʔτϑΥϫʔμʔ • ৽چΛΞτϛοΫʹ੾Γସ͑Մ • શΠϯελϯεʹૢ࡞͕ඞཁ

Slide 16

Slide 16 text

2 Monitoring ECS with Mackerel

Slide 17

Slide 17 text

2-a Docker (cgroup) Statistics

Slide 18

Slide 18 text

DockerͷϞχλϦϯάͷجຊ • cgroupͷ࡞๏ʹै͏ • ίϯςφଆ͔Β͸ɺϗετશମ͔͠ ݟ͑ͳ͍ • /proc/loadavg, /proc/stat • ݟ͍ͨ৔߹͸`-‐v`ͰऔΓࠐΉ

Slide 19

Slide 19 text

ϗετଆ͔ΒͷϞχλϦϯά • cgroupͷϨϙʔτΛར༻ • /sys/fs/cgroup/  {cpuacct, memory} • ଞʹblkio, cpu, devicesͳͲͷα ϒγεςϜ͕͋Δ(֤छ੍ݶͳͲͷ੍ ޚ༻) VCVOUVͷ৔߹  LFSOFMHFOFSJDEPDLFS

Slide 20

Slide 20 text

CPU࢖༻཰ • /sys/fs/cgroup/cpuacct/docker/{ίϯςφID}/cpuacct.stat • user … ϢʔβʔϞʔυͷCPU ࣌ؒ • system … γεςϜϞʔυͷCPU ࣌ؒ • CPU࣌ؒͷ୯Ґ: USER_HZ (σϑΥϧτ͸10ms) user 480 system 116

Slide 21

Slide 21 text

cpuacctͷଞͷࢦඪ • cpuacct.usage: ૯CPU࣌ؒ (φϊඵ)  echo 0 > /cgroup/cpuacct/cpuacct.usage   ͰϦηοτ • cpuacct.usage_percpu: ֤CPUͷCPU ࣌ؒ (φϊඵ)

Slide 22

Slide 22 text

ϝϞϦ࢖༻཰ • /sys/fs/cgroup/memory/docker/{ίϯςφID}/memory.stat cache 761856 rss 25681920 rss_huge 0 mapped_file 16384 pgpgin 12569 … total_cache 761856 total_rss 25681920 … ͍ΘΏΔ344STTNBQQFE@pMF

Slide 23

Slide 23 text

memory.statৄࡉ • cache • rss • mapped_file • pgpgin • pgpgout • swap • active_anon • inactive_anon • active_file • inactive_file • unevictable • hierarchical_memory _limit • hierarchical_memsw • total_͸ࢠάϧʔϓΛ ؚΉ஋ ৄࡉIUUQTXXXLFSOFMPSHEPD%PDVNFOUBUJPODHSPVQTNFNPSZUYU

Slide 24

Slide 24 text

memoryؔ࿈(ͦͷଞ) memory.usage_in_bytes ϝϞϦ࢖༻ྔ memory.memsw.usage_in_bytes ϝϞϦ࢖༻ྔͱεϫοϓྖҬͷ࿨ memory.max_usage_in_bytes ϝϞϦ࠷େ࢖༻ྔ memory.memsw.max_usage_in_bytes εϫοϓ࠷େ࢖༻ྔ memory.limit_in_bytes ϝϞϦͷ੍ݶ஋ memory.memsw.limit_in_bytes ϝϞϦͱεϫοϓ੍ݶ஋ memory.failcnt ϝϞϦ੍ݶ஋ʹୡͨ͠ճ਺ memory.memsw.failcnt ϝϞϦ੍ݶ஋ʹୡͨ͠ճ਺

Slide 25

Slide 25 text

ڞ௨Ϩϙʔτ • tasks ࣮ߦதͷϓϩηεͷҰཡ(PID) • cgroup.procs ࣮ߦதͷεϨουάϧʔϓͷҰཡ(TGID) % cat /sys/fs/cgroup/memory/docker/…./cgroup.procs 31218 31307 % ps aux | egrep '(31218|31307)' root 31218 0.0 0.0 4408 164 ? Ss 13:15 0:00 /bin/sh -‐ c rackup -‐p4567 root 31307 0.0 5.2 281364 26336 ? Sl 13:15 0:06 ruby1.9.1 /usr/local/bin/rackup -‐p4567

Slide 26

Slide 26 text

2-b Monitoring Docker with Mackerel

Slide 27

Slide 27 text

Mackerel https://mackerel.io/

Slide 28

Slide 28 text

ಛ௃ • ʮ໾ׂʯʹΑΔϗετ܈ͷ؅ཧ • ΤʔδΣϯτʹΑΔϦιʔε؂ࢹ ϩ ʔ ϧ

Slide 29

Slide 29 text

ϩʔϧຖͷάϥϑͰՄࢹԽ

Slide 30

Slide 30 text

mackerel-agent • Go Ͱॻ͔ΕͨެࣜΤʔδΣϯτ • Linux / Windows αϙʔτ • ϗετεϖοΫͱϝτϦοΫΛऩू mackerelio/mackerel-agent

Slide 31

Slide 31 text

Blue-green DeploymentରԠ • ϩʔϧ୯ҐͰͷՄࢹԽ • ୀ໾ࡁΈίϯςφͷάϥϑ΋อ࣋ Deployment

Slide 32

Slide 32 text

ΦʔτεέʔϧରԠ • αʔό͕ॖୀͯ͠΋άϥϑ͸࢒Δ

Slide 33

Slide 33 text

Raspberry Pi Ͱ΋ಈ͖·͢ • Թ౓ɾ࣪౓ηϯα → Slack௨஌

Slide 34

Slide 34 text

Docker  with Mackerel

Slide 35

Slide 35 text

mackerel/mackerel-agent on Docker Hub https://registry.hub.docker.com/u/stanaka/mackerel-‐agent/

Slide 36

Slide 36 text

How to Use • ίϚϯυҰൃ • ඞཁͳύεɾAPIKEYΛ౉͢ docker run -‐h `hostname` \ -‐v /var/run/docker.sock:/var/run/docker.sock \ -‐v /var/lib/mackerel-‐agent/:/var/lib/mackerel-‐agent/ \ -‐v /proc/mounts:/host/proc/mounts:ro \ -‐v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -‐e 'apikey=‘ \ mackerel/mackerel-‐agent

Slide 37

Slide 37 text

DockerFile(ൈਮ) FROM debian:jessie # setup mackerel-‐agent RUN apt-‐get update \ .. && apt-‐get -‐y install mackerel-‐agent \ .. # setup docker plugin COPY mackerel-‐plugin-‐docker.rb /mackerel-‐plugin-‐docker.rb RUN echo [plugin.metrics.docker] \ >> /etc/mackerel-‐agent/mackerel-‐agent.conf RUN echo command = \"ruby /mackerel-‐plugin-‐docker.rb\" \ >> /etc/mackerel-‐agent/mackerel-‐agent.conf # boot mackerel-‐agent CMD /startup.sh

Slide 38

Slide 38 text

֤ίϯςφͷ৘ใΛऩू )PTU qVFOUE NBDLFSFM BHFOU "QQ

Slide 39

Slide 39 text

༷ࢠ(ίϯςφຖͷCPU)

Slide 40

Slide 40 text

ίϯςφͷத਎ͷ ৘ใΛऩू͍ͨ͠

Slide 41

Slide 41 text

ίϯςφͷத਎ͷ৘ใΛऩू )PTU NBDLFSFM BHFOU "QQ

Slide 42

Slide 42 text

Memcachedͷྫ % docker run -‐d -‐P \ -‐-‐name memcached -‐p 11211:11211 \ sylvainlasnier/memcached )PTU NBDLFSFM BHFOU NFNDBDIFE

Slide 43

Slide 43 text

linkΦϓγϣϯͰ઀ଓ % docker run -‐i -‐t -‐h `hostname` \ -‐v /var/run/docker.sock:/var/run/docker.sock \ -‐v /proc/mounts:/host/proc/mounts:ro \ -‐v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \ -‐e 'apikey=' -‐-‐link memcached:memcached \ mackerel-‐agent /bin/bash root@dev:/# env HOSTNAME=develop04.host.h MEMCACHED_PORT_11211_TCP_PORT=11211 MEMCACHED_PORT_11211_TCP_PROTO=tcp MEMCACHED_NAME=/tender_curie/memcached MEMCACHED_PORT=tcp://172.17.0.88:11211 MEMCACHED_PORT_11211_TCP_ADDR=172.17.0.88 MEMCACHED_PORT_11211_TCP=tcp://172.17.0.88:11211 ݱঢ়͸Pattern2’ͰରԠ

Slide 44

Slide 44 text

؀ڥม਺ͰΞυϨεࢦఆ root@dev:/# /usr/local/bin/mackerel-‐plugin-‐memcached \ -‐host=$MEMCACHED_PORT_11211_TCP_ADDR memcached.connections.curr_connections 10 1427507503 memcached.cmd.cmd_get 0 1427507503 memcached.cmd.cmd_set 0 1427507503

Slide 45

Slide 45 text

ϓϥάΠϯઃఆΛ࢓ࠐΉ % docker run -‐h `hostname` \ -‐v /var/run/docker.sock:/var/run/docker.sock \ -‐v /var/lib/mackerel-‐agent/:/var/lib/mackerel-‐agent/ \ -‐v /proc/mounts:/host/proc/mounts:ro \ -‐v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -‐e 'apikey=‘ \ -‐-‐link memcached:memcached \ -‐v /etc/mackerel-‐agent/conf.d:/etc/mackerel-‐agent/conf.d:ro -‐e ‘include=/etc/mackerel-‐agent/conf.d/*' \ mackerel/mackerel-‐agent % cat /etc/mackerel-‐agent/conf.d/memcached.conf [plugin.metrics.memcached] command = "/usr/local/bin/mackerel-‐plugin-‐memcached -‐ host=$MEMCACHED_PORT_11211_TCP_ADDR"

Slide 46

Slide 46 text

MemcachedͷϝτϦοΫ

Slide 47

Slide 47 text

·ͱΊ • ECSͰBlue-Green Deployment • Patten 1, 2, 2’ • Monitoring Docker  with Mackerel