JAWS コンテナ支部 #2
Blue-GreenDeployment with ECSand monitoringHatena Co., Ltd.Shinji Tanaka @stanaka
View Slide
Shinji Tanaka / ాத৻࢘@stanaka / id:stanakaגࣜձࣾͯͳ ࣥߦһCTO2
͋͋࠷ۙɺిࢠ൛ग़ͨΑ͏Ͱ͢
Agenda• Blue-Green Deployment with AWS ECS• Docker (cgroup) Statics• Monitoring Docker with Mackerel
1Blue-Green Deploymentwith AWS ECS
Immutable Infrastructurehttp://chadfowler.com/blog/2013/06/23/immutable-‐deployments/
Immutable Infrastructure• Immutable server• Stateless / Disposable• ઃఆΛมߋ͢Δࡍ࠶࡞͢Δ• ίʔυσϓϩΠؚΉ
Blue-Green DeploymentHost Host HostAWSHost Host HostLoad BalancerDisposeOld hostsCreateNew hosts
Pattern 1• 2 Autoscaling Group• as ECS Cluster• Blue / Green• TaskΛ֤άϧʔϓʹσϓϩΠ
Pattern 1• ελϯόΠʹ৽͍͠AppΛσϓϩΠ• ELBͰάϧʔϓΛΓସ͑• Ұ࣌తʹ྆ํͷάϧʔϓʹଓ͢Δ
Pattern 2• 1 Autoscaling Group• 2 ECS Tasks• Blue / GreenAuto Scaling Group
Pattern 2• ৽͍͠ίʔυΛ৽Taskͱͯ͠σϓϩΠ• Delete/CreateLoadBalancerListeners• Ұॠɺ Listener ͕ͳ͘ͳΔ{ "containerDefinitions": [ … "portMappings": [ { "containerPort": 80, "hostPort": 80 } …ϙʔτΛͣΒ͢
Pros/Cons of Pattern 1• ҆શɾ࣮֬ͳΓସ͑• Ұ࣌తʹ྆ํͷάϧʔϓʹଓ͢Δ• Ϧιʔεޮѱ͍• Auto Scaling Group͕ཁஆؾ
Pros/Cons of Pattern 2• Ϧιʔεޮ͕ྑ͍• ಉ͡ΠϯελϯεͰಈ͔ͤΔ• Delete/CreateLoadBalancerListners ͕ΞτϛοΫͳૢ࡞Ͱͳ͍
Pattern 2’• ֤ΠϯελϯεʹNginx(HAProxy)Λஔ• ϙʔτϑΥϫʔμʔ• ৽چΛΞτϛοΫʹΓସ͑Մ• શΠϯελϯεʹૢ࡞͕ඞཁ
2Monitoring ECSwith Mackerel
2-aDocker (cgroup)Statistics
DockerͷϞχλϦϯάͷجຊ• cgroupͷ࡞๏ʹै͏• ίϯςφଆ͔Βɺϗετશମ͔͠ݟ͑ͳ͍• /proc/loadavg, /proc/stat • ݟ͍ͨ߹`-‐v`ͰऔΓࠐΉ
ϗετଆ͔ΒͷϞχλϦϯά• cgroupͷϨϙʔτΛར༻• /sys/fs/cgroup/ {cpuacct, memory} • ଞʹblkio, cpu, devicesͳͲͷαϒγεςϜ͕͋Δ(֤छ੍ݶͳͲͷ੍ޚ༻)VCVOUVͷ߹ LFSOFMHFOFSJDEPDLFS
CPU༻• /sys/fs/cgroup/cpuacct/docker/{ίϯςφID}/cpuacct.stat• user … ϢʔβʔϞʔυͷCPU ࣌ؒ• system … γεςϜϞʔυͷCPU ࣌ؒ• CPU࣌ؒͷ୯Ґ: USER_HZ (σϑΥϧτ10ms)user 480 system 116
cpuacctͷଞͷࢦඪ• cpuacct.usage: ૯CPU࣌ؒ (φϊඵ) echo 0 > /cgroup/cpuacct/cpuacct.usage ͰϦηοτ• cpuacct.usage_percpu: ֤CPUͷCPU ࣌ؒ (φϊඵ)
ϝϞϦ༻• /sys/fs/cgroup/memory/docker/{ίϯςφID}/memory.statcache 761856 rss 25681920 rss_huge 0 mapped_file 16384 pgpgin 12569 … total_cache 761856 total_rss 25681920 … ͍ΘΏΔ[email protected]
memory.statৄࡉ• cache • rss • mapped_file • pgpgin • pgpgout • swap • active_anon • inactive_anon • active_file • inactive_file • unevictable • hierarchical_memory_limit • hierarchical_memsw • total_ࢠάϧʔϓΛؚΉৄࡉIUUQTXXXLFSOFMPSHEPD%PDVNFOUBUJPODHSPVQTNFNPSZUYU
memoryؔ࿈(ͦͷଞ)memory.usage_in_bytes ϝϞϦ༻ྔmemory.memsw.usage_in_bytes ϝϞϦ༻ྔͱεϫοϓྖҬͷmemory.max_usage_in_bytes ϝϞϦ࠷େ༻ྔmemory.memsw.max_usage_in_bytes εϫοϓ࠷େ༻ྔmemory.limit_in_bytes ϝϞϦͷ੍ݶmemory.memsw.limit_in_bytes ϝϞϦͱεϫοϓ੍ݶmemory.failcnt ϝϞϦ੍ݶʹୡͨ͠ճmemory.memsw.failcnt ϝϞϦ੍ݶʹୡͨ͠ճ
ڞ௨Ϩϙʔτ• tasks ࣮ߦதͷϓϩηεͷҰཡ(PID)• cgroup.procs ࣮ߦதͷεϨουάϧʔϓͷҰཡ(TGID)% cat /sys/fs/cgroup/memory/docker/…./cgroup.procs 31218 31307 % ps aux | egrep '(31218|31307)' root 31218 0.0 0.0 4408 164 ? Ss 13:15 0:00 /bin/sh -‐c rackup -‐p4567 root 31307 0.0 5.2 281364 26336 ? Sl 13:15 0:06 ruby1.9.1 /usr/local/bin/rackup -‐p4567
2-bMonitoring Dockerwith Mackerel
Mackerelhttps://mackerel.io/
ಛ• ʮׂʯʹΑΔϗετ܈ͷཧ• ΤʔδΣϯτʹΑΔϦιʔεࢹϩ ʔ ϧ
ϩʔϧຖͷάϥϑͰՄࢹԽ
mackerel-agent• Go Ͱॻ͔ΕͨެࣜΤʔδΣϯτ• Linux / Windows αϙʔτ• ϗετεϖοΫͱϝτϦοΫΛऩूmackerelio/mackerel-agent
Blue-green DeploymentରԠ• ϩʔϧ୯ҐͰͷՄࢹԽ• ୀࡁΈίϯςφͷάϥϑอ࣋Deployment
ΦʔτεέʔϧରԠ• αʔό͕ॖୀͯ͠άϥϑΔ
Raspberry Pi Ͱಈ͖·͢• Թɾ࣪ηϯα → Slack௨
Docker with Mackerel
mackerel/mackerel-agenton Docker Hubhttps://registry.hub.docker.com/u/stanaka/mackerel-‐agent/
How to Use• ίϚϯυҰൃ• ඞཁͳύεɾAPIKEYΛ͢docker run -‐h `hostname` \ -‐v /var/run/docker.sock:/var/run/docker.sock \ -‐v /var/lib/mackerel-‐agent/:/var/lib/mackerel-‐agent/ \ -‐v /proc/mounts:/host/proc/mounts:ro \ -‐v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -‐e 'apikey=‘ \ mackerel/mackerel-‐agent
DockerFile(ൈਮ)FROM debian:jessie # setup mackerel-‐agent RUN apt-‐get update \ .. && apt-‐get -‐y install mackerel-‐agent \ .. # setup docker plugin COPY mackerel-‐plugin-‐docker.rb /mackerel-‐plugin-‐docker.rb RUN echo [plugin.metrics.docker] \ >> /etc/mackerel-‐agent/mackerel-‐agent.conf RUN echo command = \"ruby /mackerel-‐plugin-‐docker.rb\" \ >> /etc/mackerel-‐agent/mackerel-‐agent.conf # boot mackerel-‐agent CMD /startup.sh
֤ίϯςφͷใΛऩू)PTUqVFOUENBDLFSFMBHFOU"QQ
༷ࢠ(ίϯςφຖͷCPU)
ίϯςφͷதͷใΛऩू͍ͨ͠
ίϯςφͷதͷใΛऩू)PTUNBDLFSFMBHFOU"QQ
Memcachedͷྫ% docker run -‐d -‐P \ -‐-‐name memcached -‐p 11211:11211 \ sylvainlasnier/memcached)PTUNBDLFSFMBHFOUNFNDBDIFE
linkΦϓγϣϯͰଓ% docker run -‐i -‐t -‐h `hostname` \ -‐v /var/run/docker.sock:/var/run/docker.sock \ -‐v /proc/mounts:/host/proc/mounts:ro \ -‐v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \ -‐e 'apikey=' -‐-‐link memcached:memcached \ mackerel-‐agent /bin/bash [email protected]:/# env HOSTNAME=develop04.host.h MEMCACHED_PORT_11211_TCP_PORT=11211 MEMCACHED_PORT_11211_TCP_PROTO=tcp MEMCACHED_NAME=/tender_curie/memcached MEMCACHED_PORT=tcp://172.17.0.88:11211 MEMCACHED_PORT_11211_TCP_ADDR=172.17.0.88 MEMCACHED_PORT_11211_TCP=tcp://172.17.0.88:11211ݱঢ়Pattern2’ͰରԠ
ڥมͰΞυϨεࢦఆ[email protected]:/# /usr/local/bin/mackerel-‐plugin-‐memcached \ -‐host=$MEMCACHED_PORT_11211_TCP_ADDR memcached.connections.curr_connections 10 1427507503 memcached.cmd.cmd_get 0 1427507503 memcached.cmd.cmd_set 0 1427507503
ϓϥάΠϯઃఆΛࠐΉ% docker run -‐h `hostname` \ -‐v /var/run/docker.sock:/var/run/docker.sock \ -‐v /var/lib/mackerel-‐agent/:/var/lib/mackerel-‐agent/ \ -‐v /proc/mounts:/host/proc/mounts:ro \ -‐v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -‐e 'apikey=‘ \ -‐-‐link memcached:memcached \ -‐v /etc/mackerel-‐agent/conf.d:/etc/mackerel-‐agent/conf.d:ro -‐e ‘include=/etc/mackerel-‐agent/conf.d/*' \ mackerel/mackerel-‐agent% cat /etc/mackerel-‐agent/conf.d/memcached.conf [plugin.metrics.memcached] command = "/usr/local/bin/mackerel-‐plugin-‐memcached -‐host=$MEMCACHED_PORT_11211_TCP_ADDR"
MemcachedͷϝτϦοΫ
·ͱΊ• ECSͰBlue-Green Deployment• Patten 1, 2, 2’• Monitoring Docker with Mackerel
stanaka
jordihofc
nikkei_engineer_recruiting
abenben
miyakemito
stefafafan
you
eliasnogueira
shimashima35
odasho
ikuyamada
foursue
maggiecrowley
jacobian
jensimmons
carmenintech
robhawkes
jennybc
tammielis
mclloyd
smashingmag
bkeepers
mthomps
aleyda