# JAWS-UGおコンテナ支部 #5
- EB with multi-container Docker をproductionで1年間運用して起きたこと
JAWS-UG͓ίϯςφࢧ෦ #5EB with multi-container Docker ΛproductionͰ1ؒӡ༻ͯ͠ى͖ͨ͜ͱ1 VASILY Inc. 2016
View Slide
Me• Takumi Kanzaki (@tknzk)• VASILY Inc. Backend Engineer• 2015.04~• iQONAD/ࣾ༻ٻॻπʔϧ Λ୲ ($∀$)• ࠷ۙiQON API ୲2 VASILY Inc. 2016
3 VASILY Inc. 2016
iQONAD• iQON͓Αͼ֎෦ΞϓϦʹࠂ৴Λߦ͍ͬͯΔADωοτϫʔΫ• 2014.10~• ॳظ࣮~ϩʔϯν• 2weeks4 VASILY Inc. 2016
iQONAD• software, middleware• Ruby, Rails, sinatra• MySQL, memcached, Redis• infra• EC2, RDS• docker, ElasticBeanstalk (2015.06~)• auto-scaling5 VASILY Inc. 2016
EB w/ multi-containerdocker• 2015/06~• 1ؒproductionڥͰӡ༻ͯ͠ى͖ͨ͜ͱΛ͝հ͠·͢6 VASILY Inc. 2016
Container ߏ• 4ͭͷcontainerͰՔಇ1. ຊମͷΞϓϦέʔγϣϯ͕͍ͬͨContainer (ad-server)• ruby,supervisord2. httpͷϦΫΤετΛड͚͚ΔͨΊͷnginx3. logίϨΫλͱͯ͠ͷtd-agent4. ࢹ༻ͷmackerel-agent7 VASILY Inc. 2016
։ൃ/ӡ༻ฤ8 VASILY Inc. 2016
#1 middlewareͷߋ৽• ruby 2.0 -> 2.2.3• ϩʔϯν·Ͱ2िؒͰ։ൃΛߦͬͨ͜ͱ͔Βطଘͷ։ൃӡ༻࣮ͷ͋Δruby 2.0.0 Ͱ։ൃ/ӡ༻͞Ε͍ͯ·ͨ͠• 2015/11ʹdockerͷڧΈΛ׆͔ͯ͠ɺruby2.2ܥʹҠߦ• ruby2.2• GGͷվળͷԸܙ9 VASILY Inc. 2016
#1 middlewareͷߋ৽• ruby 2.0 -> 2.2.310 VASILY Inc. 2016
#1 middlewareͷߋ৽• ruby 2.0 -> 2.2.3 -> 2.3.1• ࠓͷ͡Ίʹ ruby 2.3.1Ҡߦ• ଞͷmiddlewareΧδϡΞϧʹ࠷৽• nginx: 1.11.1• mackerel-agent: 0.31.2• td-agent: v2.3.1 (fluentd:v0.12.20)11 VASILY Inc. 2016
#1 middlewareͷߋ৽• ruby 2.4.0ਵ࣌ςετΛ͓͜ͳ͍ͬͯ·͢• ruby 2.4.0-preview112 VASILY Inc. 2016
#1 middlewareͷߋ৽• docker ͷར• ΧδϡΞϧʹimageΛೖΕସ͑ΒΕΔ13 VASILY Inc. 2016
#2 docker imageංେԽ• ͱ͋ΔλΠϛϯάͷdocker image size• ad_server: 924.6MB• nginx: 134.1MB• td-agent: 448.4MB• mackerel-agent: 423.1MB14 VASILY Inc. 2016
#2 docker imageංେԽ͑ͬʂࢲͷdocker imageଠΓա͗..!15 VASILY Inc. 2016
#2-2 docker imageݮ• ํ1. Ͱ͖Δ͚ͩܰྔͳOSΛϕʔεʹ͢Δ2. ෆඞཁͳύοέʔδΛΠϯετʔϧ͠ͳ͍3. build͢Δͱ͖ʹ͚ͩඞཁͳύοέʔδΛదٓআ͢Δ16 VASILY Inc. 2016
#2-2 docker imageݮnginx• ެࣜͰalpine linuxΛϕʔεʹͨ͠ͷ͕͋Δ• ࠾༻!17 VASILY Inc. 2016
#2-2 docker imageݮad-server• ͱͱ CentOS6Λϕʔεʹbuild• 2ஈߏ• ruby, supervisord, mysql ΛΈࠐΜͩimage• ↑ʹad-serverͱͯ͠ඞཁͳGemΛΈࠐΉ18 VASILY Inc. 2016
#2-2 docker imageݮad-server• alpine linuxΛϕʔεʹͯ͠build• ॳظ• build͢Δ͚ͩͯ͠আΛ͍ͯ͠ͳ͍ঢ়ଶ19 VASILY Inc. 2016
#2-2 docker imageݮad-server20 VASILY Inc. 2016
#2-2 docker imageݮad-server• ਫ਼ࠪ1• mysql (mariadb-dev) ͕ͱͯେ͖͍...• ෆཁͳόΠφϦΛআrm /usr/lib/libmysqld* && \rm /usr/bin/mysql*21 VASILY Inc. 2016
#2-2 docker imageݮad-server• ਫ਼ࠪ2• buildʹඞཁͳύοέʔδΛཧ• native-extensionͳGemΠϯετʔϧʹඞཁͳΛtemporaryͰinstall/uninstall• build-base (gccͳͲ)22 VASILY Inc. 2016
#2 docker imageංେԽ• ࠷৽ͷ docker image size• ad_server: 924.6MB -> 342.7MB• nginx: 134.1MB -> 59.63MB• td-agent: 448.4MB -> 430.8MB• mackerel-agent: 423.1MB -> 357.5MB23 VASILY Inc. 2016
#2 docker imageංେԽ• ࠷େ࣌ͷ6ׂ͙Β͍·Ͱݮ• blogॻ͖·ͨ͠http://tech.vasily.jp/docker_image_slim_in_alpinelinux24 VASILY Inc. 2016
#3 base platform• EB w/multi-container docker• Amazon Linux ͱ dockerͷΈ߹ΘͤͰෳ25 VASILY Inc. 2016
#3 base platform• EB w/multi-container docker• Amazon Linux 2015.03 v1.4.0 docker 1.6.0• Amazon Linux 2015.03 v1.4.1 docker 1.6.0• Amazon Linux 2015.03 v1.4.6 docker 1.6.2• Amazon Linux 2015.09 v2.0.8 docker 1.9.1• Amazon Linux 2016.03 v2.1.0 docker 1.9.1• Amazon Linux 2016.03 v2.1.1 docker 1.9.126 VASILY Inc. 2016
#3 base platform• ͍·গ͠ݹΊversionΛར༻• Amazon Linux 2015.09 v2.0.8 docker 1.9.1• Amazon Linux 2016.03mackerel-agent͕ىಈͰ͖ͳ͍• docker͔ͩΒؔͳ͍ͱࢥ͍ͬͯͨΒىಈͰ͖ͳ͔ͬͨ..27 VASILY Inc. 2016
#3 base platform• hostͷdocker ͕গ͠ݹ͍ͷͰɺclientଆ͕৽͗͢͠ΔͱىಈͰ͖ͳ͍..• docker 1.9.1• EBʹґଘ͠ͳ͍ܗߟ͍͖͍͑ͯͨ28 VASILY Inc. 2016
#3 base platform• ࠓݟͨΒ, Amazon Linux 2016.03 docker 1.11.1 ͕Ͱͯͨ• ໌ࢼ͍ͨ͠ͱ͓͍·͢29 VASILY Inc. 2016
#3 Envoirnment clone• production/stagingͷڥΛ·ΔͬͱίϐʔͰ͖Δ• base platformΛมߋ͢Δࡍͷςετ30 VASILY Inc. 2016
#3 Envoirnment clone31 VASILY Inc. 2016
#3 Envoirnment clone32 VASILY Inc. 2016
#3 Envoirnment clone• ΫϦοΫͰίϐʔͰ͖ΔͷͰͱͯศར!33 VASILY Inc. 2016
#4 auto-scaling• ௨ৗͷΞΫηε૿Ճ/ݮগʹAuto-ScalingͰରԠ• EBͷweb consoleͷScaling configurationͰઃఆ• Sum RequestCountϕʔε34 VASILY Inc. 2016
#5 Time Scheduled-based auto-scaling• ADωοτϫʔΫ• iQONҎ֎ͷΞϓϦʹ৴35 VASILY Inc. 2016
#5 Time Scheduled-based auto-scaling• ͱ͋ΔΞϓϦ͕pushΛ͏ͭ• spike͢Δ• ௨ৗͷauto-scalingͰ͍͔ͭͳ͍• pushʹΑΔspike͕ऩ·ͬͨ͋ͱʹىಈྃ..• ҙຯͳ͍36 VASILY Inc. 2016
#5 Time Scheduled-based auto-scaling• time scheduled base auto-scalingΛઃఆ• web console͔ΒઃఆͰ͖ͳ͍ͷͰaws cliͰߦ͏37 VASILY Inc. 2016
#5 Time Scheduled-based auto-scaling• time scheduled base auto-scalingΛઃఆ• scale outaws autoscaling put-scheduled-update-group-action \--auto-scaling-group-name awseb-e-XXXXXX \--scheduled-action-name ScaleOut \--recurrence "00 10 * * *" \--desired-capacity 10 \--min-size 10 \--max-size 1238 VASILY Inc. 2016
#5 Time Scheduled-based auto-scaling• time scheduled base auto-scalingΛઃఆ• scale inaws autoscaling put-scheduled-update-group-action \--auto-scaling-group-name awseb-e-XXXXXX \--scheduled-action-name ScaleIn \--recurrence "30 16 * * *" \--desired-capacity 6 \--min-size 6 \--max-size 839 VASILY Inc. 2016
#5 Time Scheduled-based auto-scaling• ઌํӡ༻͠ͳ͕ΒͩͬͨͷͰɺ͏·͍͘ͳ͔͍͜ͱଟ͔ͬͨ• ݱࡏpushͷӡ༻͕ͱ·ͬͨ• time scheduled base auto scalingઃఆͯ͠ͳ͍40 VASILY Inc. 2016
#6 ։ൃڥ• docker-compose ͰproductionͱಉͷڥΛىಈͰ͖ΔΑ͏ʹ͍ͯ͠Δ• mysql,redis,memcacheࣾ։ൃαʔόΛࢀর41 VASILY Inc. 2016
#6 ։ൃڥ• ීஈMBPͷruby,nginxͰ։ൃΛ͍ͯ͠Δ• ։ൃ࣌جຊతʹҰਓͰΈͯΔͷͰContainerͷྑ͍ॴ͕ग़ͤͳ͍..• docker-compose run app rake ͰCIͷΈ࣮ࢪ42 VASILY Inc. 2016
#7 Docker build• DockerfileผrepositoryͰGithubͰཧ• docker resitory(quay.io)ͷbuild hookະ࠾༻• खݩͷMBP(docker-machine) Ͱdocker build & pushͰӡ༻• docker build -t quay.io/vasilyjp/ruby:2.3.1-alpine .• docker push quay.io/vasilyjp/ruby:2.3.1-alpine43 VASILY Inc. 2016
#7 Docker build• ad-serverͷ2ஈͷimageCircleCI͕build• ઐ༻ͷbranchʹpush͢ΔͱCircleCI͕build & push44 VASILY Inc. 2016
#7 Docker build• alpineԽ͔ͯ͠ΒDockerfile͓͡͞ΜԽ͕ݦஶʹਐΜͰΔؾ͠·͢..45 VASILY Inc. 2016
োฤ• ओʹӡ༻ϛεʹΑΔো͕ൃੜ͠·ͨ͠46 VASILY Inc. 2016
#1 quay.ioύεϫʔυมߋʹ͏deployͷࣦഊ• imageΛquay.ioͰϗετ• ϚελͷύεϫʔυΛͦͷ··deploy࣌ʹ͍ͬͯͨ• .dockercfgʹϚελΞΧϯτͷͷΛೖΕ͍ͯͨ47 VASILY Inc. 2016
#1 quay.ioύεϫʔυมߋʹ͏deployͷࣦഊ• μϝ! ઈର!!48 VASILY Inc. 2016
#1 quay.ioύεϫʔυมߋʹ͏deployͷࣦഊ• ϚελύεϫʔυͷมߋΛߦͬͨ• ͦͷޙͷσϓϩΠʹࣦഊ• docker pull͕Ͱ͖ͳ͘ͳͬͨ..49 VASILY Inc. 2016
#1 quay.ioύεϫʔυมߋʹ͏deployͷࣦഊ• auto-scalingͷλΠϛϯάͱ૬·ͬͯΞΧϯτ͕ϩοΫ• େྔͷdocker pullͷࣦഊ͕ൃੜ• ϚελͷΞΧϯτͰdocker pull ͕ϩʔΧϧڥͰͰ͖ͳ͘ͳͬͨ50 VASILY Inc. 2016
#1 quay.ioύεϫʔυมߋʹ͏deployͷࣦഊ• ରԠ• σϓϩΠ༻ͷϩϘοτΞΧϯτΛ࡞(quay.ioͷػೳ)• ͦͷޙɺোى͖͍ͯͳ͍51 VASILY Inc. 2016
#1 quay.ioύεϫʔυมߋʹ͏deployͷࣦഊ• quay.io͕SPOFʹͳ͍ͬͯΔͷͰɺηΧϯμϦʹ͍ͭͯݕ౼த• ECRϦϦʔεॳʹ؆୯ʹςετͨ͠ͷΈ• ݟ͋Εڭ͑ͯԼ͍͞!52 VASILY Inc. 2016
#2 mackerel-agentͷΞϥʔτ• mackerelͰࢹ• auto-scalingͰΧδϡΞϧʹinstance͕ࢭ·Δ• unreachableͷΞϥʔτ͕ඈͿ• unreachable Ͱ͋Δ͚ͲɺোͰͳ͍..53 VASILY Inc. 2016
#2 mackerel-agentͷΞϥʔτ• auto-scalingͳڥ༻ͷઃఆΛ࡞ͬͯΒͬͨ54 VASILY Inc. 2016
#2 mackerel-agentͷΞϥʔτಈ͔ͳ͔ͬͨ..55 VASILY Inc. 2016
#2 mackerel-agentͷΞϥʔτ• Ұ୴ɺରͷroleͷunreachableͷΞϥʔτઃఆΛͬͨ..• ࢹͷҙຯ͕ແ͍..56 VASILY Inc. 2016
#2 mackerel-agentͷΞϥʔτ• base platformΛมߋͨ͠Β͏·͍͘͘Α͏ʹͳͬͨ• Amazon Linux 2015.09 v2.0.8 docker 1.9.1• ڥʹΑͬͯtrap͕ಈ͍ͨΓಈ͔ͳ͔ͬͨΓͯ͠Δͱͷ͜ͱ..• EBͰͷঢ়گQiitaʹॻ͍ͯ·͢-> http://qiita.com/tknzk/items/a6b1a9003ff26683b88a57 VASILY Inc. 2016
#3 كʹσϓϩΠʹࣦഊ͢Δ• deployϑϩʔ• CircleCIͰmasterͷbranchͷCI͕·ΘΔͱdeploy hook͕͠Δ• eb deploy ad-server-prod58 VASILY Inc. 2016
#3 كʹσϓϩΠʹࣦഊ͢Δ• deploy͕ࣦഊ͢Δ͕࣌͋Δ• mackerel-agentͷcontainerͷstopʹࣦഊ͢Δ͜ͱ͕͋Δ[[email protected] ~]$ sudo docker exec CONTAINER_ID ps axunsenter: Unable to fork: Cannot allocate memory59 VASILY Inc. 2016
#3 كʹσϓϩΠʹࣦഊ͢Δ• ෮چͰ͖ͳ͍ͷͰinstance͝ͱstopͯ͠ରԠ• mackerel-agentͷதͷͷΑ͏ͳؾ͕͍ͯ͠Δ͕ɺৄࡉѲग़དྷ͍ͯͳ͍60 VASILY Inc. 2016
#3 كʹσϓϩΠʹࣦഊ͢Δ• deploy͕ࣦഊͨ͠Β• ΧδϡΞϧʹinstanceΛstopͯ͠ೖΕସ͑• ࠶deploy61 VASILY Inc. 2016
#3 كʹσϓϩΠʹࣦഊ͢Δ• auto-scalingͷhealth-checkͷରΛELBʹ͢Δ• deployʹࣦഊͯ͠ELB͔ΒΓ͞Εͨঢ়ଶ• ߏʹͳ͍ͬͯΕͳ͘ӡ༻Ͱ͖Δ62 VASILY Inc. 2016
#4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ͖Γ)• deployCircleCI͔Β eb deploy ad-server-prod ͰσϓϩΠ• applicationΛzipͨ͠ϑΝΠϧΛs3ʹput• ֤instance͕pullͯ͘͠Δ63 VASILY Inc. 2016
#4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ͖Γ)• ಛఆͷawsebcli s3ʹpush͢ΔϑΝΠϧ໊͕gitͷcommithash 4ܻͰݻఆ• ຊ൪ӡ༻։࢝લʹੜͨ͠ϑΝΠϧ໊ͱconflictͯ͠deploy͕ίέͨ• ͜ͷ࣌ίέ͚͕ͨͩͩͬͨޭ͢ΔͱσάϨ͢Δ͜ͱʹͳͬͨ64 VASILY Inc. 2016
#4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ͖Γ)• ݱ࣌ͰɺTimeStamp͕suffixͱͯͭ͘͠Α͏ʹͳͬͨ• versino 3.6.2 Ҏ߱65 VASILY Inc. 2016
#4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ͖Γ)• ఆظతʹapplication versionsͷͱ͜Ζʹ͋ΔϑΝΠϧΛআ• σϑΥϧτ500·Ͱอ࣋Ͱ͖Δ• ݹ͍ͷΛ͍ͯͯ͋͠·Γҙຯͳ͍ͷͰফͯ͠Δ66 VASILY Inc. 2016
Conclusion• 1ؒ EB w/multi-container docker Ͱӡ༻ͯ͠ى͖ͨ͜ͱΛ͓͠·ͨ͠• ӡ༻ϛεʹΑΔো͋Γ·͕ͨ͠ɺEBdockerʹىҼ͢Δোͳ͠• alpine linuxΛproductionͰӡ༻தͰ͢• iQONຊମͷAPI͍ͣΕ...67 VASILY Inc. 2016
We are hiring!• [email protected]68 VASILY Inc. 2016
end͋Γ͕ͱ͏͍͟͝·ͨ͠69 VASILY Inc. 2016
tknzk
kenichirokimura
you
harukasakihara
miyakemito
megabitsenmzq
yayoi_dd
ikuyamada
sat
masashi_sutou
toshiaizawa
gen519
shimashima35
afnizarnur
holman
chrislema
3n
swwweet
yeseniaperezcruz
62gerente
lauravandoore
tanoku
brianwarren
hatefulcrawdad
marcduiker