eb_with_multi_container_docker

by Takumi KANZAKI

Slide 1

Slide 1 text

JAWS-UG͓ίϯςφࢧ ෦ #5 EB with multi-container Docker Λ productionͰ1೥ؒӡ༻ͯ͠ى͖ͨ͜ ͱ 1 VASILY Inc. 2016

Slide 2

Slide 2 text

Me • Takumi Kanzaki (@tknzk) • VASILY Inc. Backend Engineer • 2015.04~ • iQONAD/ࣾ಺༻੥ٻॻπʔϧ Λ୲౰ ($∀$) • ࠷ۙiQON API ΋୲౰ 2 VASILY Inc. 2016

Slide 3

Slide 3 text

3 VASILY Inc. 2016

Slide 4

Slide 4 text

iQONAD • iQON͓Αͼ֎෦ΞϓϦʹ޿ࠂ഑৴Λߦ͍ͬͯΔADωοτϫʔΫ • 2014.10~ • ॳظ࣮૷~ϩʔϯν • 2weeks 4 VASILY Inc. 2016

Slide 5

Slide 5 text

iQONAD • software, middleware • Ruby, Rails, sinatra • MySQL, memcached, Redis • infra • EC2, RDS • docker, ElasticBeanstalk (2015.06~) • auto-scaling 5 VASILY Inc. 2016

Slide 6

Slide 6 text

EB w/ multi-container docker • 2015/06~ • 1೥ؒproduction؀ڥͰӡ༻ͯ͠ى͖ͨ͜ͱΛ͝঺հ͠·͢ 6 VASILY Inc. 2016

Slide 7

Slide 7 text

Container ߏ੒ • 4ͭͷcontainerͰՔಇ 1. ຊମͷΞϓϦέʔγϣϯ͕͸͍ͬͨContainer (ad-server) • ruby,supervisord 2. httpͷϦΫΤετΛड͚෇͚ΔͨΊͷnginx 3. logίϨΫλͱͯ͠ͷtd-agent 4. ؂ࢹ༻ͷmackerel-agent 7 VASILY Inc. 2016

Slide 8

Slide 8 text

։ൃ/ӡ༻ฤ 8 VASILY Inc. 2016

Slide 9

Slide 9 text

#1 middlewareͷߋ৽ • ruby 2.0 -> 2.2.3 • ϩʔϯν·Ͱ2िؒͰ։ൃΛߦͬͨ͜ͱ͔Βطଘͷ։ൃӡ༻࣮੷ ͷ͋Δruby 2.0.0 Ͱ։ൃ/ӡ༻͞Ε͍ͯ·ͨ͠ • 2015/11ʹdockerͷڧΈΛ׆͔ͯ͠ɺruby2.2ܥʹҠߦ • ruby2.2 • GGͷվળͷԸܙ 9 VASILY Inc. 2016

Slide 10

Slide 10 text

#1 middlewareͷߋ৽ • ruby 2.0 -> 2.2.3 10 VASILY Inc. 2016

Slide 11

Slide 11 text

#1 middlewareͷߋ৽ • ruby 2.0 -> 2.2.3 -> 2.3.1 • ࠓ೥ͷ͸͡Ίʹ ruby 2.3.1΁Ҡߦ • ଞͷmiddleware΋ΧδϡΞϧʹ࠷৽΁ • nginx: 1.11.1 • mackerel-agent: 0.31.2 • td-agent: v2.3.1 (fluentd:v0.12.20) 11 VASILY Inc. 2016

Slide 12

Slide 12 text

#1 middlewareͷߋ৽ • ruby 2.4.0΋ਵ࣌ςετΛ͓͜ͳ͍ͬͯ·͢ • ruby 2.4.0-preview1 12 VASILY Inc. 2016

Slide 13

Slide 13 text

#1 middlewareͷߋ৽ • docker ͷར఺ • ΧδϡΞϧʹimageΛೖΕସ͑ΒΕΔ 13 VASILY Inc. 2016

Slide 14

Slide 14 text

#2 docker imageංେԽ • ͱ͋ΔλΠϛϯάͷdocker image size • ad_server: 924.6MB • nginx: 134.1MB • td-agent: 448.4MB • mackerel-agent: 423.1MB 14 VASILY Inc. 2016

Slide 15

Slide 15 text

#2 docker imageංେԽ ͑ͬʂࢲͷdocker imageଠΓա͗..! 15 VASILY Inc. 2016

Slide 16

Slide 16 text

#2-2 docker image࡟ݮ • ํ਑ 1. Ͱ͖Δ͚ͩܰྔͳOSΛϕʔεʹ͢Δ 2. ෆඞཁͳύοέʔδΛΠϯετʔϧ͠ͳ͍ 3. build͢Δͱ͖ʹ͚ͩඞཁͳύοέʔδΛదٓ࡟আ͢Δ 16 VASILY Inc. 2016

Slide 17

Slide 17 text

#2-2 docker image࡟ݮ nginx • ެࣜͰalpine linuxΛϕʔεʹͨ͠΋ͷ ͕͋Δ • ࠾༻! 17 VASILY Inc. 2016

Slide 18

Slide 18 text

#2-2 docker image࡟ݮ ad-server • ΋ͱ΋ͱ CentOS6Λϕʔεʹbuild • 2ஈߏ੒ • ruby, supervisord, mysql Λ૊ΈࠐΜͩimage • ↑ʹad-serverͱͯ͠ඞཁͳGemΛ૊ΈࠐΉ 18 VASILY Inc. 2016

Slide 19

Slide 19 text

#2-2 docker image࡟ݮ ad-server • alpine linuxΛϕʔεʹͯ͠build • ॳظ • build͢Δ͚ͩͯ͠૟আΛ͍ͯ͠ͳ͍ঢ়ଶ 19 VASILY Inc. 2016

Slide 20

Slide 20 text

#2-2 docker image࡟ݮ ad-server 20 VASILY Inc. 2016

Slide 21

Slide 21 text

#2-2 docker image࡟ݮ ad-server • ਫ਼ࠪ1 • mysql (mariadb-dev) ͕ͱͯ΋େ͖͍... • ෆཁͳόΠφϦΛ࡟আ rm /usr/lib/libmysqld* && \ rm /usr/bin/mysql* 21 VASILY Inc. 2016

Slide 22

Slide 22 text

#2-2 docker image࡟ݮ ad-server • ਫ਼ࠪ2 • buildʹඞཁͳύοέʔδΛ੔ཧ • native-extensionͳGemΠϯετʔϧʹඞཁͳ෺ΛtemporaryͰ install/uninstall • build-base (gccͳͲ) 22 VASILY Inc. 2016

Slide 23

Slide 23 text

#2 docker imageංେԽ • ࠷৽ͷ docker image size • ad_server: 924.6MB -> 342.7MB • nginx: 134.1MB -> 59.63MB • td-agent: 448.4MB -> 430.8MB • mackerel-agent: 423.1MB -> 357.5MB 23 VASILY Inc. 2016

Slide 24

Slide 24 text

#2 docker imageංେԽ • ࠷େ࣌ͷ6ׂ͙Β͍·Ͱ࡟ݮ • blogॻ͖·ͨ͠ http://tech.vasily.jp/ docker_image_slim_in_alpinelinux 24 VASILY Inc. 2016

Slide 25

Slide 25 text

#3 base platform • EB w/multi-container docker • Amazon Linux ͱ dockerͷ૊Έ߹ΘͤͰෳ਺ 25 VASILY Inc. 2016

Slide 26

Slide 26 text

#3 base platform • EB w/multi-container docker • Amazon Linux 2015.03 v1.4.0 docker 1.6.0 • Amazon Linux 2015.03 v1.4.1 docker 1.6.0 • Amazon Linux 2015.03 v1.4.6 docker 1.6.2 • Amazon Linux 2015.09 v2.0.8 docker 1.9.1 • Amazon Linux 2016.03 v2.1.0 docker 1.9.1 • Amazon Linux 2016.03 v2.1.1 docker 1.9.1 26 VASILY Inc. 2016

Slide 27

Slide 27 text

#3 base platform • ͍·͸গ͠ݹΊversionΛར༻ • Amazon Linux 2015.09 v2.0.8 docker 1.9.1 • Amazon Linux 2016.03͸mackerel-agent͕ىಈͰ͖ͳ͍໰୊ • docker͔ͩΒؔ܎ͳ͍ͱࢥ͍ͬͯͨΒىಈͰ͖ͳ͔ͬͨ.. 27 VASILY Inc. 2016

Slide 28

Slide 28 text

#3 base platform • hostͷdocker ͕গ͠ݹ͍ͷͰɺclientଆ͕৽͗͢͠ΔͱىಈͰ͖ ͳ͍໰୊΋.. • docker 1.9.1 • EBʹґଘ͠ͳ͍ܗ΋ߟ͍͖͍͑ͯͨ 28 VASILY Inc. 2016

Slide 29

Slide 29 text

#3 base platform • ࠓݟͨΒ, Amazon Linux 2016.03 docker 1.11.1 ͕Ͱͯͨ • ໌೔ࢼ͍ͨ͠ͱ͓΋͍·͢ 29 VASILY Inc. 2016

Slide 30

Slide 30 text

#3 Envoirnment clone • production/stagingͷ؀ڥΛ·ΔͬͱίϐʔͰ͖Δ • base platformΛมߋ͢Δࡍͷςετ 30 VASILY Inc. 2016

Slide 31

Slide 31 text

#3 Envoirnment clone 31 VASILY Inc. 2016

Slide 32

Slide 32 text

#3 Envoirnment clone 32 VASILY Inc. 2016

Slide 33

Slide 33 text

#3 Envoirnment clone • ਺ΫϦοΫͰίϐʔͰ͖ΔͷͰͱͯ΋ศར! 33 VASILY Inc. 2016

Slide 34

Slide 34 text

#4 auto-scaling • ௨ৗͷΞΫηε૿Ճ/ݮগʹ͸Auto-ScalingͰରԠ • EBͷweb consoleͷScaling configurationͰઃఆ • Sum RequestCountϕʔε 34 VASILY Inc. 2016

Slide 35

Slide 35 text

#5 Time Scheduled-based auto- scaling • ADωοτϫʔΫ • iQONҎ֎ͷΞϓϦʹ΋഑৴ 35 VASILY Inc. 2016

Slide 36

Slide 36 text

#5 Time Scheduled-based auto- scaling • ͱ͋ΔΞϓϦ͕pushΛ͏ͭ • spike͢Δ • ௨ৗͷauto-scalingͰ͸௥͍͔ͭͳ͍ • pushʹΑΔspike͕ऩ·ͬͨ͋ͱʹىಈ׬ྃ.. • ҙຯͳ͍ 36 VASILY Inc. 2016

Slide 37

Slide 37 text

#5 Time Scheduled-based auto- scaling • time scheduled base auto-scalingΛઃఆ • web console͔Β͸ઃఆͰ͖ͳ͍ͷͰaws cliͰߦ͏ 37 VASILY Inc. 2016

Slide 38

Slide 38 text

#5 Time Scheduled-based auto- scaling • time scheduled base auto-scalingΛઃఆ • scale out aws autoscaling put-scheduled-update-group-action \ --auto-scaling-group-name awseb-e-XXXXXX \ --scheduled-action-name ScaleOut \ --recurrence "00 10 * * *" \ --desired-capacity 10 \ --min-size 10 \ --max-size 12 38 VASILY Inc. 2016

Slide 39

Slide 39 text

#5 Time Scheduled-based auto- scaling • time scheduled base auto-scalingΛઃఆ • scale in aws autoscaling put-scheduled-update-group-action \ --auto-scaling-group-name awseb-e-XXXXXX \ --scheduled-action-name ScaleIn \ --recurrence "30 16 * * *" \ --desired-capacity 6 \ --min-size 6 \ --max-size 8 39 VASILY Inc. 2016

Slide 40

Slide 40 text

#5 Time Scheduled-based auto- scaling • ઌํ΋ӡ༻͠ͳ͕ΒͩͬͨͷͰɺ͏·͍͘ͳ͔͍͜ͱ΋ଟ͔ͬ ͨ • ݱࡏ͸pushͷӡ༻͕ͱ·ͬͨ • time scheduled base auto scaling͸ઃఆͯ͠ͳ͍ 40 VASILY Inc. 2016

Slide 41

Slide 41 text

#6 ։ൃ؀ڥ • docker-compose Ͱproductionͱಉ౳ͷ؀ڥΛىಈͰ͖ΔΑ͏ʹ ͍ͯ͠Δ • mysql,redis,memcache͸ࣾ಺։ൃαʔόΛࢀর 41 VASILY Inc. 2016

Slide 42

Slide 42 text

#6 ։ൃ؀ڥ • ීஈ͸MBPͷruby,nginxͰ։ൃΛ͍ͯ͠Δ • ։ൃ࣌͸جຊతʹҰਓͰΈͯΔͷͰContainerͷྑ͍ॴ͕ग़ͤ ͳ͍.. • docker-compose run app rake ͰCIͷΈ࣮ࢪ 42 VASILY Inc. 2016

Slide 43

Slide 43 text

#7 Docker build • Dockerfile͸ผrepositoryͰGithubͰ؅ཧ • docker resitory(quay.io)ͷbuild hook͸ະ࠾༻ • खݩͷMBP(docker-machine) Ͱdocker build & pushͰӡ༻ • docker build -t quay.io/vasilyjp/ruby:2.3.1-alpine . • docker push quay.io/vasilyjp/ruby:2.3.1-alpine 43 VASILY Inc. 2016

Slide 44

Slide 44 text

#7 Docker build • ad-serverͷ2ஈ໨ͷimage͸CircleCI͕build • ઐ༻ͷbranchʹpush͢ΔͱCircleCI͕build & push 44 VASILY Inc. 2016

Slide 45

Slide 45 text

#7 Docker build • alpineԽ͔ͯ͠Β͸Dockerfile͓͡͞ΜԽ͕ݦஶʹਐΜͰΔؾ΋͠ ·͢.. 45 VASILY Inc. 2016

Slide 46

Slide 46 text

ো֐ฤ • ओʹӡ༻ϛεʹΑΔো֐͕ൃੜ͠·ͨ͠ 46 VASILY Inc. 2016

Slide 47

Slide 47 text

#1 quay.ioύεϫʔυมߋʹ൐͏ deployͷࣦഊ • imageΛquay.ioͰϗετ • ϚελͷύεϫʔυΛͦͷ··deploy࣌ʹ΋࢖͍ͬͯͨ • .dockercfgʹϚελΞΧ΢ϯτͷ΋ͷΛೖΕ͍ͯͨ 47 VASILY Inc. 2016

Slide 48

Slide 48 text

#1 quay.ioύεϫʔυมߋʹ൐͏ deployͷࣦഊ • μϝ! ઈର!! 48 VASILY Inc. 2016

Slide 49

Slide 49 text

#1 quay.ioύεϫʔυมߋʹ൐͏ deployͷࣦഊ • ϚελύεϫʔυͷมߋΛߦͬͨ • ͦͷޙͷσϓϩΠʹࣦഊ • docker pull͕Ͱ͖ͳ͘ͳͬͨ.. 49 VASILY Inc. 2016

Slide 50

Slide 50 text

#1 quay.ioύεϫʔυมߋʹ൐͏ deployͷࣦഊ • auto-scalingͷλΠϛϯάͱ΋૬·ͬͯΞΧ΢ϯτ͕ϩοΫ • େྔͷdocker pullͷࣦഊ͕ൃੜ • ϚελͷΞΧ΢ϯτͰdocker pull ͕ϩʔΧϧ؀ڥͰ΋Ͱ͖ͳ͘ ͳͬͨ 50 VASILY Inc. 2016

Slide 51

Slide 51 text

#1 quay.ioύεϫʔυมߋʹ൐͏ deployͷࣦഊ • ରԠ • σϓϩΠ༻ͷϩϘοτΞΧ΢ϯτΛ࡞੒(quay.ioͷػೳ) • ͦͷޙɺো֐͸ى͖͍ͯͳ͍ 51 VASILY Inc. 2016

Slide 52

Slide 52 text

#1 quay.ioύεϫʔυมߋʹ൐͏ deployͷࣦഊ • quay.io͕SPOFʹͳ͍ͬͯΔͷͰɺηΧϯμϦʹ͍ͭͯ͸ݕ౼த • ECR͸ϦϦʔε౰ॳʹ؆୯ʹςετͨ͠ͷΈ • ஌ݟ͋Ε͹ڭ͑ͯԼ͍͞! 52 VASILY Inc. 2016

Slide 53

Slide 53 text

#2 mackerel-agentͷΞϥʔτ • mackerelͰ؂ࢹ • auto-scalingͰΧδϡΞϧʹinstance͕ࢭ·Δ • unreachableͷΞϥʔτ͕ඈͿ • unreachable Ͱ͸͋Δ͚Ͳɺো֐Ͱ͸ͳ͍.. 53 VASILY Inc. 2016

Slide 54

Slide 54 text

#2 mackerel-agentͷΞϥʔτ • auto-scalingͳ؀ڥ༻ͷઃఆΛ࡞ͬͯ΋Βͬͨ 54 VASILY Inc. 2016

Slide 55

Slide 55 text

#2 mackerel-agentͷΞ ϥʔτ ಈ͔ͳ͔ͬͨ.. 55 VASILY Inc. 2016

Slide 56

Slide 56 text

#2 mackerel-agentͷΞϥʔτ • Ұ୴ɺର৅ͷroleͷunreachableͷΞϥʔτઃఆΛ੾ͬͨ.. • ؂ࢹͷҙຯ͕ແ͍.. 56 VASILY Inc. 2016

Slide 57

Slide 57 text

#2 mackerel-agentͷΞϥʔτ • base platformΛมߋͨ͠Β͏·͍͘͘Α͏ʹͳͬͨ • Amazon Linux 2015.09 v2.0.8 docker 1.9.1 • ؀ڥʹΑͬͯtrap͕ಈ͍ͨΓಈ͔ͳ͔ͬͨΓͯ͠Δͱͷ͜ͱ.. • EBͰͷঢ়گ͸Qiitaʹॻ͍ͯ·͢ -> http://qiita.com/tknzk/items/a6b1a9003ff26683b88a 57 VASILY Inc. 2016

Slide 58

Slide 58 text

#3 كʹσϓϩΠʹࣦഊ͢Δ • deployϑϩʔ • CircleCIͰmasterͷbranchͷCI͕·ΘΔͱdeploy hook͕͸͠Δ • eb deploy ad-server-prod 58 VASILY Inc. 2016

Slide 59

Slide 59 text

#3 كʹσϓϩΠʹࣦഊ͢Δ • deploy͕ࣦഊ͢Δ͕࣌͋Δ • mackerel-agentͷcontainerͷstopʹࣦഊ͢Δ͜ͱ͕͋Δ [ec2-user@ip-xx-xx-xxx ~]$ sudo docker exec CONTAINER_ID ps axu nsenter: Unable to fork: Cannot allocate memory 59 VASILY Inc. 2016

Slide 60

Slide 60 text

#3 كʹσϓϩΠʹࣦഊ͢Δ • ෮چͰ͖ͳ͍ͷͰinstance͝ͱstopͯ͠ରԠ • mackerel-agentͷதͷ໰୊ͷΑ͏ͳؾ͕͍ͯ͠Δ͕ɺৄࡉ͸೺ Ѳग़དྷ͍ͯͳ͍ 60 VASILY Inc. 2016

Slide 61

Slide 61 text

#3 كʹσϓϩΠʹࣦഊ͢Δ • deploy͕ࣦഊͨ͠Β • ΧδϡΞϧʹinstanceΛstopͯ͠ೖΕସ͑ • ࠶deploy 61 VASILY Inc. 2016

Slide 62

Slide 62 text

#3 كʹσϓϩΠʹࣦഊ͢Δ • auto-scalingͷhealth-checkͷର৅ΛELBʹ͢Δ • deployʹࣦഊͯ͠΋ELB͔Β੾Γ཭͞Εͨঢ়ଶ • ৑௕ߏ੒ʹͳ͍ͬͯΕ͹໰୊ͳ͘ӡ༻Ͱ͖Δ 62 VASILY Inc. 2016

Slide 63

Slide 63 text

#4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ ౓͖Γ) • deploy͸CircleCI͔Β eb deploy ad-server-prod ͰσϓϩΠ • applicationΛzipͨ͠ϑΝΠϧΛs3ʹput • ֤instance͕pullͯ͘͠Δ 63 VASILY Inc. 2016

Slide 64

Slide 64 text

#4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ ౓͖Γ) • ಛఆͷawsebcli͸ s3ʹpush͢ΔϑΝΠϧ໊͕gitͷcommithash 4 ܻͰݻఆ • ຊ൪ӡ༻։࢝લʹੜ੒ͨ͠ϑΝΠϧ໊ͱconflictͯ͠deploy͕ί έͨ • ͜ͷ࣌͸ίέ͚͕ͨͩͩͬͨ੒ޭ͢ΔͱσάϨ͢Δ͜ͱʹͳͬ ͨ 64 VASILY Inc. 2016

Slide 65

Slide 65 text

#4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ ౓͖Γ) • ݱ࣌఺Ͱ͸ɺTimeStamp͕suffixͱͯͭ͘͠Α͏ʹͳͬͨ • versino 3.6.2 Ҏ߱ 65 VASILY Inc. 2016

Slide 66

Slide 66 text

#4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ ౓͖Γ) • ఆظతʹapplication versionsͷͱ͜Ζʹ͋ΔϑΝΠϧΛ૟আ • σϑΥϧτ500·Ͱ͸อ࣋Ͱ͖Δ • ݹ͍ͷΛ࢒͍ͯͯ͠΋͋·Γҙຯͳ͍ͷͰফͯ͠Δ 66 VASILY Inc. 2016

Slide 67

Slide 67 text

Conclusion • 1೥ؒ EB w/multi-container docker Ͱӡ༻ͯ͠ى͖ͨ͜ͱΛ͓࿩ ͠·ͨ͠ • ӡ༻ϛεʹΑΔো֐΋͋Γ·͕ͨ͠ɺEB΍dockerʹىҼ͢Δো ֐͸ͳ͠ • alpine linuxΛproductionͰӡ༻தͰ͢ • iQONຊମͷAPI΋͍ͣΕ͸... 67 VASILY Inc. 2016