Speaker Deck
Sign in
Sign up
for free
eb_with_multi_container_docker
Takumi KANZAKI
June 27, 2016
Technology
1
1.7k
eb_with_multi_container_docker
# JAWS-UGおコンテナ支部 #5
- EB with multi-container Docker をproductionで1年間運用して起きたこと
Takumi KANZAKI
June 27, 2016
Tweet
Share
Want more?
tknzk
5
1k
tknzk
4
700
tknzk
3
3.5k
reverentgeek
8
410
rmw
1
230
skipperchong
0
150
carmenhchung
6
280
keathley
6
200
productmarketing
3
290
clairelew
4
530
thoeni
1
150
jcasabona
4
240
Transcript
1
.
JAWS-UG͓ίϯςφࢧ ෦ #5 EB with multi-container Docker Λ productionͰ1ؒӡ༻ͯ͠ى͖ͨ͜ ͱ
1 VASILY Inc. 2016
2
.
Me • Takumi Kanzaki (@tknzk) • VASILY Inc. Backend Engineer
• 2015.04~ • iQONAD/ࣾ༻ٻॻπʔϧ Λ୲ ($∀$) • ࠷ۙiQON API ୲ 2 VASILY Inc. 2016
3
.
3 VASILY Inc. 2016
4
.
iQONAD • iQON͓Αͼ֎෦ΞϓϦʹࠂ৴Λߦ͍ͬͯΔADωοτϫʔΫ • 2014.10~ • ॳظ࣮~ϩʔϯν • 2weeks 4
VASILY Inc. 2016
5
.
iQONAD • software, middleware • Ruby, Rails, sinatra • MySQL,
memcached, Redis • infra • EC2, RDS • docker, ElasticBeanstalk (2015.06~) • auto-scaling 5 VASILY Inc. 2016
6
.
EB w/ multi-container docker • 2015/06~ • 1ؒproductionڥͰӡ༻ͯ͠ى͖ͨ͜ͱΛ͝հ͠·͢ 6 VASILY
Inc. 2016
7
.
Container ߏ • 4ͭͷcontainerͰՔಇ 1. ຊମͷΞϓϦέʔγϣϯ͕͍ͬͨContainer (ad-server) • ruby,supervisord 2.
httpͷϦΫΤετΛड͚͚ΔͨΊͷnginx 3. logίϨΫλͱͯ͠ͷtd-agent 4. ࢹ༻ͷmackerel-agent 7 VASILY Inc. 2016
8
.
։ൃ/ӡ༻ฤ 8 VASILY Inc. 2016
9
.
#1 middlewareͷߋ৽ • ruby 2.0 -> 2.2.3 • ϩʔϯν·Ͱ2िؒͰ։ൃΛߦͬͨ͜ͱ͔Βطଘͷ։ൃӡ༻࣮ ͷ͋Δruby
2.0.0 Ͱ։ൃ/ӡ༻͞Ε͍ͯ·ͨ͠ • 2015/11ʹdockerͷڧΈΛ׆͔ͯ͠ɺruby2.2ܥʹҠߦ • ruby2.2 • GGͷվળͷԸܙ 9 VASILY Inc. 2016
10
.
#1 middlewareͷߋ৽ • ruby 2.0 -> 2.2.3 10 VASILY Inc.
2016
11
.
#1 middlewareͷߋ৽ • ruby 2.0 -> 2.2.3 -> 2.3.1 •
ࠓͷ͡Ίʹ ruby 2.3.1Ҡߦ • ଞͷmiddlewareΧδϡΞϧʹ࠷৽ • nginx: 1.11.1 • mackerel-agent: 0.31.2 • td-agent: v2.3.1 (fluentd:v0.12.20) 11 VASILY Inc. 2016
12
.
#1 middlewareͷߋ৽ • ruby 2.4.0ਵ࣌ςετΛ͓͜ͳ͍ͬͯ·͢ • ruby 2.4.0-preview1 12 VASILY
Inc. 2016
13
.
#1 middlewareͷߋ৽ • docker ͷར • ΧδϡΞϧʹimageΛೖΕସ͑ΒΕΔ 13 VASILY Inc.
2016
14
.
#2 docker imageංେԽ • ͱ͋ΔλΠϛϯάͷdocker image size • ad_server: 924.6MB
• nginx: 134.1MB • td-agent: 448.4MB • mackerel-agent: 423.1MB 14 VASILY Inc. 2016
15
.
#2 docker imageංେԽ ͑ͬʂࢲͷdocker imageଠΓա͗..! 15 VASILY Inc. 2016
16
.
#2-2 docker imageݮ • ํ 1. Ͱ͖Δ͚ͩܰྔͳOSΛϕʔεʹ͢Δ 2. ෆඞཁͳύοέʔδΛΠϯετʔϧ͠ͳ͍ 3.
build͢Δͱ͖ʹ͚ͩඞཁͳύοέʔδΛదٓআ͢Δ 16 VASILY Inc. 2016
17
.
#2-2 docker imageݮ nginx • ެࣜͰalpine linuxΛϕʔεʹͨ͠ͷ ͕͋Δ • ࠾༻!
17 VASILY Inc. 2016
18
.
#2-2 docker imageݮ ad-server • ͱͱ CentOS6Λϕʔεʹbuild • 2ஈߏ •
ruby, supervisord, mysql ΛΈࠐΜͩimage • ↑ʹad-serverͱͯ͠ඞཁͳGemΛΈࠐΉ 18 VASILY Inc. 2016
19
.
#2-2 docker imageݮ ad-server • alpine linuxΛϕʔεʹͯ͠build • ॳظ •
build͢Δ͚ͩͯ͠আΛ͍ͯ͠ͳ͍ঢ়ଶ 19 VASILY Inc. 2016
20
.
#2-2 docker imageݮ ad-server 20 VASILY Inc. 2016
21
.
#2-2 docker imageݮ ad-server • ਫ਼ࠪ1 • mysql (mariadb-dev) ͕ͱͯେ͖͍...
• ෆཁͳόΠφϦΛআ rm /usr/lib/libmysqld* && \ rm /usr/bin/mysql* 21 VASILY Inc. 2016
22
.
#2-2 docker imageݮ ad-server • ਫ਼ࠪ2 • buildʹඞཁͳύοέʔδΛཧ • native-extensionͳGemΠϯετʔϧʹඞཁͳΛtemporaryͰ
install/uninstall • build-base (gccͳͲ) 22 VASILY Inc. 2016
23
.
#2 docker imageංେԽ • ࠷৽ͷ docker image size • ad_server:
924.6MB -> 342.7MB • nginx: 134.1MB -> 59.63MB • td-agent: 448.4MB -> 430.8MB • mackerel-agent: 423.1MB -> 357.5MB 23 VASILY Inc. 2016
24
.
#2 docker imageංେԽ • ࠷େ࣌ͷ6ׂ͙Β͍·Ͱݮ • blogॻ͖·ͨ͠ http://tech.vasily.jp/ docker_image_slim_in_alpinelinux 24
VASILY Inc. 2016
25
.
#3 base platform • EB w/multi-container docker • Amazon Linux
ͱ dockerͷΈ߹ΘͤͰෳ 25 VASILY Inc. 2016
26
.
#3 base platform • EB w/multi-container docker • Amazon Linux
2015.03 v1.4.0 docker 1.6.0 • Amazon Linux 2015.03 v1.4.1 docker 1.6.0 • Amazon Linux 2015.03 v1.4.6 docker 1.6.2 • Amazon Linux 2015.09 v2.0.8 docker 1.9.1 • Amazon Linux 2016.03 v2.1.0 docker 1.9.1 • Amazon Linux 2016.03 v2.1.1 docker 1.9.1 26 VASILY Inc. 2016
27
.
#3 base platform • ͍·গ͠ݹΊversionΛར༻ • Amazon Linux 2015.09 v2.0.8
docker 1.9.1 • Amazon Linux 2016.03mackerel-agent͕ىಈͰ͖ͳ͍ • docker͔ͩΒؔͳ͍ͱࢥ͍ͬͯͨΒىಈͰ͖ͳ͔ͬͨ.. 27 VASILY Inc. 2016
28
.
#3 base platform • hostͷdocker ͕গ͠ݹ͍ͷͰɺclientଆ͕৽͗͢͠ΔͱىಈͰ͖ ͳ͍.. • docker 1.9.1
• EBʹґଘ͠ͳ͍ܗߟ͍͖͍͑ͯͨ 28 VASILY Inc. 2016
29
.
#3 base platform • ࠓݟͨΒ, Amazon Linux 2016.03 docker 1.11.1
͕Ͱͯͨ • ໌ࢼ͍ͨ͠ͱ͓͍·͢ 29 VASILY Inc. 2016
30
.
#3 Envoirnment clone • production/stagingͷڥΛ·ΔͬͱίϐʔͰ͖Δ • base platformΛมߋ͢Δࡍͷςετ 30 VASILY
Inc. 2016
31
.
#3 Envoirnment clone 31 VASILY Inc. 2016
32
.
#3 Envoirnment clone 32 VASILY Inc. 2016
33
.
#3 Envoirnment clone • ΫϦοΫͰίϐʔͰ͖ΔͷͰͱͯศར! 33 VASILY Inc. 2016
34
.
#4 auto-scaling • ௨ৗͷΞΫηε૿Ճ/ݮগʹAuto-ScalingͰରԠ • EBͷweb consoleͷScaling configurationͰઃఆ • Sum
RequestCountϕʔε 34 VASILY Inc. 2016
35
.
#5 Time Scheduled-based auto- scaling • ADωοτϫʔΫ • iQONҎ֎ͷΞϓϦʹ৴ 35
VASILY Inc. 2016
36
.
#5 Time Scheduled-based auto- scaling • ͱ͋ΔΞϓϦ͕pushΛ͏ͭ • spike͢Δ •
௨ৗͷauto-scalingͰ͍͔ͭͳ͍ • pushʹΑΔspike͕ऩ·ͬͨ͋ͱʹىಈྃ.. • ҙຯͳ͍ 36 VASILY Inc. 2016
37
.
#5 Time Scheduled-based auto- scaling • time scheduled base auto-scalingΛઃఆ
• web console͔ΒઃఆͰ͖ͳ͍ͷͰaws cliͰߦ͏ 37 VASILY Inc. 2016
38
.
#5 Time Scheduled-based auto- scaling • time scheduled base auto-scalingΛઃఆ
• scale out aws autoscaling put-scheduled-update-group-action \ --auto-scaling-group-name awseb-e-XXXXXX \ --scheduled-action-name ScaleOut \ --recurrence "00 10 * * *" \ --desired-capacity 10 \ --min-size 10 \ --max-size 12 38 VASILY Inc. 2016
39
.
#5 Time Scheduled-based auto- scaling • time scheduled base auto-scalingΛઃఆ
• scale in aws autoscaling put-scheduled-update-group-action \ --auto-scaling-group-name awseb-e-XXXXXX \ --scheduled-action-name ScaleIn \ --recurrence "30 16 * * *" \ --desired-capacity 6 \ --min-size 6 \ --max-size 8 39 VASILY Inc. 2016
40
.
#5 Time Scheduled-based auto- scaling • ઌํӡ༻͠ͳ͕ΒͩͬͨͷͰɺ͏·͍͘ͳ͔͍͜ͱଟ͔ͬ ͨ • ݱࡏpushͷӡ༻͕ͱ·ͬͨ
• time scheduled base auto scalingઃఆͯ͠ͳ͍ 40 VASILY Inc. 2016
41
.
#6 ։ൃڥ • docker-compose ͰproductionͱಉͷڥΛىಈͰ͖ΔΑ͏ʹ ͍ͯ͠Δ • mysql,redis,memcacheࣾ։ൃαʔόΛࢀর 41 VASILY
Inc. 2016
42
.
#6 ։ൃڥ • ීஈMBPͷruby,nginxͰ։ൃΛ͍ͯ͠Δ • ։ൃ࣌جຊతʹҰਓͰΈͯΔͷͰContainerͷྑ͍ॴ͕ग़ͤ ͳ͍.. • docker-compose run
app rake ͰCIͷΈ࣮ࢪ 42 VASILY Inc. 2016
43
.
#7 Docker build • DockerfileผrepositoryͰGithubͰཧ • docker resitory(quay.io)ͷbuild hookະ࠾༻ •
खݩͷMBP(docker-machine) Ͱdocker build & pushͰӡ༻ • docker build -t quay.io/vasilyjp/ruby:2.3.1-alpine . • docker push quay.io/vasilyjp/ruby:2.3.1-alpine 43 VASILY Inc. 2016
44
.
#7 Docker build • ad-serverͷ2ஈͷimageCircleCI͕build • ઐ༻ͷbranchʹpush͢ΔͱCircleCI͕build & push 44
VASILY Inc. 2016
45
.
#7 Docker build • alpineԽ͔ͯ͠ΒDockerfile͓͡͞ΜԽ͕ݦஶʹਐΜͰΔؾ͠ ·͢.. 45 VASILY Inc. 2016
46
.
োฤ • ओʹӡ༻ϛεʹΑΔো͕ൃੜ͠·ͨ͠ 46 VASILY Inc. 2016
47
.
#1 quay.ioύεϫʔυมߋʹ͏ deployͷࣦഊ • imageΛquay.ioͰϗετ • ϚελͷύεϫʔυΛͦͷ··deploy࣌ʹ͍ͬͯͨ • .dockercfgʹϚελΞΧϯτͷͷΛೖΕ͍ͯͨ 47
VASILY Inc. 2016
48
.
#1 quay.ioύεϫʔυมߋʹ͏ deployͷࣦഊ • μϝ! ઈର!! 48 VASILY Inc. 2016
49
.
#1 quay.ioύεϫʔυมߋʹ͏ deployͷࣦഊ • ϚελύεϫʔυͷมߋΛߦͬͨ • ͦͷޙͷσϓϩΠʹࣦഊ • docker pull͕Ͱ͖ͳ͘ͳͬͨ..
49 VASILY Inc. 2016
50
.
#1 quay.ioύεϫʔυมߋʹ͏ deployͷࣦഊ • auto-scalingͷλΠϛϯάͱ૬·ͬͯΞΧϯτ͕ϩοΫ • େྔͷdocker pullͷࣦഊ͕ൃੜ • ϚελͷΞΧϯτͰdocker
pull ͕ϩʔΧϧڥͰͰ͖ͳ͘ ͳͬͨ 50 VASILY Inc. 2016
51
.
#1 quay.ioύεϫʔυมߋʹ͏ deployͷࣦഊ • ରԠ • σϓϩΠ༻ͷϩϘοτΞΧϯτΛ࡞(quay.ioͷػೳ) • ͦͷޙɺোى͖͍ͯͳ͍ 51
VASILY Inc. 2016
52
.
#1 quay.ioύεϫʔυมߋʹ͏ deployͷࣦഊ • quay.io͕SPOFʹͳ͍ͬͯΔͷͰɺηΧϯμϦʹ͍ͭͯݕ౼த • ECRϦϦʔεॳʹ؆୯ʹςετͨ͠ͷΈ • ݟ͋Εڭ͑ͯԼ͍͞! 52
VASILY Inc. 2016
53
.
#2 mackerel-agentͷΞϥʔτ • mackerelͰࢹ • auto-scalingͰΧδϡΞϧʹinstance͕ࢭ·Δ • unreachableͷΞϥʔτ͕ඈͿ • unreachable
Ͱ͋Δ͚ͲɺোͰͳ͍.. 53 VASILY Inc. 2016
54
.
#2 mackerel-agentͷΞϥʔτ • auto-scalingͳڥ༻ͷઃఆΛ࡞ͬͯΒͬͨ 54 VASILY Inc. 2016
55
.
#2 mackerel-agentͷΞ ϥʔτ ಈ͔ͳ͔ͬͨ.. 55 VASILY Inc. 2016
56
.
#2 mackerel-agentͷΞϥʔτ • Ұ୴ɺରͷroleͷunreachableͷΞϥʔτઃఆΛͬͨ.. • ࢹͷҙຯ͕ແ͍.. 56 VASILY Inc. 2016
57
.
#2 mackerel-agentͷΞϥʔτ • base platformΛมߋͨ͠Β͏·͍͘͘Α͏ʹͳͬͨ • Amazon Linux 2015.09 v2.0.8
docker 1.9.1 • ڥʹΑͬͯtrap͕ಈ͍ͨΓಈ͔ͳ͔ͬͨΓͯ͠Δͱͷ͜ͱ.. • EBͰͷঢ়گQiitaʹॻ͍ͯ·͢ -> http://qiita.com/tknzk/items/a6b1a9003ff26683b88a 57 VASILY Inc. 2016
58
.
#3 كʹσϓϩΠʹࣦഊ͢Δ • deployϑϩʔ • CircleCIͰmasterͷbranchͷCI͕·ΘΔͱdeploy hook͕͠Δ • eb deploy
ad-server-prod 58 VASILY Inc. 2016
59
.
#3 كʹσϓϩΠʹࣦഊ͢Δ • deploy͕ࣦഊ͢Δ͕࣌͋Δ • mackerel-agentͷcontainerͷstopʹࣦഊ͢Δ͜ͱ͕͋Δ [ec2-user@ip-xx-xx-xxx ~]$ sudo docker
exec CONTAINER_ID ps axu nsenter: Unable to fork: Cannot allocate memory 59 VASILY Inc. 2016
60
.
#3 كʹσϓϩΠʹࣦഊ͢Δ • ෮چͰ͖ͳ͍ͷͰinstance͝ͱstopͯ͠ରԠ • mackerel-agentͷதͷͷΑ͏ͳؾ͕͍ͯ͠Δ͕ɺৄࡉ Ѳग़དྷ͍ͯͳ͍ 60 VASILY Inc.
2016
61
.
#3 كʹσϓϩΠʹࣦഊ͢Δ • deploy͕ࣦഊͨ͠Β • ΧδϡΞϧʹinstanceΛstopͯ͠ೖΕସ͑ • ࠶deploy 61 VASILY
Inc. 2016
62
.
#3 كʹσϓϩΠʹࣦഊ͢Δ • auto-scalingͷhealth-checkͷରΛELBʹ͢Δ • deployʹࣦഊͯ͠ELB͔ΒΓ͞Εͨঢ়ଶ • ߏʹͳ͍ͬͯΕͳ͘ӡ༻Ͱ͖Δ 62 VASILY
Inc. 2016
63
.
#4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ ͖Γ) • deployCircleCI͔Β eb deploy ad-server-prod ͰσϓϩΠ
• applicationΛzipͨ͠ϑΝΠϧΛs3ʹput • ֤instance͕pullͯ͘͠Δ 63 VASILY Inc. 2016
64
.
#4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ ͖Γ) • ಛఆͷawsebcli s3ʹpush͢ΔϑΝΠϧ໊͕gitͷcommithash 4 ܻͰݻఆ •
ຊ൪ӡ༻։࢝લʹੜͨ͠ϑΝΠϧ໊ͱconflictͯ͠deploy͕ί έͨ • ͜ͷ࣌ίέ͚͕ͨͩͩͬͨޭ͢ΔͱσάϨ͢Δ͜ͱʹͳͬ ͨ 64 VASILY Inc. 2016
65
.
#4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ ͖Γ) • ݱ࣌ͰɺTimeStamp͕suffixͱͯͭ͘͠Α͏ʹͳͬͨ • versino 3.6.2 Ҏ߱
65 VASILY Inc. 2016
66
.
#4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ ͖Γ) • ఆظతʹapplication versionsͷͱ͜Ζʹ͋ΔϑΝΠϧΛআ • σϑΥϧτ500·Ͱอ࣋Ͱ͖Δ •
ݹ͍ͷΛ͍ͯͯ͋͠·Γҙຯͳ͍ͷͰফͯ͠Δ 66 VASILY Inc. 2016
67
.
Conclusion • 1ؒ EB w/multi-container docker Ͱӡ༻ͯ͠ى͖ͨ͜ͱΛ͓ ͠·ͨ͠ • ӡ༻ϛεʹΑΔো͋Γ·͕ͨ͠ɺEBdockerʹىҼ͢Δো
ͳ͠ • alpine linuxΛproductionͰӡ༻தͰ͢ • iQONຊମͷAPI͍ͣΕ... 67 VASILY Inc. 2016
68
.
We are hiring! • info@vasily.jp 68 VASILY Inc. 2016
69
.
end ͋Γ͕ͱ͏͍͟͝·ͨ͠ 69 VASILY Inc. 2016
Takumi KANZAKI
tknzk
reverentgeek
rmw
skipperchong
carmenhchung
keathley
productmarketing
clairelew
thoeni
jcasabona
![#3 كʹσϓϩΠʹࣦഊ͢Δ • deploy͕ࣦഊ͢Δ͕࣌͋Δ • mackerel-agentͷcontainerͷstopʹࣦഊ͢Δ͜ͱ͕͋Δ [ec2-user@ip-xx-xx-xxx ~]$ sudo docker](https://files.speakerdeck.com/presentations/2531b164d8a04565aff9d38366a1b2d2/slide_58.jpg){kind=link}
