eb_with_multi_container_docker

# JAWS-UGおコンテナ支部 #5

- EB with multi-container Docker をproductionで1年間運用して起きたこと

Transcript

1. JAWS-UG͓ίϯςφࢧ ෦ #5 EB with multi-container Docker Λ productionͰ1೥ؒӡ༻ͯ͠ى͖ͨ͜ ͱ 1 VASILY Inc. 2016
2. Me • Takumi Kanzaki (@tknzk) • VASILY Inc. Backend Engineer • 2015.04~ • iQONAD/ࣾ಺༻੥ٻॻπʔϧ Λ୲౰ ($∀$) • ࠷ۙiQON API ΋୲౰ 2 VASILY Inc. 2016
3. 3 VASILY Inc. 2016
4. iQONAD • iQON͓Αͼ֎෦ΞϓϦʹ޿ࠂ഑৴Λߦ͍ͬͯΔADωοτϫʔΫ • 2014.10~ • ॳظ࣮૷~ϩʔϯν • 2weeks 4 VASILY Inc. 2016
5. iQONAD • software, middleware • Ruby, Rails, sinatra • MySQL, memcached, Redis • infra • EC2, RDS • docker, ElasticBeanstalk (2015.06~) • auto-scaling 5 VASILY Inc. 2016
6. EB w/ multi-container docker • 2015/06~ • 1೥ؒproduction؀ڥͰӡ༻ͯ͠ى͖ͨ͜ͱΛ͝঺հ͠·͢ 6 VASILY Inc. 2016
7. Container ߏ੒ • 4ͭͷcontainerͰՔಇ 1. ຊମͷΞϓϦέʔγϣϯ͕͸͍ͬͨContainer (ad-server) • ruby,supervisord 2. httpͷϦΫΤετΛड͚෇͚ΔͨΊͷnginx 3. logίϨΫλͱͯ͠ͷtd-agent 4. ؂ࢹ༻ͷmackerel-agent 7 VASILY Inc. 2016
8. ։ൃ/ӡ༻ฤ 8 VASILY Inc. 2016
9. #1 middlewareͷߋ৽ • ruby 2.0 -> 2.2.3 • ϩʔϯν·Ͱ2िؒͰ։ൃΛߦͬͨ͜ͱ͔Βطଘͷ։ൃӡ༻࣮੷ ͷ͋Δruby 2.0.0 Ͱ։ൃ/ӡ༻͞Ε͍ͯ·ͨ͠ • 2015/11ʹdockerͷڧΈΛ׆͔ͯ͠ɺruby2.2ܥʹҠߦ • ruby2.2 • GGͷվળͷԸܙ 9 VASILY Inc. 2016
10. #1 middlewareͷߋ৽ • ruby 2.0 -> 2.2.3 10 VASILY Inc. 2016
11. #1 middlewareͷߋ৽ • ruby 2.0 -> 2.2.3 -> 2.3.1 • ࠓ೥ͷ͸͡Ίʹ ruby 2.3.1΁Ҡߦ • ଞͷmiddleware΋ΧδϡΞϧʹ࠷৽΁ • nginx: 1.11.1 • mackerel-agent: 0.31.2 • td-agent: v2.3.1 (fluentd:v0.12.20) 11 VASILY Inc. 2016
12. #1 middlewareͷߋ৽ • ruby 2.4.0΋ਵ࣌ςετΛ͓͜ͳ͍ͬͯ·͢ • ruby 2.4.0-preview1 12 VASILY Inc. 2016
13. #1 middlewareͷߋ৽ • docker ͷར఺ • ΧδϡΞϧʹimageΛೖΕସ͑ΒΕΔ 13 VASILY Inc. 2016
14. #2 docker imageංେԽ • ͱ͋ΔλΠϛϯάͷdocker image size • ad_server: 924.6MB • nginx: 134.1MB • td-agent: 448.4MB • mackerel-agent: 423.1MB 14 VASILY Inc. 2016
15. #2 docker imageංେԽ ͑ͬʂࢲͷdocker imageଠΓա͗..! 15 VASILY Inc. 2016
16. #2-2 docker image࡟ݮ • ํ਑ 1. Ͱ͖Δ͚ͩܰྔͳOSΛϕʔεʹ͢Δ 2. ෆඞཁͳύοέʔδΛΠϯετʔϧ͠ͳ͍ 3. build͢Δͱ͖ʹ͚ͩඞཁͳύοέʔδΛదٓ࡟আ͢Δ 16 VASILY Inc. 2016
17. #2-2 docker image࡟ݮ nginx • ެࣜͰalpine linuxΛϕʔεʹͨ͠΋ͷ ͕͋Δ • ࠾༻! 17 VASILY Inc. 2016
18. #2-2 docker image࡟ݮ ad-server • ΋ͱ΋ͱ CentOS6Λϕʔεʹbuild • 2ஈߏ੒ • ruby, supervisord, mysql Λ૊ΈࠐΜͩimage • ↑ʹad-serverͱͯ͠ඞཁͳGemΛ૊ΈࠐΉ 18 VASILY Inc. 2016
19. #2-2 docker image࡟ݮ ad-server • alpine linuxΛϕʔεʹͯ͠build • ॳظ • build͢Δ͚ͩͯ͠૟আΛ͍ͯ͠ͳ͍ঢ়ଶ 19 VASILY Inc. 2016
20. #2-2 docker image࡟ݮ ad-server 20 VASILY Inc. 2016
21. #2-2 docker image࡟ݮ ad-server • ਫ਼ࠪ1 • mysql (mariadb-dev) ͕ͱͯ΋େ͖͍... • ෆཁͳόΠφϦΛ࡟আ rm /usr/lib/libmysqld* && \ rm /usr/bin/mysql* 21 VASILY Inc. 2016
22. #2-2 docker image࡟ݮ ad-server • ਫ਼ࠪ2 • buildʹඞཁͳύοέʔδΛ੔ཧ • native-extensionͳGemΠϯετʔϧʹඞཁͳ෺ΛtemporaryͰ install/uninstall • build-base (gccͳͲ) 22 VASILY Inc. 2016
23. #2 docker imageංେԽ • ࠷৽ͷ docker image size • ad_server: 924.6MB -> 342.7MB • nginx: 134.1MB -> 59.63MB • td-agent: 448.4MB -> 430.8MB • mackerel-agent: 423.1MB -> 357.5MB 23 VASILY Inc. 2016
24. #2 docker imageංେԽ • ࠷େ࣌ͷ6ׂ͙Β͍·Ͱ࡟ݮ • blogॻ͖·ͨ͠ http://tech.vasily.jp/ docker_image_slim_in_alpinelinux 24 VASILY Inc. 2016
25. #3 base platform • EB w/multi-container docker • Amazon Linux ͱ dockerͷ૊Έ߹ΘͤͰෳ਺ 25 VASILY Inc. 2016
26. #3 base platform • EB w/multi-container docker • Amazon Linux 2015.03 v1.4.0 docker 1.6.0 • Amazon Linux 2015.03 v1.4.1 docker 1.6.0 • Amazon Linux 2015.03 v1.4.6 docker 1.6.2 • Amazon Linux 2015.09 v2.0.8 docker 1.9.1 • Amazon Linux 2016.03 v2.1.0 docker 1.9.1 • Amazon Linux 2016.03 v2.1.1 docker 1.9.1 26 VASILY Inc. 2016
27. #3 base platform • ͍·͸গ͠ݹΊversionΛར༻ • Amazon Linux 2015.09 v2.0.8 docker 1.9.1 • Amazon Linux 2016.03͸mackerel-agent͕ىಈͰ͖ͳ͍໰୊ • docker͔ͩΒؔ܎ͳ͍ͱࢥ͍ͬͯͨΒىಈͰ͖ͳ͔ͬͨ.. 27 VASILY Inc. 2016
28. #3 base platform • hostͷdocker ͕গ͠ݹ͍ͷͰɺclientଆ͕৽͗͢͠ΔͱىಈͰ͖ ͳ͍໰୊΋.. • docker 1.9.1 • EBʹґଘ͠ͳ͍ܗ΋ߟ͍͖͍͑ͯͨ 28 VASILY Inc. 2016
29. #3 base platform • ࠓݟͨΒ, Amazon Linux 2016.03 docker 1.11.1 ͕Ͱͯͨ • ໌೔ࢼ͍ͨ͠ͱ͓΋͍·͢ 29 VASILY Inc. 2016
30. #3 Envoirnment clone • production/stagingͷ؀ڥΛ·ΔͬͱίϐʔͰ͖Δ • base platformΛมߋ͢Δࡍͷςετ 30 VASILY Inc. 2016
31. #3 Envoirnment clone 31 VASILY Inc. 2016
32. #3 Envoirnment clone 32 VASILY Inc. 2016
33. #3 Envoirnment clone • ਺ΫϦοΫͰίϐʔͰ͖ΔͷͰͱͯ΋ศར! 33 VASILY Inc. 2016
34. #4 auto-scaling • ௨ৗͷΞΫηε૿Ճ/ݮগʹ͸Auto-ScalingͰରԠ • EBͷweb consoleͷScaling configurationͰઃఆ • Sum RequestCountϕʔε 34 VASILY Inc. 2016
35. #5 Time Scheduled-based auto- scaling • ADωοτϫʔΫ • iQONҎ֎ͷΞϓϦʹ΋഑৴ 35 VASILY Inc. 2016
36. #5 Time Scheduled-based auto- scaling • ͱ͋ΔΞϓϦ͕pushΛ͏ͭ • spike͢Δ • ௨ৗͷauto-scalingͰ͸௥͍͔ͭͳ͍ • pushʹΑΔspike͕ऩ·ͬͨ͋ͱʹىಈ׬ྃ.. • ҙຯͳ͍ 36 VASILY Inc. 2016
37. #5 Time Scheduled-based auto- scaling • time scheduled base auto-scalingΛઃఆ • web console͔Β͸ઃఆͰ͖ͳ͍ͷͰaws cliͰߦ͏ 37 VASILY Inc. 2016
38. #5 Time Scheduled-based auto- scaling • time scheduled base auto-scalingΛઃఆ • scale out aws autoscaling put-scheduled-update-group-action \ --auto-scaling-group-name awseb-e-XXXXXX \ --scheduled-action-name ScaleOut \ --recurrence "00 10 * * *" \ --desired-capacity 10 \ --min-size 10 \ --max-size 12 38 VASILY Inc. 2016
39. #5 Time Scheduled-based auto- scaling • time scheduled base auto-scalingΛઃఆ • scale in aws autoscaling put-scheduled-update-group-action \ --auto-scaling-group-name awseb-e-XXXXXX \ --scheduled-action-name ScaleIn \ --recurrence "30 16 * * *" \ --desired-capacity 6 \ --min-size 6 \ --max-size 8 39 VASILY Inc. 2016
40. #5 Time Scheduled-based auto- scaling • ઌํ΋ӡ༻͠ͳ͕ΒͩͬͨͷͰɺ͏·͍͘ͳ͔͍͜ͱ΋ଟ͔ͬ ͨ • ݱࡏ͸pushͷӡ༻͕ͱ·ͬͨ • time scheduled base auto scaling͸ઃఆͯ͠ͳ͍ 40 VASILY Inc. 2016
41. #6 ։ൃ؀ڥ • docker-compose Ͱproductionͱಉ౳ͷ؀ڥΛىಈͰ͖ΔΑ͏ʹ ͍ͯ͠Δ • mysql,redis,memcache͸ࣾ಺։ൃαʔόΛࢀর 41 VASILY Inc. 2016
42. #6 ։ൃ؀ڥ • ීஈ͸MBPͷruby,nginxͰ։ൃΛ͍ͯ͠Δ • ։ൃ࣌͸جຊతʹҰਓͰΈͯΔͷͰContainerͷྑ͍ॴ͕ग़ͤ ͳ͍.. • docker-compose run app rake ͰCIͷΈ࣮ࢪ 42 VASILY Inc. 2016
43. #7 Docker build • Dockerfile͸ผrepositoryͰGithubͰ؅ཧ • docker resitory(quay.io)ͷbuild hook͸ະ࠾༻ • खݩͷMBP(docker-machine) Ͱdocker build & pushͰӡ༻ • docker build -t quay.io/vasilyjp/ruby:2.3.1-alpine . • docker push quay.io/vasilyjp/ruby:2.3.1-alpine 43 VASILY Inc. 2016
44. #7 Docker build • ad-serverͷ2ஈ໨ͷimage͸CircleCI͕build • ઐ༻ͷbranchʹpush͢ΔͱCircleCI͕build & push 44 VASILY Inc. 2016
45. #7 Docker build • alpineԽ͔ͯ͠Β͸Dockerfile͓͡͞ΜԽ͕ݦஶʹਐΜͰΔؾ΋͠ ·͢.. 45 VASILY Inc. 2016
46. ো֐ฤ • ओʹӡ༻ϛεʹΑΔো֐͕ൃੜ͠·ͨ͠ 46 VASILY Inc. 2016
47. #1 quay.ioύεϫʔυมߋʹ൐͏ deployͷࣦഊ • imageΛquay.ioͰϗετ • ϚελͷύεϫʔυΛͦͷ··deploy࣌ʹ΋࢖͍ͬͯͨ • .dockercfgʹϚελΞΧ΢ϯτͷ΋ͷΛೖΕ͍ͯͨ 47 VASILY Inc. 2016
48. #1 quay.ioύεϫʔυมߋʹ൐͏ deployͷࣦഊ • μϝ! ઈର!! 48 VASILY Inc. 2016
49. #1 quay.ioύεϫʔυมߋʹ൐͏ deployͷࣦഊ • ϚελύεϫʔυͷมߋΛߦͬͨ • ͦͷޙͷσϓϩΠʹࣦഊ • docker pull͕Ͱ͖ͳ͘ͳͬͨ.. 49 VASILY Inc. 2016
50. #1 quay.ioύεϫʔυมߋʹ൐͏ deployͷࣦഊ • auto-scalingͷλΠϛϯάͱ΋૬·ͬͯΞΧ΢ϯτ͕ϩοΫ • େྔͷdocker pullͷࣦഊ͕ൃੜ • ϚελͷΞΧ΢ϯτͰdocker pull ͕ϩʔΧϧ؀ڥͰ΋Ͱ͖ͳ͘ ͳͬͨ 50 VASILY Inc. 2016
51. #1 quay.ioύεϫʔυมߋʹ൐͏ deployͷࣦഊ • ରԠ • σϓϩΠ༻ͷϩϘοτΞΧ΢ϯτΛ࡞੒(quay.ioͷػೳ) • ͦͷޙɺো֐͸ى͖͍ͯͳ͍ 51 VASILY Inc. 2016
52. #1 quay.ioύεϫʔυมߋʹ൐͏ deployͷࣦഊ • quay.io͕SPOFʹͳ͍ͬͯΔͷͰɺηΧϯμϦʹ͍ͭͯ͸ݕ౼த • ECR͸ϦϦʔε౰ॳʹ؆୯ʹςετͨ͠ͷΈ • ஌ݟ͋Ε͹ڭ͑ͯԼ͍͞! 52 VASILY Inc. 2016
53. #2 mackerel-agentͷΞϥʔτ • mackerelͰ؂ࢹ • auto-scalingͰΧδϡΞϧʹinstance͕ࢭ·Δ • unreachableͷΞϥʔτ͕ඈͿ • unreachable Ͱ͸͋Δ͚Ͳɺো֐Ͱ͸ͳ͍.. 53 VASILY Inc. 2016
54. #2 mackerel-agentͷΞϥʔτ • auto-scalingͳ؀ڥ༻ͷઃఆΛ࡞ͬͯ΋Βͬͨ 54 VASILY Inc. 2016
55. #2 mackerel-agentͷΞ ϥʔτ ಈ͔ͳ͔ͬͨ.. 55 VASILY Inc. 2016
56. #2 mackerel-agentͷΞϥʔτ • Ұ୴ɺର৅ͷroleͷunreachableͷΞϥʔτઃఆΛ੾ͬͨ.. • ؂ࢹͷҙຯ͕ແ͍.. 56 VASILY Inc. 2016
57. #2 mackerel-agentͷΞϥʔτ • base platformΛมߋͨ͠Β͏·͍͘͘Α͏ʹͳͬͨ • Amazon Linux 2015.09 v2.0.8 docker 1.9.1 • ؀ڥʹΑͬͯtrap͕ಈ͍ͨΓಈ͔ͳ͔ͬͨΓͯ͠Δͱͷ͜ͱ.. • EBͰͷঢ়گ͸Qiitaʹॻ͍ͯ·͢ -> http://qiita.com/tknzk/items/a6b1a9003ff26683b88a 57 VASILY Inc. 2016
58. #3 كʹσϓϩΠʹࣦഊ͢Δ • deployϑϩʔ • CircleCIͰmasterͷbranchͷCI͕·ΘΔͱdeploy hook͕͸͠Δ • eb deploy ad-server-prod 58 VASILY Inc. 2016
59. #3 كʹσϓϩΠʹࣦഊ͢Δ • deploy͕ࣦഊ͢Δ͕࣌͋Δ • mackerel-agentͷcontainerͷstopʹࣦഊ͢Δ͜ͱ͕͋Δ [ec2-user@ip-xx-xx-xxx ~]$ sudo docker exec CONTAINER_ID ps axu nsenter: Unable to fork: Cannot allocate memory 59 VASILY Inc. 2016
60. #3 كʹσϓϩΠʹࣦഊ͢Δ • ෮چͰ͖ͳ͍ͷͰinstance͝ͱstopͯ͠ରԠ • mackerel-agentͷதͷ໰୊ͷΑ͏ͳؾ͕͍ͯ͠Δ͕ɺৄࡉ͸೺ Ѳग़དྷ͍ͯͳ͍ 60 VASILY Inc. 2016
61. #3 كʹσϓϩΠʹࣦഊ͢Δ • deploy͕ࣦഊͨ͠Β • ΧδϡΞϧʹinstanceΛstopͯ͠ೖΕସ͑ • ࠶deploy 61 VASILY Inc. 2016
62. #3 كʹσϓϩΠʹࣦഊ͢Δ • auto-scalingͷhealth-checkͷର৅ΛELBʹ͢Δ • deployʹࣦഊͯ͠΋ELB͔Β੾Γ཭͞Εͨঢ়ଶ • ৑௕ߏ੒ʹͳ͍ͬͯΕ͹໰୊ͳ͘ӡ༻Ͱ͖Δ 62 VASILY Inc. 2016
63. #4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ ౓͖Γ) • deploy͸CircleCI͔Β eb deploy ad-server-prod ͰσϓϩΠ • applicationΛzipͨ͠ϑΝΠϧΛs3ʹput • ֤instance͕pullͯ͘͠Δ 63 VASILY Inc. 2016
64. #4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ ౓͖Γ) • ಛఆͷawsebcli͸ s3ʹpush͢ΔϑΝΠϧ໊͕gitͷcommithash 4 ܻͰݻఆ • ຊ൪ӡ༻։࢝લʹੜ੒ͨ͠ϑΝΠϧ໊ͱconflictͯ͠deploy͕ί έͨ • ͜ͷ࣌͸ίέ͚͕ͨͩͩͬͨ੒ޭ͢ΔͱσάϨ͢Δ͜ͱʹͳͬ ͨ 64 VASILY Inc. 2016
65. #4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ ౓͖Γ) • ݱ࣌఺Ͱ͸ɺTimeStamp͕suffixͱͯͭ͘͠Α͏ʹͳͬͨ • versino 3.6.2 Ҏ߱ 65 VASILY Inc. 2016
66. #4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ ౓͖Γ) • ఆظతʹapplication versionsͷͱ͜Ζʹ͋ΔϑΝΠϧΛ૟আ • σϑΥϧτ500·Ͱ͸อ࣋Ͱ͖Δ • ݹ͍ͷΛ࢒͍ͯͯ͠΋͋·Γҙຯͳ͍ͷͰফͯ͠Δ 66 VASILY Inc. 2016
67. Conclusion • 1೥ؒ EB w/multi-container docker Ͱӡ༻ͯ͠ى͖ͨ͜ͱΛ͓࿩ ͠·ͨ͠ • ӡ༻ϛεʹΑΔো֐΋͋Γ·͕ͨ͠ɺEB΍dockerʹىҼ͢Δো ֐͸ͳ͠ • alpine linuxΛproductionͰӡ༻தͰ͢ • iQONຊମͷAPI΋͍ͣΕ͸... 67 VASILY Inc. 2016
68. We are hiring! • info@vasily.jp 68 VASILY Inc. 2016
69. end ͋Γ͕ͱ͏͍͟͝·ͨ͠ 69 VASILY Inc. 2016