eb_with_multi_container_docker

Transcript

1. JAWS-UG͓ίϯςφࢧ ෦ #5 EB with multi-container Docker Λ productionͰ1೥ؒӡ༻ͯ͠ى͖ͨ͜ ͱ

   1 VASILY Inc. 2016

2. Me • Takumi Kanzaki (@tknzk) • VASILY Inc. Backend Engineer

   • 2015.04~ • iQONAD/ࣾ಺༻੥ٻॻπʔϧ Λ୲౰ ($∀$) • ࠷ۙiQON API ΋୲౰ 2 VASILY Inc. 2016

3. 3 VASILY Inc. 2016

4. iQONAD • iQON͓Αͼ֎෦ΞϓϦʹ޿ࠂ഑৴Λߦ͍ͬͯΔADωοτϫʔΫ • 2014.10~ • ॳظ࣮૷~ϩʔϯν • 2weeks 4

   VASILY Inc. 2016

5. iQONAD • software, middleware • Ruby, Rails, sinatra • MySQL,

   memcached, Redis • infra • EC2, RDS • docker, ElasticBeanstalk (2015.06~) • auto-scaling 5 VASILY Inc. 2016

6. EB w/ multi-container docker • 2015/06~ • 1೥ؒproduction؀ڥͰӡ༻ͯ͠ى͖ͨ͜ͱΛ͝঺հ͠·͢ 6 VASILY

   Inc. 2016

7. Container ߏ੒ • 4ͭͷcontainerͰՔಇ 1. ຊମͷΞϓϦέʔγϣϯ͕͸͍ͬͨContainer (ad-server) • ruby,supervisord 2.

   httpͷϦΫΤετΛड͚෇͚ΔͨΊͷnginx 3. logίϨΫλͱͯ͠ͷtd-agent 4. ؂ࢹ༻ͷmackerel-agent 7 VASILY Inc. 2016

8. ։ൃ/ӡ༻ฤ 8 VASILY Inc. 2016

9. #1 middlewareͷߋ৽ • ruby 2.0 -> 2.2.3 • ϩʔϯν·Ͱ2िؒͰ։ൃΛߦͬͨ͜ͱ͔Βطଘͷ։ൃӡ༻࣮੷ ͷ͋Δruby

   2.0.0 Ͱ։ൃ/ӡ༻͞Ε͍ͯ·ͨ͠ • 2015/11ʹdockerͷڧΈΛ׆͔ͯ͠ɺruby2.2ܥʹҠߦ • ruby2.2 • GGͷվળͷԸܙ 9 VASILY Inc. 2016

10. #1 middlewareͷߋ৽ • ruby 2.0 -> 2.2.3 10 VASILY Inc.

    2016

11. #1 middlewareͷߋ৽ • ruby 2.0 -> 2.2.3 -> 2.3.1 •

    ࠓ೥ͷ͸͡Ίʹ ruby 2.3.1΁Ҡߦ • ଞͷmiddleware΋ΧδϡΞϧʹ࠷৽΁ • nginx: 1.11.1 • mackerel-agent: 0.31.2 • td-agent: v2.3.1 (fluentd:v0.12.20) 11 VASILY Inc. 2016

12. #1 middlewareͷߋ৽ • ruby 2.4.0΋ਵ࣌ςετΛ͓͜ͳ͍ͬͯ·͢ • ruby 2.4.0-preview1 12 VASILY

    Inc. 2016

13. #1 middlewareͷߋ৽ • docker ͷར఺ • ΧδϡΞϧʹimageΛೖΕସ͑ΒΕΔ 13 VASILY Inc.

    2016

14. #2 docker imageංେԽ • ͱ͋ΔλΠϛϯάͷdocker image size • ad_server: 924.6MB

    • nginx: 134.1MB • td-agent: 448.4MB • mackerel-agent: 423.1MB 14 VASILY Inc. 2016

15. #2 docker imageංେԽ ͑ͬʂࢲͷdocker imageଠΓա͗..! 15 VASILY Inc. 2016

16. #2-2 docker image࡟ݮ • ํ਑ 1. Ͱ͖Δ͚ͩܰྔͳOSΛϕʔεʹ͢Δ 2. ෆඞཁͳύοέʔδΛΠϯετʔϧ͠ͳ͍ 3.

    build͢Δͱ͖ʹ͚ͩඞཁͳύοέʔδΛదٓ࡟আ͢Δ 16 VASILY Inc. 2016

17. #2-2 docker image࡟ݮ nginx • ެࣜͰalpine linuxΛϕʔεʹͨ͠΋ͷ ͕͋Δ • ࠾༻!

    17 VASILY Inc. 2016

18. #2-2 docker image࡟ݮ ad-server • ΋ͱ΋ͱ CentOS6Λϕʔεʹbuild • 2ஈߏ੒ •

    ruby, supervisord, mysql Λ૊ΈࠐΜͩimage • ↑ʹad-serverͱͯ͠ඞཁͳGemΛ૊ΈࠐΉ 18 VASILY Inc. 2016

19. #2-2 docker image࡟ݮ ad-server • alpine linuxΛϕʔεʹͯ͠build • ॳظ •

    build͢Δ͚ͩͯ͠૟আΛ͍ͯ͠ͳ͍ঢ়ଶ 19 VASILY Inc. 2016

20. #2-2 docker image࡟ݮ ad-server 20 VASILY Inc. 2016

21. #2-2 docker image࡟ݮ ad-server • ਫ਼ࠪ1 • mysql (mariadb-dev) ͕ͱͯ΋େ͖͍...

    • ෆཁͳόΠφϦΛ࡟আ rm /usr/lib/libmysqld* && \ rm /usr/bin/mysql* 21 VASILY Inc. 2016

22. #2-2 docker image࡟ݮ ad-server • ਫ਼ࠪ2 • buildʹඞཁͳύοέʔδΛ੔ཧ • native-extensionͳGemΠϯετʔϧʹඞཁͳ෺ΛtemporaryͰ

    install/uninstall • build-base (gccͳͲ) 22 VASILY Inc. 2016

23. #2 docker imageංେԽ • ࠷৽ͷ docker image size • ad_server:

    924.6MB -> 342.7MB • nginx: 134.1MB -> 59.63MB • td-agent: 448.4MB -> 430.8MB • mackerel-agent: 423.1MB -> 357.5MB 23 VASILY Inc. 2016

24. #2 docker imageංେԽ • ࠷େ࣌ͷ6ׂ͙Β͍·Ͱ࡟ݮ • blogॻ͖·ͨ͠ http://tech.vasily.jp/ docker_image_slim_in_alpinelinux 24

    VASILY Inc. 2016

25. #3 base platform • EB w/multi-container docker • Amazon Linux

    ͱ dockerͷ૊Έ߹ΘͤͰෳ਺ 25 VASILY Inc. 2016

26. #3 base platform • EB w/multi-container docker • Amazon Linux

    2015.03 v1.4.0 docker 1.6.0 • Amazon Linux 2015.03 v1.4.1 docker 1.6.0 • Amazon Linux 2015.03 v1.4.6 docker 1.6.2 • Amazon Linux 2015.09 v2.0.8 docker 1.9.1 • Amazon Linux 2016.03 v2.1.0 docker 1.9.1 • Amazon Linux 2016.03 v2.1.1 docker 1.9.1 26 VASILY Inc. 2016

27. #3 base platform • ͍·͸গ͠ݹΊversionΛར༻ • Amazon Linux 2015.09 v2.0.8

    docker 1.9.1 • Amazon Linux 2016.03͸mackerel-agent͕ىಈͰ͖ͳ͍໰୊ • docker͔ͩΒؔ܎ͳ͍ͱࢥ͍ͬͯͨΒىಈͰ͖ͳ͔ͬͨ.. 27 VASILY Inc. 2016

28. #3 base platform • hostͷdocker ͕গ͠ݹ͍ͷͰɺclientଆ͕৽͗͢͠ΔͱىಈͰ͖ ͳ͍໰୊΋.. • docker 1.9.1

    • EBʹґଘ͠ͳ͍ܗ΋ߟ͍͖͍͑ͯͨ 28 VASILY Inc. 2016

29. #3 base platform • ࠓݟͨΒ, Amazon Linux 2016.03 docker 1.11.1

    ͕Ͱͯͨ • ໌೔ࢼ͍ͨ͠ͱ͓΋͍·͢ 29 VASILY Inc. 2016

30. #3 Envoirnment clone • production/stagingͷ؀ڥΛ·ΔͬͱίϐʔͰ͖Δ • base platformΛมߋ͢Δࡍͷςετ 30 VASILY

    Inc. 2016

31. #3 Envoirnment clone 31 VASILY Inc. 2016

32. #3 Envoirnment clone 32 VASILY Inc. 2016

33. #3 Envoirnment clone • ਺ΫϦοΫͰίϐʔͰ͖ΔͷͰͱͯ΋ศར! 33 VASILY Inc. 2016

34. #4 auto-scaling • ௨ৗͷΞΫηε૿Ճ/ݮগʹ͸Auto-ScalingͰରԠ • EBͷweb consoleͷScaling configurationͰઃఆ • Sum

    RequestCountϕʔε 34 VASILY Inc. 2016

35. #5 Time Scheduled-based auto- scaling • ADωοτϫʔΫ • iQONҎ֎ͷΞϓϦʹ΋഑৴ 35

    VASILY Inc. 2016

36. #5 Time Scheduled-based auto- scaling • ͱ͋ΔΞϓϦ͕pushΛ͏ͭ • spike͢Δ •

    ௨ৗͷauto-scalingͰ͸௥͍͔ͭͳ͍ • pushʹΑΔspike͕ऩ·ͬͨ͋ͱʹىಈ׬ྃ.. • ҙຯͳ͍ 36 VASILY Inc. 2016

37. #5 Time Scheduled-based auto- scaling • time scheduled base auto-scalingΛઃఆ

    • web console͔Β͸ઃఆͰ͖ͳ͍ͷͰaws cliͰߦ͏ 37 VASILY Inc. 2016

38. #5 Time Scheduled-based auto- scaling • time scheduled base auto-scalingΛઃఆ

    • scale out aws autoscaling put-scheduled-update-group-action \ --auto-scaling-group-name awseb-e-XXXXXX \ --scheduled-action-name ScaleOut \ --recurrence "00 10 * * *" \ --desired-capacity 10 \ --min-size 10 \ --max-size 12 38 VASILY Inc. 2016

39. #5 Time Scheduled-based auto- scaling • time scheduled base auto-scalingΛઃఆ

    • scale in aws autoscaling put-scheduled-update-group-action \ --auto-scaling-group-name awseb-e-XXXXXX \ --scheduled-action-name ScaleIn \ --recurrence "30 16 * * *" \ --desired-capacity 6 \ --min-size 6 \ --max-size 8 39 VASILY Inc. 2016

40. #5 Time Scheduled-based auto- scaling • ઌํ΋ӡ༻͠ͳ͕ΒͩͬͨͷͰɺ͏·͍͘ͳ͔͍͜ͱ΋ଟ͔ͬ ͨ • ݱࡏ͸pushͷӡ༻͕ͱ·ͬͨ

    • time scheduled base auto scaling͸ઃఆͯ͠ͳ͍ 40 VASILY Inc. 2016

41. #6 ։ൃ؀ڥ • docker-compose Ͱproductionͱಉ౳ͷ؀ڥΛىಈͰ͖ΔΑ͏ʹ ͍ͯ͠Δ • mysql,redis,memcache͸ࣾ಺։ൃαʔόΛࢀর 41 VASILY

    Inc. 2016

42. #6 ։ൃ؀ڥ • ීஈ͸MBPͷruby,nginxͰ։ൃΛ͍ͯ͠Δ • ։ൃ࣌͸جຊతʹҰਓͰΈͯΔͷͰContainerͷྑ͍ॴ͕ग़ͤ ͳ͍.. • docker-compose run

    app rake ͰCIͷΈ࣮ࢪ 42 VASILY Inc. 2016

43. #7 Docker build • Dockerfile͸ผrepositoryͰGithubͰ؅ཧ • docker resitory(quay.io)ͷbuild hook͸ະ࠾༻ •

    खݩͷMBP(docker-machine) Ͱdocker build & pushͰӡ༻ • docker build -t quay.io/vasilyjp/ruby:2.3.1-alpine . • docker push quay.io/vasilyjp/ruby:2.3.1-alpine 43 VASILY Inc. 2016

44. #7 Docker build • ad-serverͷ2ஈ໨ͷimage͸CircleCI͕build • ઐ༻ͷbranchʹpush͢ΔͱCircleCI͕build & push 44

    VASILY Inc. 2016

45. #7 Docker build • alpineԽ͔ͯ͠Β͸Dockerfile͓͡͞ΜԽ͕ݦஶʹਐΜͰΔؾ΋͠ ·͢.. 45 VASILY Inc. 2016

46. ো֐ฤ • ओʹӡ༻ϛεʹΑΔো֐͕ൃੜ͠·ͨ͠ 46 VASILY Inc. 2016

47. #1 quay.ioύεϫʔυมߋʹ൐͏ deployͷࣦഊ • imageΛquay.ioͰϗετ • ϚελͷύεϫʔυΛͦͷ··deploy࣌ʹ΋࢖͍ͬͯͨ • .dockercfgʹϚελΞΧ΢ϯτͷ΋ͷΛೖΕ͍ͯͨ 47

    VASILY Inc. 2016

48. #1 quay.ioύεϫʔυมߋʹ൐͏ deployͷࣦഊ • μϝ! ઈର!! 48 VASILY Inc. 2016

49. #1 quay.ioύεϫʔυมߋʹ൐͏ deployͷࣦഊ • ϚελύεϫʔυͷมߋΛߦͬͨ • ͦͷޙͷσϓϩΠʹࣦഊ • docker pull͕Ͱ͖ͳ͘ͳͬͨ..

    49 VASILY Inc. 2016

50. #1 quay.ioύεϫʔυมߋʹ൐͏ deployͷࣦഊ • auto-scalingͷλΠϛϯάͱ΋૬·ͬͯΞΧ΢ϯτ͕ϩοΫ • େྔͷdocker pullͷࣦഊ͕ൃੜ • ϚελͷΞΧ΢ϯτͰdocker

    pull ͕ϩʔΧϧ؀ڥͰ΋Ͱ͖ͳ͘ ͳͬͨ 50 VASILY Inc. 2016

51. #1 quay.ioύεϫʔυมߋʹ൐͏ deployͷࣦഊ • ରԠ • σϓϩΠ༻ͷϩϘοτΞΧ΢ϯτΛ࡞੒(quay.ioͷػೳ) • ͦͷޙɺো֐͸ى͖͍ͯͳ͍ 51

    VASILY Inc. 2016

52. #1 quay.ioύεϫʔυมߋʹ൐͏ deployͷࣦഊ • quay.io͕SPOFʹͳ͍ͬͯΔͷͰɺηΧϯμϦʹ͍ͭͯ͸ݕ౼த • ECR͸ϦϦʔε౰ॳʹ؆୯ʹςετͨ͠ͷΈ • ஌ݟ͋Ε͹ڭ͑ͯԼ͍͞! 52

    VASILY Inc. 2016

53. #2 mackerel-agentͷΞϥʔτ • mackerelͰ؂ࢹ • auto-scalingͰΧδϡΞϧʹinstance͕ࢭ·Δ • unreachableͷΞϥʔτ͕ඈͿ • unreachable

    Ͱ͸͋Δ͚Ͳɺো֐Ͱ͸ͳ͍.. 53 VASILY Inc. 2016

54. #2 mackerel-agentͷΞϥʔτ • auto-scalingͳ؀ڥ༻ͷઃఆΛ࡞ͬͯ΋Βͬͨ 54 VASILY Inc. 2016

55. #2 mackerel-agentͷΞ ϥʔτ ಈ͔ͳ͔ͬͨ.. 55 VASILY Inc. 2016

56. #2 mackerel-agentͷΞϥʔτ • Ұ୴ɺର৅ͷroleͷunreachableͷΞϥʔτઃఆΛ੾ͬͨ.. • ؂ࢹͷҙຯ͕ແ͍.. 56 VASILY Inc. 2016

57. #2 mackerel-agentͷΞϥʔτ • base platformΛมߋͨ͠Β͏·͍͘͘Α͏ʹͳͬͨ • Amazon Linux 2015.09 v2.0.8

    docker 1.9.1 • ؀ڥʹΑͬͯtrap͕ಈ͍ͨΓಈ͔ͳ͔ͬͨΓͯ͠Δͱͷ͜ͱ.. • EBͰͷঢ়گ͸Qiitaʹॻ͍ͯ·͢ -> http://qiita.com/tknzk/items/a6b1a9003ff26683b88a 57 VASILY Inc. 2016

58. #3 كʹσϓϩΠʹࣦഊ͢Δ • deployϑϩʔ • CircleCIͰmasterͷbranchͷCI͕·ΘΔͱdeploy hook͕͸͠Δ • eb deploy

    ad-server-prod 58 VASILY Inc. 2016

59. #3 كʹσϓϩΠʹࣦഊ͢Δ • deploy͕ࣦഊ͢Δ͕࣌͋Δ • mackerel-agentͷcontainerͷstopʹࣦഊ͢Δ͜ͱ͕͋Δ [ec2-user@ip-xx-xx-xxx ~]$ sudo docker

    exec CONTAINER_ID ps axu nsenter: Unable to fork: Cannot allocate memory 59 VASILY Inc. 2016

60. #3 كʹσϓϩΠʹࣦഊ͢Δ • ෮چͰ͖ͳ͍ͷͰinstance͝ͱstopͯ͠ରԠ • mackerel-agentͷதͷ໰୊ͷΑ͏ͳؾ͕͍ͯ͠Δ͕ɺৄࡉ͸೺ Ѳग़དྷ͍ͯͳ͍ 60 VASILY Inc.

    2016

61. #3 كʹσϓϩΠʹࣦഊ͢Δ • deploy͕ࣦഊͨ͠Β • ΧδϡΞϧʹinstanceΛstopͯ͠ೖΕସ͑ • ࠶deploy 61 VASILY

    Inc. 2016

62. #3 كʹσϓϩΠʹࣦഊ͢Δ • auto-scalingͷhealth-checkͷର৅ΛELBʹ͢Δ • deployʹࣦഊͯ͠΋ELB͔Β੾Γ཭͞Εͨঢ়ଶ • ৑௕ߏ੒ʹͳ͍ͬͯΕ͹໰୊ͳ͘ӡ༻Ͱ͖Δ 62 VASILY

    Inc. 2016

63. #4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ ౓͖Γ) • deploy͸CircleCI͔Β eb deploy ad-server-prod ͰσϓϩΠ

    • applicationΛzipͨ͠ϑΝΠϧΛs3ʹput • ֤instance͕pullͯ͘͠Δ 63 VASILY Inc. 2016

64. #4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ ౓͖Γ) • ಛఆͷawsebcli͸ s3ʹpush͢ΔϑΝΠϧ໊͕gitͷcommithash 4 ܻͰݻఆ •

    ຊ൪ӡ༻։࢝લʹੜ੒ͨ͠ϑΝΠϧ໊ͱconflictͯ͠deploy͕ί έͨ • ͜ͷ࣌͸ίέ͚͕ͨͩͩͬͨ੒ޭ͢ΔͱσάϨ͢Δ͜ͱʹͳͬ ͨ 64 VASILY Inc. 2016

65. #4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ ౓͖Γ) • ݱ࣌఺Ͱ͸ɺTimeStamp͕suffixͱͯͭ͘͠Α͏ʹͳͬͨ • versino 3.6.2 Ҏ߱

    65 VASILY Inc. 2016

66. #4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ ౓͖Γ) • ఆظతʹapplication versionsͷͱ͜Ζʹ͋ΔϑΝΠϧΛ૟আ • σϑΥϧτ500·Ͱ͸อ࣋Ͱ͖Δ •

    ݹ͍ͷΛ࢒͍ͯͯ͠΋͋·Γҙຯͳ͍ͷͰফͯ͠Δ 66 VASILY Inc. 2016

67. Conclusion • 1೥ؒ EB w/multi-container docker Ͱӡ༻ͯ͠ى͖ͨ͜ͱΛ͓࿩ ͠·ͨ͠ • ӡ༻ϛεʹΑΔো֐΋͋Γ·͕ͨ͠ɺEB΍dockerʹىҼ͢Δো

    ֐͸ͳ͠ • alpine linuxΛproductionͰӡ༻தͰ͢ • iQONຊମͷAPI΋͍ͣΕ͸... 67 VASILY Inc. 2016

68. We are hiring! • info@vasily.jp 68 VASILY Inc. 2016

69. end ͋Γ͕ͱ͏͍͟͝·ͨ͠ 69 VASILY Inc. 2016