$30 off During Our Annual Pro Sale. View Details »

eb_with_multi_container_docker

 eb_with_multi_container_docker

# JAWS-UGおコンテナ支部 #5

- EB with multi-container Docker をproductionで1年間運用して起きたこと

Takumi KANZAKI

June 27, 2016
Tweet

More Decks by Takumi KANZAKI

Other Decks in Technology

Transcript

  1. JAWS-UG͓ίϯςφࢧ
    ෦ #5
    EB with multi-container Docker Λ
    productionͰ1೥ؒӡ༻ͯ͠ى͖ͨ͜
    ͱ
    1 VASILY Inc. 2016

    View Slide

  2. Me
    • Takumi Kanzaki (@tknzk)
    • VASILY Inc. Backend Engineer
    • 2015.04~
    • iQONAD/ࣾ಺༻੥ٻॻπʔϧ Λ୲౰ ($∀$)
    • ࠷ۙiQON API ΋୲౰
    2 VASILY Inc. 2016

    View Slide

  3. 3 VASILY Inc. 2016

    View Slide

  4. iQONAD
    • iQON͓Αͼ֎෦ΞϓϦʹ޿ࠂ഑৴Λߦ͍ͬͯΔADωοτϫʔΫ
    • 2014.10~
    • ॳظ࣮૷~ϩʔϯν
    • 2weeks
    4 VASILY Inc. 2016

    View Slide

  5. iQONAD
    • software, middleware
    • Ruby, Rails, sinatra
    • MySQL, memcached, Redis
    • infra
    • EC2, RDS
    • docker, ElasticBeanstalk (2015.06~)
    • auto-scaling
    5 VASILY Inc. 2016

    View Slide

  6. EB w/ multi-container
    docker
    • 2015/06~
    • 1೥ؒproduction؀ڥͰӡ༻ͯ͠ى͖ͨ͜ͱΛ͝঺հ͠·͢
    6 VASILY Inc. 2016

    View Slide

  7. Container ߏ੒
    • 4ͭͷcontainerͰՔಇ
    1. ຊମͷΞϓϦέʔγϣϯ͕͸͍ͬͨContainer (ad-server)
    • ruby,supervisord
    2. httpͷϦΫΤετΛड͚෇͚ΔͨΊͷnginx
    3. logίϨΫλͱͯ͠ͷtd-agent
    4. ؂ࢹ༻ͷmackerel-agent
    7 VASILY Inc. 2016

    View Slide

  8. ։ൃ/ӡ༻ฤ
    8 VASILY Inc. 2016

    View Slide

  9. #1 middlewareͷߋ৽
    • ruby 2.0 -> 2.2.3
    • ϩʔϯν·Ͱ2िؒͰ։ൃΛߦͬͨ͜ͱ͔Βطଘͷ։ൃӡ༻࣮੷
    ͷ͋Δruby 2.0.0 Ͱ։ൃ/ӡ༻͞Ε͍ͯ·ͨ͠
    • 2015/11ʹdockerͷڧΈΛ׆͔ͯ͠ɺruby2.2ܥʹҠߦ
    • ruby2.2
    • GGͷվળͷԸܙ
    9 VASILY Inc. 2016

    View Slide

  10. #1 middlewareͷߋ৽
    • ruby 2.0 -> 2.2.3
    10 VASILY Inc. 2016

    View Slide

  11. #1 middlewareͷߋ৽
    • ruby 2.0 -> 2.2.3 -> 2.3.1
    • ࠓ೥ͷ͸͡Ίʹ ruby 2.3.1΁Ҡߦ
    • ଞͷmiddleware΋ΧδϡΞϧʹ࠷৽΁
    • nginx: 1.11.1
    • mackerel-agent: 0.31.2
    • td-agent: v2.3.1 (fluentd:v0.12.20)
    11 VASILY Inc. 2016

    View Slide

  12. #1 middlewareͷߋ৽
    • ruby 2.4.0΋ਵ࣌ςετΛ͓͜ͳ͍ͬͯ·͢
    • ruby 2.4.0-preview1
    12 VASILY Inc. 2016

    View Slide

  13. #1 middlewareͷߋ৽
    • docker ͷར఺
    • ΧδϡΞϧʹimageΛೖΕସ͑ΒΕΔ
    13 VASILY Inc. 2016

    View Slide

  14. #2 docker imageංେԽ
    • ͱ͋ΔλΠϛϯάͷdocker image size
    • ad_server: 924.6MB
    • nginx: 134.1MB
    • td-agent: 448.4MB
    • mackerel-agent: 423.1MB
    14 VASILY Inc. 2016

    View Slide

  15. #2 docker imageංେԽ
    ͑ͬʂࢲͷdocker imageଠΓա͗..!
    15 VASILY Inc. 2016

    View Slide

  16. #2-2 docker image࡟ݮ
    • ํ਑
    1. Ͱ͖Δ͚ͩܰྔͳOSΛϕʔεʹ͢Δ
    2. ෆඞཁͳύοέʔδΛΠϯετʔϧ͠ͳ͍
    3. build͢Δͱ͖ʹ͚ͩඞཁͳύοέʔδΛదٓ࡟আ͢Δ
    16 VASILY Inc. 2016

    View Slide

  17. #2-2 docker image࡟ݮ
    nginx
    • ެࣜͰalpine linuxΛϕʔεʹͨ͠΋ͷ
    ͕͋Δ
    • ࠾༻!
    17 VASILY Inc. 2016

    View Slide

  18. #2-2 docker image࡟ݮ
    ad-server
    • ΋ͱ΋ͱ CentOS6Λϕʔεʹbuild
    • 2ஈߏ੒
    • ruby, supervisord, mysql Λ૊ΈࠐΜͩimage
    • ↑ʹad-serverͱͯ͠ඞཁͳGemΛ૊ΈࠐΉ
    18 VASILY Inc. 2016

    View Slide

  19. #2-2 docker image࡟ݮ
    ad-server
    • alpine linuxΛϕʔεʹͯ͠build
    • ॳظ
    • build͢Δ͚ͩͯ͠૟আΛ͍ͯ͠ͳ͍ঢ়ଶ
    19 VASILY Inc. 2016

    View Slide

  20. #2-2 docker image࡟ݮ
    ad-server
    20 VASILY Inc. 2016

    View Slide

  21. #2-2 docker image࡟ݮ
    ad-server
    • ਫ਼ࠪ1
    • mysql (mariadb-dev) ͕ͱͯ΋େ͖͍...
    • ෆཁͳόΠφϦΛ࡟আ
    rm /usr/lib/libmysqld* && \
    rm /usr/bin/mysql*
    21 VASILY Inc. 2016

    View Slide

  22. #2-2 docker image࡟ݮ
    ad-server
    • ਫ਼ࠪ2
    • buildʹඞཁͳύοέʔδΛ੔ཧ
    • native-extensionͳGemΠϯετʔϧʹඞཁͳ෺ΛtemporaryͰ
    install/uninstall
    • build-base (gccͳͲ)
    22 VASILY Inc. 2016

    View Slide

  23. #2 docker imageංେԽ
    • ࠷৽ͷ docker image size
    • ad_server: 924.6MB -> 342.7MB
    • nginx: 134.1MB -> 59.63MB
    • td-agent: 448.4MB -> 430.8MB
    • mackerel-agent: 423.1MB -> 357.5MB
    23 VASILY Inc. 2016

    View Slide

  24. #2 docker imageංେԽ
    • ࠷େ࣌ͷ6ׂ͙Β͍·Ͱ࡟ݮ
    • blogॻ͖·ͨ͠
    http://tech.vasily.jp/
    docker_image_slim_in_alpinelinux
    24 VASILY Inc. 2016

    View Slide

  25. #3 base platform
    • EB w/multi-container docker
    • Amazon Linux ͱ dockerͷ૊Έ߹ΘͤͰෳ਺
    25 VASILY Inc. 2016

    View Slide

  26. #3 base platform
    • EB w/multi-container docker
    • Amazon Linux 2015.03 v1.4.0 docker 1.6.0
    • Amazon Linux 2015.03 v1.4.1 docker 1.6.0
    • Amazon Linux 2015.03 v1.4.6 docker 1.6.2
    • Amazon Linux 2015.09 v2.0.8 docker 1.9.1
    • Amazon Linux 2016.03 v2.1.0 docker 1.9.1
    • Amazon Linux 2016.03 v2.1.1 docker 1.9.1
    26 VASILY Inc. 2016

    View Slide

  27. #3 base platform
    • ͍·͸গ͠ݹΊversionΛར༻
    • Amazon Linux 2015.09 v2.0.8 docker 1.9.1
    • Amazon Linux 2016.03͸mackerel-agent͕ىಈͰ͖ͳ͍໰୊
    • docker͔ͩΒؔ܎ͳ͍ͱࢥ͍ͬͯͨΒىಈͰ͖ͳ͔ͬͨ..
    27 VASILY Inc. 2016

    View Slide

  28. #3 base platform
    • hostͷdocker ͕গ͠ݹ͍ͷͰɺclientଆ͕৽͗͢͠ΔͱىಈͰ͖
    ͳ͍໰୊΋..
    • docker 1.9.1
    • EBʹґଘ͠ͳ͍ܗ΋ߟ͍͖͍͑ͯͨ
    28 VASILY Inc. 2016

    View Slide

  29. #3 base platform
    • ࠓݟͨΒ, Amazon Linux 2016.03 docker 1.11.1 ͕Ͱͯͨ
    • ໌೔ࢼ͍ͨ͠ͱ͓΋͍·͢
    29 VASILY Inc. 2016

    View Slide

  30. #3 Envoirnment clone
    • production/stagingͷ؀ڥΛ·ΔͬͱίϐʔͰ͖Δ
    • base platformΛมߋ͢Δࡍͷςετ
    30 VASILY Inc. 2016

    View Slide

  31. #3 Envoirnment clone
    31 VASILY Inc. 2016

    View Slide

  32. #3 Envoirnment clone
    32 VASILY Inc. 2016

    View Slide

  33. #3 Envoirnment clone
    • ਺ΫϦοΫͰίϐʔͰ͖ΔͷͰͱͯ΋ศར!
    33 VASILY Inc. 2016

    View Slide

  34. #4 auto-scaling
    • ௨ৗͷΞΫηε૿Ճ/ݮগʹ͸Auto-ScalingͰରԠ
    • EBͷweb consoleͷScaling configurationͰઃఆ
    • Sum RequestCountϕʔε
    34 VASILY Inc. 2016

    View Slide

  35. #5 Time Scheduled-based auto-
    scaling
    • ADωοτϫʔΫ
    • iQONҎ֎ͷΞϓϦʹ΋഑৴
    35 VASILY Inc. 2016

    View Slide

  36. #5 Time Scheduled-based auto-
    scaling
    • ͱ͋ΔΞϓϦ͕pushΛ͏ͭ
    • spike͢Δ
    • ௨ৗͷauto-scalingͰ͸௥͍͔ͭͳ͍
    • pushʹΑΔspike͕ऩ·ͬͨ͋ͱʹىಈ׬ྃ..
    • ҙຯͳ͍
    36 VASILY Inc. 2016

    View Slide

  37. #5 Time Scheduled-based auto-
    scaling
    • time scheduled base auto-scalingΛઃఆ
    • web console͔Β͸ઃఆͰ͖ͳ͍ͷͰaws cliͰߦ͏
    37 VASILY Inc. 2016

    View Slide

  38. #5 Time Scheduled-based auto-
    scaling
    • time scheduled base auto-scalingΛઃఆ
    • scale out
    aws autoscaling put-scheduled-update-group-action \
    --auto-scaling-group-name awseb-e-XXXXXX \
    --scheduled-action-name ScaleOut \
    --recurrence "00 10 * * *" \
    --desired-capacity 10 \
    --min-size 10 \
    --max-size 12
    38 VASILY Inc. 2016

    View Slide

  39. #5 Time Scheduled-based auto-
    scaling
    • time scheduled base auto-scalingΛઃఆ
    • scale in
    aws autoscaling put-scheduled-update-group-action \
    --auto-scaling-group-name awseb-e-XXXXXX \
    --scheduled-action-name ScaleIn \
    --recurrence "30 16 * * *" \
    --desired-capacity 6 \
    --min-size 6 \
    --max-size 8
    39 VASILY Inc. 2016

    View Slide

  40. #5 Time Scheduled-based auto-
    scaling
    • ઌํ΋ӡ༻͠ͳ͕ΒͩͬͨͷͰɺ͏·͍͘ͳ͔͍͜ͱ΋ଟ͔ͬ
    ͨ
    • ݱࡏ͸pushͷӡ༻͕ͱ·ͬͨ
    • time scheduled base auto scaling͸ઃఆͯ͠ͳ͍
    40 VASILY Inc. 2016

    View Slide

  41. #6 ։ൃ؀ڥ
    • docker-compose Ͱproductionͱಉ౳ͷ؀ڥΛىಈͰ͖ΔΑ͏ʹ
    ͍ͯ͠Δ
    • mysql,redis,memcache͸ࣾ಺։ൃαʔόΛࢀর
    41 VASILY Inc. 2016

    View Slide

  42. #6 ։ൃ؀ڥ
    • ීஈ͸MBPͷruby,nginxͰ։ൃΛ͍ͯ͠Δ
    • ։ൃ࣌͸جຊతʹҰਓͰΈͯΔͷͰContainerͷྑ͍ॴ͕ग़ͤ
    ͳ͍..
    • docker-compose run app rake ͰCIͷΈ࣮ࢪ
    42 VASILY Inc. 2016

    View Slide

  43. #7 Docker build
    • Dockerfile͸ผrepositoryͰGithubͰ؅ཧ
    • docker resitory(quay.io)ͷbuild hook͸ະ࠾༻
    • खݩͷMBP(docker-machine) Ͱdocker build & pushͰӡ༻
    • docker build -t quay.io/vasilyjp/ruby:2.3.1-alpine .
    • docker push quay.io/vasilyjp/ruby:2.3.1-alpine
    43 VASILY Inc. 2016

    View Slide

  44. #7 Docker build
    • ad-serverͷ2ஈ໨ͷimage͸CircleCI͕build
    • ઐ༻ͷbranchʹpush͢ΔͱCircleCI͕build & push
    44 VASILY Inc. 2016

    View Slide

  45. #7 Docker build
    • alpineԽ͔ͯ͠Β͸Dockerfile͓͡͞ΜԽ͕ݦஶʹਐΜͰΔؾ΋͠
    ·͢..
    45 VASILY Inc. 2016

    View Slide

  46. ো֐ฤ
    • ओʹӡ༻ϛεʹΑΔো֐͕ൃੜ͠·ͨ͠
    46 VASILY Inc. 2016

    View Slide

  47. #1 quay.ioύεϫʔυมߋʹ൐͏
    deployͷࣦഊ
    • imageΛquay.ioͰϗετ
    • ϚελͷύεϫʔυΛͦͷ··deploy࣌ʹ΋࢖͍ͬͯͨ
    • .dockercfgʹϚελΞΧ΢ϯτͷ΋ͷΛೖΕ͍ͯͨ
    47 VASILY Inc. 2016

    View Slide

  48. #1 quay.ioύεϫʔυมߋʹ൐͏
    deployͷࣦഊ
    • μϝ! ઈର!!
    48 VASILY Inc. 2016

    View Slide

  49. #1 quay.ioύεϫʔυมߋʹ൐͏
    deployͷࣦഊ
    • ϚελύεϫʔυͷมߋΛߦͬͨ
    • ͦͷޙͷσϓϩΠʹࣦഊ
    • docker pull͕Ͱ͖ͳ͘ͳͬͨ..
    49 VASILY Inc. 2016

    View Slide

  50. #1 quay.ioύεϫʔυมߋʹ൐͏
    deployͷࣦഊ
    • auto-scalingͷλΠϛϯάͱ΋૬·ͬͯΞΧ΢ϯτ͕ϩοΫ
    • େྔͷdocker pullͷࣦഊ͕ൃੜ
    • ϚελͷΞΧ΢ϯτͰdocker pull ͕ϩʔΧϧ؀ڥͰ΋Ͱ͖ͳ͘
    ͳͬͨ
    50 VASILY Inc. 2016

    View Slide

  51. #1 quay.ioύεϫʔυมߋʹ൐͏
    deployͷࣦഊ
    • ରԠ
    • σϓϩΠ༻ͷϩϘοτΞΧ΢ϯτΛ࡞੒(quay.ioͷػೳ)
    • ͦͷޙɺো֐͸ى͖͍ͯͳ͍
    51 VASILY Inc. 2016

    View Slide

  52. #1 quay.ioύεϫʔυมߋʹ൐͏
    deployͷࣦഊ
    • quay.io͕SPOFʹͳ͍ͬͯΔͷͰɺηΧϯμϦʹ͍ͭͯ͸ݕ౼த
    • ECR͸ϦϦʔε౰ॳʹ؆୯ʹςετͨ͠ͷΈ
    • ஌ݟ͋Ε͹ڭ͑ͯԼ͍͞!
    52 VASILY Inc. 2016

    View Slide

  53. #2 mackerel-agentͷΞϥʔτ
    • mackerelͰ؂ࢹ
    • auto-scalingͰΧδϡΞϧʹinstance͕ࢭ·Δ
    • unreachableͷΞϥʔτ͕ඈͿ
    • unreachable Ͱ͸͋Δ͚Ͳɺো֐Ͱ͸ͳ͍..
    53 VASILY Inc. 2016

    View Slide

  54. #2 mackerel-agentͷΞϥʔτ
    • auto-scalingͳ؀ڥ༻ͷઃఆΛ࡞ͬͯ΋Βͬͨ
    54 VASILY Inc. 2016

    View Slide

  55. #2 mackerel-agentͷΞ
    ϥʔτ
    ಈ͔ͳ͔ͬͨ..
    55 VASILY Inc. 2016

    View Slide

  56. #2 mackerel-agentͷΞϥʔτ
    • Ұ୴ɺର৅ͷroleͷunreachableͷΞϥʔτઃఆΛ੾ͬͨ..
    • ؂ࢹͷҙຯ͕ແ͍..
    56 VASILY Inc. 2016

    View Slide

  57. #2 mackerel-agentͷΞϥʔτ
    • base platformΛมߋͨ͠Β͏·͍͘͘Α͏ʹͳͬͨ
    • Amazon Linux 2015.09 v2.0.8 docker 1.9.1
    • ؀ڥʹΑͬͯtrap͕ಈ͍ͨΓಈ͔ͳ͔ͬͨΓͯ͠Δͱͷ͜ͱ..
    • EBͰͷঢ়گ͸Qiitaʹॻ͍ͯ·͢
    -> http://qiita.com/tknzk/items/a6b1a9003ff26683b88a
    57 VASILY Inc. 2016

    View Slide

  58. #3 كʹσϓϩΠʹࣦഊ͢Δ
    • deployϑϩʔ
    • CircleCIͰmasterͷbranchͷCI͕·ΘΔͱdeploy hook͕͸͠Δ
    • eb deploy ad-server-prod
    58 VASILY Inc. 2016

    View Slide

  59. #3 كʹσϓϩΠʹࣦഊ͢Δ
    • deploy͕ࣦഊ͢Δ͕࣌͋Δ
    • mackerel-agentͷcontainerͷstopʹࣦഊ͢Δ͜ͱ͕͋Δ
    [ec2-user@ip-xx-xx-xxx ~]$ sudo docker exec CONTAINER_ID ps axu
    nsenter: Unable to fork: Cannot allocate memory
    59 VASILY Inc. 2016

    View Slide

  60. #3 كʹσϓϩΠʹࣦഊ͢Δ
    • ෮چͰ͖ͳ͍ͷͰinstance͝ͱstopͯ͠ରԠ
    • mackerel-agentͷதͷ໰୊ͷΑ͏ͳؾ͕͍ͯ͠Δ͕ɺৄࡉ͸೺
    Ѳग़དྷ͍ͯͳ͍
    60 VASILY Inc. 2016

    View Slide

  61. #3 كʹσϓϩΠʹࣦഊ͢Δ
    • deploy͕ࣦഊͨ͠Β
    • ΧδϡΞϧʹinstanceΛstopͯ͠ೖΕସ͑
    • ࠶deploy
    61 VASILY Inc. 2016

    View Slide

  62. #3 كʹσϓϩΠʹࣦഊ͢Δ
    • auto-scalingͷhealth-checkͷର৅ΛELBʹ͢Δ
    • deployʹࣦഊͯ͠΋ELB͔Β੾Γ཭͞Εͨঢ়ଶ
    • ৑௕ߏ੒ʹͳ͍ͬͯΕ͹໰୊ͳ͘ӡ༻Ͱ͖Δ
    62 VASILY Inc. 2016

    View Slide

  63. #4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ
    ౓͖Γ)
    • deploy͸CircleCI͔Β eb deploy ad-server-prod ͰσϓϩΠ
    • applicationΛzipͨ͠ϑΝΠϧΛs3ʹput
    • ֤instance͕pullͯ͘͠Δ
    63 VASILY Inc. 2016

    View Slide

  64. #4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ
    ౓͖Γ)
    • ಛఆͷawsebcli͸ s3ʹpush͢ΔϑΝΠϧ໊͕gitͷcommithash 4
    ܻͰݻఆ
    • ຊ൪ӡ༻։࢝લʹੜ੒ͨ͠ϑΝΠϧ໊ͱconflictͯ͠deploy͕ί
    έͨ
    • ͜ͷ࣌͸ίέ͚͕ͨͩͩͬͨ੒ޭ͢ΔͱσάϨ͢Δ͜ͱʹͳͬ
    ͨ
    64 VASILY Inc. 2016

    View Slide

  65. #4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ
    ౓͖Γ)
    • ݱ࣌఺Ͱ͸ɺTimeStamp͕suffixͱͯͭ͘͠Α͏ʹͳͬͨ
    • versino 3.6.2 Ҏ߱
    65 VASILY Inc. 2016

    View Slide

  66. #4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ
    ౓͖Γ)
    • ఆظతʹapplication versionsͷͱ͜Ζʹ͋ΔϑΝΠϧΛ૟আ
    • σϑΥϧτ500·Ͱ͸อ࣋Ͱ͖Δ
    • ݹ͍ͷΛ࢒͍ͯͯ͠΋͋·Γҙຯͳ͍ͷͰফͯ͠Δ
    66 VASILY Inc. 2016

    View Slide

  67. Conclusion
    • 1೥ؒ EB w/multi-container docker Ͱӡ༻ͯ͠ى͖ͨ͜ͱΛ͓࿩
    ͠·ͨ͠
    • ӡ༻ϛεʹΑΔো֐΋͋Γ·͕ͨ͠ɺEB΍dockerʹىҼ͢Δো
    ֐͸ͳ͠
    • alpine linuxΛproductionͰӡ༻தͰ͢
    • iQONຊମͷAPI΋͍ͣΕ͸...
    67 VASILY Inc. 2016

    View Slide

  68. We are hiring!
    [email protected]
    68 VASILY Inc. 2016

    View Slide

  69. end
    ͋Γ͕ͱ͏͍͟͝·ͨ͠
    69 VASILY Inc. 2016

    View Slide