eb_with_multi_container_docker

# JAWS-UGおコンテナ支部 #5

- EB with multi-container Docker をproductionで1年間運用して起きたこと

Transcript

1. JAWS-UG͓ίϯςφࢧ
   ෦ #5
   EB with multi-container Docker Λ
   productionͰ1೥ؒӡ༻ͯ͠ى͖ͨ͜
   ͱ
   1 VASILY Inc. 2016
   

   View Slide

2. Me
   • Takumi Kanzaki (@tknzk)
   • VASILY Inc. Backend Engineer
   • 2015.04~
   • iQONAD/ࣾ಺༻੥ٻॻπʔϧ Λ୲౰ ($∀$)
   • ࠷ۙiQON API ΋୲౰
   2 VASILY Inc. 2016
   

   View Slide

3. 3 VASILY Inc. 2016
   

   View Slide

4. iQONAD
   • iQON͓Αͼ֎෦ΞϓϦʹ޿ࠂ഑৴Λߦ͍ͬͯΔADωοτϫʔΫ
   • 2014.10~
   • ॳظ࣮૷~ϩʔϯν
   • 2weeks
   4 VASILY Inc. 2016
   

   View Slide

5. iQONAD
   • software, middleware
   • Ruby, Rails, sinatra
   • MySQL, memcached, Redis
   • infra
   • EC2, RDS
   • docker, ElasticBeanstalk (2015.06~)
   • auto-scaling
   5 VASILY Inc. 2016
   

   View Slide

6. EB w/ multi-container
   docker
   • 2015/06~
   • 1೥ؒproduction؀ڥͰӡ༻ͯ͠ى͖ͨ͜ͱΛ͝঺հ͠·͢
   6 VASILY Inc. 2016
   

   View Slide

7. Container ߏ੒
   • 4ͭͷcontainerͰՔಇ
   1. ຊମͷΞϓϦέʔγϣϯ͕͸͍ͬͨContainer (ad-server)
   • ruby,supervisord
   2. httpͷϦΫΤετΛड͚෇͚ΔͨΊͷnginx
   3. logίϨΫλͱͯ͠ͷtd-agent
   4. ؂ࢹ༻ͷmackerel-agent
   7 VASILY Inc. 2016
   

   View Slide

8. ։ൃ/ӡ༻ฤ
   8 VASILY Inc. 2016
   

   View Slide

9. #1 middlewareͷߋ৽
   • ruby 2.0 -> 2.2.3
   • ϩʔϯν·Ͱ2िؒͰ։ൃΛߦͬͨ͜ͱ͔Βطଘͷ։ൃӡ༻࣮੷
   ͷ͋Δruby 2.0.0 Ͱ։ൃ/ӡ༻͞Ε͍ͯ·ͨ͠
   • 2015/11ʹdockerͷڧΈΛ׆͔ͯ͠ɺruby2.2ܥʹҠߦ
   • ruby2.2
   • GGͷվળͷԸܙ
   9 VASILY Inc. 2016
   

   View Slide

10. #1 middlewareͷߋ৽
    • ruby 2.0 -> 2.2.3
    10 VASILY Inc. 2016
    

    View Slide

11. #1 middlewareͷߋ৽
    • ruby 2.0 -> 2.2.3 -> 2.3.1
    • ࠓ೥ͷ͸͡Ίʹ ruby 2.3.1΁Ҡߦ
    • ଞͷmiddleware΋ΧδϡΞϧʹ࠷৽΁
    • nginx: 1.11.1
    • mackerel-agent: 0.31.2
    • td-agent: v2.3.1 (fluentd:v0.12.20)
    11 VASILY Inc. 2016
    

    View Slide

12. #1 middlewareͷߋ৽
    • ruby 2.4.0΋ਵ࣌ςετΛ͓͜ͳ͍ͬͯ·͢
    • ruby 2.4.0-preview1
    12 VASILY Inc. 2016
    

    View Slide

13. #1 middlewareͷߋ৽
    • docker ͷར఺
    • ΧδϡΞϧʹimageΛೖΕସ͑ΒΕΔ
    13 VASILY Inc. 2016
    

    View Slide

14. #2 docker imageංେԽ
    • ͱ͋ΔλΠϛϯάͷdocker image size
    • ad_server: 924.6MB
    • nginx: 134.1MB
    • td-agent: 448.4MB
    • mackerel-agent: 423.1MB
    14 VASILY Inc. 2016
    

    View Slide

15. #2 docker imageංେԽ
    ͑ͬʂࢲͷdocker imageଠΓա͗..!
    15 VASILY Inc. 2016
    

    View Slide

16. #2-2 docker image࡟ݮ
    • ํ਑
    1. Ͱ͖Δ͚ͩܰྔͳOSΛϕʔεʹ͢Δ
    2. ෆඞཁͳύοέʔδΛΠϯετʔϧ͠ͳ͍
    3. build͢Δͱ͖ʹ͚ͩඞཁͳύοέʔδΛదٓ࡟আ͢Δ
    16 VASILY Inc. 2016
    

    View Slide

17. #2-2 docker image࡟ݮ
    nginx
    • ެࣜͰalpine linuxΛϕʔεʹͨ͠΋ͷ
    ͕͋Δ
    • ࠾༻!
    17 VASILY Inc. 2016
    

    View Slide

18. #2-2 docker image࡟ݮ
    ad-server
    • ΋ͱ΋ͱ CentOS6Λϕʔεʹbuild
    • 2ஈߏ੒
    • ruby, supervisord, mysql Λ૊ΈࠐΜͩimage
    • ↑ʹad-serverͱͯ͠ඞཁͳGemΛ૊ΈࠐΉ
    18 VASILY Inc. 2016
    

    View Slide

19. #2-2 docker image࡟ݮ
    ad-server
    • alpine linuxΛϕʔεʹͯ͠build
    • ॳظ
    • build͢Δ͚ͩͯ͠૟আΛ͍ͯ͠ͳ͍ঢ়ଶ
    19 VASILY Inc. 2016
    

    View Slide

20. #2-2 docker image࡟ݮ
    ad-server
    20 VASILY Inc. 2016
    

    View Slide

21. #2-2 docker image࡟ݮ
    ad-server
    • ਫ਼ࠪ1
    • mysql (mariadb-dev) ͕ͱͯ΋େ͖͍...
    • ෆཁͳόΠφϦΛ࡟আ
    rm /usr/lib/libmysqld* && \
    rm /usr/bin/mysql*
    21 VASILY Inc. 2016
    

    View Slide

22. #2-2 docker image࡟ݮ
    ad-server
    • ਫ਼ࠪ2
    • buildʹඞཁͳύοέʔδΛ੔ཧ
    • native-extensionͳGemΠϯετʔϧʹඞཁͳ෺ΛtemporaryͰ
    install/uninstall
    • build-base (gccͳͲ)
    22 VASILY Inc. 2016
    

    View Slide

23. #2 docker imageංେԽ
    • ࠷৽ͷ docker image size
    • ad_server: 924.6MB -> 342.7MB
    • nginx: 134.1MB -> 59.63MB
    • td-agent: 448.4MB -> 430.8MB
    • mackerel-agent: 423.1MB -> 357.5MB
    23 VASILY Inc. 2016
    

    View Slide

24. #2 docker imageංେԽ
    • ࠷େ࣌ͷ6ׂ͙Β͍·Ͱ࡟ݮ
    • blogॻ͖·ͨ͠
    http://tech.vasily.jp/
    docker_image_slim_in_alpinelinux
    24 VASILY Inc. 2016
    

    View Slide

25. #3 base platform
    • EB w/multi-container docker
    • Amazon Linux ͱ dockerͷ૊Έ߹ΘͤͰෳ਺
    25 VASILY Inc. 2016
    

    View Slide

26. #3 base platform
    • EB w/multi-container docker
    • Amazon Linux 2015.03 v1.4.0 docker 1.6.0
    • Amazon Linux 2015.03 v1.4.1 docker 1.6.0
    • Amazon Linux 2015.03 v1.4.6 docker 1.6.2
    • Amazon Linux 2015.09 v2.0.8 docker 1.9.1
    • Amazon Linux 2016.03 v2.1.0 docker 1.9.1
    • Amazon Linux 2016.03 v2.1.1 docker 1.9.1
    26 VASILY Inc. 2016
    

    View Slide

27. #3 base platform
    • ͍·͸গ͠ݹΊversionΛར༻
    • Amazon Linux 2015.09 v2.0.8 docker 1.9.1
    • Amazon Linux 2016.03͸mackerel-agent͕ىಈͰ͖ͳ͍໰୊
    • docker͔ͩΒؔ܎ͳ͍ͱࢥ͍ͬͯͨΒىಈͰ͖ͳ͔ͬͨ..
    27 VASILY Inc. 2016
    

    View Slide

28. #3 base platform
    • hostͷdocker ͕গ͠ݹ͍ͷͰɺclientଆ͕৽͗͢͠ΔͱىಈͰ͖
    ͳ͍໰୊΋..
    • docker 1.9.1
    • EBʹґଘ͠ͳ͍ܗ΋ߟ͍͖͍͑ͯͨ
    28 VASILY Inc. 2016
    

    View Slide

29. #3 base platform
    • ࠓݟͨΒ, Amazon Linux 2016.03 docker 1.11.1 ͕Ͱͯͨ
    • ໌೔ࢼ͍ͨ͠ͱ͓΋͍·͢
    29 VASILY Inc. 2016
    

    View Slide

30. #3 Envoirnment clone
    • production/stagingͷ؀ڥΛ·ΔͬͱίϐʔͰ͖Δ
    • base platformΛมߋ͢Δࡍͷςετ
    30 VASILY Inc. 2016
    

    View Slide

31. #3 Envoirnment clone
    31 VASILY Inc. 2016
    

    View Slide

32. #3 Envoirnment clone
    32 VASILY Inc. 2016
    

    View Slide

33. #3 Envoirnment clone
    • ਺ΫϦοΫͰίϐʔͰ͖ΔͷͰͱͯ΋ศར!
    33 VASILY Inc. 2016
    

    View Slide

34. #4 auto-scaling
    • ௨ৗͷΞΫηε૿Ճ/ݮগʹ͸Auto-ScalingͰରԠ
    • EBͷweb consoleͷScaling configurationͰઃఆ
    • Sum RequestCountϕʔε
    34 VASILY Inc. 2016
    

    View Slide

35. #5 Time Scheduled-based auto-
    scaling
    • ADωοτϫʔΫ
    • iQONҎ֎ͷΞϓϦʹ΋഑৴
    35 VASILY Inc. 2016
    

    View Slide

36. #5 Time Scheduled-based auto-
    scaling
    • ͱ͋ΔΞϓϦ͕pushΛ͏ͭ
    • spike͢Δ
    • ௨ৗͷauto-scalingͰ͸௥͍͔ͭͳ͍
    • pushʹΑΔspike͕ऩ·ͬͨ͋ͱʹىಈ׬ྃ..
    • ҙຯͳ͍
    36 VASILY Inc. 2016
    

    View Slide

37. #5 Time Scheduled-based auto-
    scaling
    • time scheduled base auto-scalingΛઃఆ
    • web console͔Β͸ઃఆͰ͖ͳ͍ͷͰaws cliͰߦ͏
    37 VASILY Inc. 2016
    

    View Slide

38. #5 Time Scheduled-based auto-
    scaling
    • time scheduled base auto-scalingΛઃఆ
    • scale out
    aws autoscaling put-scheduled-update-group-action \
    --auto-scaling-group-name awseb-e-XXXXXX \
    --scheduled-action-name ScaleOut \
    --recurrence "00 10 * * *" \
    --desired-capacity 10 \
    --min-size 10 \
    --max-size 12
    38 VASILY Inc. 2016
    

    View Slide

39. #5 Time Scheduled-based auto-
    scaling
    • time scheduled base auto-scalingΛઃఆ
    • scale in
    aws autoscaling put-scheduled-update-group-action \
    --auto-scaling-group-name awseb-e-XXXXXX \
    --scheduled-action-name ScaleIn \
    --recurrence "30 16 * * *" \
    --desired-capacity 6 \
    --min-size 6 \
    --max-size 8
    39 VASILY Inc. 2016
    

    View Slide

40. #5 Time Scheduled-based auto-
    scaling
    • ઌํ΋ӡ༻͠ͳ͕ΒͩͬͨͷͰɺ͏·͍͘ͳ͔͍͜ͱ΋ଟ͔ͬ
    ͨ
    • ݱࡏ͸pushͷӡ༻͕ͱ·ͬͨ
    • time scheduled base auto scaling͸ઃఆͯ͠ͳ͍
    40 VASILY Inc. 2016
    

    View Slide

41. #6 ։ൃ؀ڥ
    • docker-compose Ͱproductionͱಉ౳ͷ؀ڥΛىಈͰ͖ΔΑ͏ʹ
    ͍ͯ͠Δ
    • mysql,redis,memcache͸ࣾ಺։ൃαʔόΛࢀর
    41 VASILY Inc. 2016
    

    View Slide

42. #6 ։ൃ؀ڥ
    • ීஈ͸MBPͷruby,nginxͰ։ൃΛ͍ͯ͠Δ
    • ։ൃ࣌͸جຊతʹҰਓͰΈͯΔͷͰContainerͷྑ͍ॴ͕ग़ͤ
    ͳ͍..
    • docker-compose run app rake ͰCIͷΈ࣮ࢪ
    42 VASILY Inc. 2016
    

    View Slide

43. #7 Docker build
    • Dockerfile͸ผrepositoryͰGithubͰ؅ཧ
    • docker resitory(quay.io)ͷbuild hook͸ະ࠾༻
    • खݩͷMBP(docker-machine) Ͱdocker build & pushͰӡ༻
    • docker build -t quay.io/vasilyjp/ruby:2.3.1-alpine .
    • docker push quay.io/vasilyjp/ruby:2.3.1-alpine
    43 VASILY Inc. 2016
    

    View Slide

44. #7 Docker build
    • ad-serverͷ2ஈ໨ͷimage͸CircleCI͕build
    • ઐ༻ͷbranchʹpush͢ΔͱCircleCI͕build & push
    44 VASILY Inc. 2016
    

    View Slide

45. #7 Docker build
    • alpineԽ͔ͯ͠Β͸Dockerfile͓͡͞ΜԽ͕ݦஶʹਐΜͰΔؾ΋͠
    ·͢..
    45 VASILY Inc. 2016
    

    View Slide

46. ো֐ฤ
    • ओʹӡ༻ϛεʹΑΔো֐͕ൃੜ͠·ͨ͠
    46 VASILY Inc. 2016
    

    View Slide

47. #1 quay.ioύεϫʔυมߋʹ൐͏
    deployͷࣦഊ
    • imageΛquay.ioͰϗετ
    • ϚελͷύεϫʔυΛͦͷ··deploy࣌ʹ΋࢖͍ͬͯͨ
    • .dockercfgʹϚελΞΧ΢ϯτͷ΋ͷΛೖΕ͍ͯͨ
    47 VASILY Inc. 2016
    

    View Slide

48. #1 quay.ioύεϫʔυมߋʹ൐͏
    deployͷࣦഊ
    • μϝ! ઈର!!
    48 VASILY Inc. 2016
    

    View Slide

49. #1 quay.ioύεϫʔυมߋʹ൐͏
    deployͷࣦഊ
    • ϚελύεϫʔυͷมߋΛߦͬͨ
    • ͦͷޙͷσϓϩΠʹࣦഊ
    • docker pull͕Ͱ͖ͳ͘ͳͬͨ..
    49 VASILY Inc. 2016
    

    View Slide

50. #1 quay.ioύεϫʔυมߋʹ൐͏
    deployͷࣦഊ
    • auto-scalingͷλΠϛϯάͱ΋૬·ͬͯΞΧ΢ϯτ͕ϩοΫ
    • େྔͷdocker pullͷࣦഊ͕ൃੜ
    • ϚελͷΞΧ΢ϯτͰdocker pull ͕ϩʔΧϧ؀ڥͰ΋Ͱ͖ͳ͘
    ͳͬͨ
    50 VASILY Inc. 2016
    

    View Slide

51. #1 quay.ioύεϫʔυมߋʹ൐͏
    deployͷࣦഊ
    • ରԠ
    • σϓϩΠ༻ͷϩϘοτΞΧ΢ϯτΛ࡞੒(quay.ioͷػೳ)
    • ͦͷޙɺো֐͸ى͖͍ͯͳ͍
    51 VASILY Inc. 2016
    

    View Slide

52. #1 quay.ioύεϫʔυมߋʹ൐͏
    deployͷࣦഊ
    • quay.io͕SPOFʹͳ͍ͬͯΔͷͰɺηΧϯμϦʹ͍ͭͯ͸ݕ౼த
    • ECR͸ϦϦʔε౰ॳʹ؆୯ʹςετͨ͠ͷΈ
    • ஌ݟ͋Ε͹ڭ͑ͯԼ͍͞!
    52 VASILY Inc. 2016
    

    View Slide

53. #2 mackerel-agentͷΞϥʔτ
    • mackerelͰ؂ࢹ
    • auto-scalingͰΧδϡΞϧʹinstance͕ࢭ·Δ
    • unreachableͷΞϥʔτ͕ඈͿ
    • unreachable Ͱ͸͋Δ͚Ͳɺো֐Ͱ͸ͳ͍..
    53 VASILY Inc. 2016
    

    View Slide

54. #2 mackerel-agentͷΞϥʔτ
    • auto-scalingͳ؀ڥ༻ͷઃఆΛ࡞ͬͯ΋Βͬͨ
    54 VASILY Inc. 2016
    

    View Slide

55. #2 mackerel-agentͷΞ
    ϥʔτ
    ಈ͔ͳ͔ͬͨ..
    55 VASILY Inc. 2016
    

    View Slide

56. #2 mackerel-agentͷΞϥʔτ
    • Ұ୴ɺର৅ͷroleͷunreachableͷΞϥʔτઃఆΛ੾ͬͨ..
    • ؂ࢹͷҙຯ͕ແ͍..
    56 VASILY Inc. 2016
    

    View Slide

57. #2 mackerel-agentͷΞϥʔτ
    • base platformΛมߋͨ͠Β͏·͍͘͘Α͏ʹͳͬͨ
    • Amazon Linux 2015.09 v2.0.8 docker 1.9.1
    • ؀ڥʹΑͬͯtrap͕ಈ͍ͨΓಈ͔ͳ͔ͬͨΓͯ͠Δͱͷ͜ͱ..
    • EBͰͷঢ়گ͸Qiitaʹॻ͍ͯ·͢
    -> http://qiita.com/tknzk/items/a6b1a9003ff26683b88a
    57 VASILY Inc. 2016
    

    View Slide

58. #3 كʹσϓϩΠʹࣦഊ͢Δ
    • deployϑϩʔ
    • CircleCIͰmasterͷbranchͷCI͕·ΘΔͱdeploy hook͕͸͠Δ
    • eb deploy ad-server-prod
    58 VASILY Inc. 2016
    

    View Slide

59. #3 كʹσϓϩΠʹࣦഊ͢Δ
    • deploy͕ࣦഊ͢Δ͕࣌͋Δ
    • mackerel-agentͷcontainerͷstopʹࣦഊ͢Δ͜ͱ͕͋Δ
    [ec2-user@ip-xx-xx-xxx ~]$ sudo docker exec CONTAINER_ID ps axu
    nsenter: Unable to fork: Cannot allocate memory
    59 VASILY Inc. 2016
    

    View Slide

60. #3 كʹσϓϩΠʹࣦഊ͢Δ
    • ෮چͰ͖ͳ͍ͷͰinstance͝ͱstopͯ͠ରԠ
    • mackerel-agentͷதͷ໰୊ͷΑ͏ͳؾ͕͍ͯ͠Δ͕ɺৄࡉ͸೺
    Ѳग़དྷ͍ͯͳ͍
    60 VASILY Inc. 2016
    

    View Slide

61. #3 كʹσϓϩΠʹࣦഊ͢Δ
    • deploy͕ࣦഊͨ͠Β
    • ΧδϡΞϧʹinstanceΛstopͯ͠ೖΕସ͑
    • ࠶deploy
    61 VASILY Inc. 2016
    

    View Slide

62. #3 كʹσϓϩΠʹࣦഊ͢Δ
    • auto-scalingͷhealth-checkͷର৅ΛELBʹ͢Δ
    • deployʹࣦഊͯ͠΋ELB͔Β੾Γ཭͞Εͨঢ়ଶ
    • ৑௕ߏ੒ʹͳ͍ͬͯΕ͹໰୊ͳ͘ӡ༻Ͱ͖Δ
    62 VASILY Inc. 2016
    

    View Slide

63. #4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ
    ౓͖Γ)
    • deploy͸CircleCI͔Β eb deploy ad-server-prod ͰσϓϩΠ
    • applicationΛzipͨ͠ϑΝΠϧΛs3ʹput
    • ֤instance͕pullͯ͘͠Δ
    63 VASILY Inc. 2016
    

    View Slide

64. #4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ
    ౓͖Γ)
    • ಛఆͷawsebcli͸ s3ʹpush͢ΔϑΝΠϧ໊͕gitͷcommithash 4
    ܻͰݻఆ
    • ຊ൪ӡ༻։࢝લʹੜ੒ͨ͠ϑΝΠϧ໊ͱconflictͯ͠deploy͕ί
    έͨ
    • ͜ͷ࣌͸ίέ͚͕ͨͩͩͬͨ੒ޭ͢ΔͱσάϨ͢Δ͜ͱʹͳͬ
    ͨ
    64 VASILY Inc. 2016
    

    View Slide

65. #4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ
    ౓͖Γ)
    • ݱ࣌఺Ͱ͸ɺTimeStamp͕suffixͱͯͭ͘͠Α͏ʹͳͬͨ
    • versino 3.6.2 Ҏ߱
    65 VASILY Inc. 2016
    

    View Slide

66. #4 awsebcli ʹΑΔσϓϩΠʹࣦഊ(Ұ
    ౓͖Γ)
    • ఆظతʹapplication versionsͷͱ͜Ζʹ͋ΔϑΝΠϧΛ૟আ
    • σϑΥϧτ500·Ͱ͸อ࣋Ͱ͖Δ
    • ݹ͍ͷΛ࢒͍ͯͯ͠΋͋·Γҙຯͳ͍ͷͰফͯ͠Δ
    66 VASILY Inc. 2016
    

    View Slide

67. Conclusion
    • 1೥ؒ EB w/multi-container docker Ͱӡ༻ͯ͠ى͖ͨ͜ͱΛ͓࿩
    ͠·ͨ͠
    • ӡ༻ϛεʹΑΔো֐΋͋Γ·͕ͨ͠ɺEB΍dockerʹىҼ͢Δো
    ֐͸ͳ͠
    • alpine linuxΛproductionͰӡ༻தͰ͢
    • iQONຊମͷAPI΋͍ͣΕ͸...
    67 VASILY Inc. 2016
    

    View Slide

68. We are hiring!
    • info@vasily.jp
    68 VASILY Inc. 2016
    

    View Slide

69. end
    ͋Γ͕ͱ͏͍͟͝·ͨ͠
    69 VASILY Inc. 2016
    

    View Slide