Replace a batch application to ECS

2016/09/09 M3 tech meetup! #3 #m3sushi
http://m3-engineer.connpass.com/event/36062/

@yukiyan_w
https://twitter.com/yukiyan_w
http://yukiyan.hatenablog.jp/

Transcript

1. όονΞϓϦͷӡ༻ج൫Λ&$4΁Ҡߦ͢Δ࿩ 2016/09/09 M3 tech meetup! #3 #m3sushi @yukiyan_w
2. Who? ʙ
   4*FS
   ʙ
   
   GFFEGPSDF
   *OD
   "QQMJDBUJPO
   &OHJOFFS
   όονɾ$47ɾ3VCZΛओʹ৮Δɻ Ώ͖΍Μ
   @yukiyan_w http://yukiyan.hatenablog.jp/
3. Introduction લஔ͖ ׬શҠߦ͸·ͩͰ͖͍ͯͳ͍
4. Introduction લஔ͖ %PDLFSJ[F΍&$4Խ͸ɺ݄͘Β͍ʹ࢝Ίͨ
5. Introduction લஔ͖ ͜͜ϲ݄Ͱ΍ͬͨ͜ͱΛ࿩͠·͢
6. Introduction લஔ͖ ࣌ؒͷ౎߹্ɺ͍ͩͿ಺༰Λ࡟ͬͨͷͰ
   ͋ͱͰิ଍هࣄॻ͖·͢
7. Introduction લఏ஌ࣝ ͲΜͳόονΛ૸Β͍ͤͯΔͷ͔ʁ
8. Introduction લఏ஌ࣝ $47Λड͚औΓɺม׵ॲཧΛࢪ͢όον
9. Introduction લఏ஌ࣝ &$4ͱ͸ʁ
10. Introduction લఏ஌ࣝ ߴੑೳͳίϯςφ؅ཧαʔϏε
11. Introduction લఏ஌ࣝ &$ΠϯελϯεΛऔΓ·ͱΊͯ؅ཧ্ͨ͠Ͱɺ
    ͋Β͔͡ΊఆΊͨ௨ΓʹίϯςφΛάϦάϦಈ͔ͨ͢Ίͷ ϚωʔδυαʔϏε
12. Table Of Contents Agenda ݱঢ় 1 2 ਐΊํ 3 ؾΛ͚ͭͨ͜ͱ 4 ͭΒ͔ͬͨ͜ͱ 5 ·ͱΊ 6 ໨ࢦ͢΋ͷ
13. ݱঢ়
14. ݱঢ় αʔόߏ੒ਤ ˞
    Ұ෦ׂѪ 3BJMTΞϓϦ %#ແ͠
15. ݱঢ় εέʔϧ͢ΔͱͭΒ͘ͳΔ ɾDSPO
    TFSWFS͕410'
    ɾෛՙ͕ෆۉߧ Ջͳαʔό͕͋Δ
    ɾखಈεέʔϧ
16. ໨ࢦ͢΋ͷ
17. ໨ࢦ͢΋ͷ ΄ͬͱ͍͓͚ͯΔঢ়ଶ ˞
    ఆ࣮࣌ߦ͸MBNCEBʹ೚ͤΔ
18. ໨ࢦ͢΋ͷ ྑ͘ͳΔ͜ͱ ɾδϣϒ͕Ͳ͜Ͱಈ͔͘ؾʹ͠ͳͯ͘Α͘ͳΔ
    ɾϦιʔε͕༨ͬͯΔΠϯελϯεͰ࣮ߦ͞ΕΔ
    ɾ଍Γͳ͘ͳͬͨΒΦʔτεέʔϧ
19. ਐΊํ
20. ਐΊํ ։ൃମ੍ Engineer Leader Engineer Engineer  !  %PDLFSৄ͍͠ %PDLFSॳ৺ऀ ೥৽ଔೖࣾ
    %PDLFSॳ৺ऀ શһ"QQMJDBUJPO
    &OHJOFFS
21. %PDLFSॳ৺ऀूஂʜ
22. ৽ਓ΋͍Δ͚Ͳʜ
23. ڭҭίετ͸੎͍ͰΧόʔ
    
24. ࢿྉ΋๛෋ͩ͠վΊͯڭҭ͢Δඞཁͳ͍Ͱ͠ΐ
25. ڧؾͷ࢟੎
26. ڧؾͷ࢟੎
27. ੎͍ͷ͋Δ৽ਓ IUUQCJUMZD3L:J
28. ۀ຿࣌ؒ֎ͳͷʹͳ͔ͥଈϨε
29. ؾ͍ͮͨΒΈΜͳৄ͘͠ͳͬͯͨ
30. ೔ʹ೔ʹৄ͘͠ͳ͍ͬͯ͘ਤ
31. ਐΊํ &$4ԽΛͲͷΑ͏ʹਐΊ͍͔ͯͬͨ ։ൃ؀ڥͷDockerԽ ɾDockerize ɾdocker-compose ECS؀ڥͷߏங ɾrun-taskͰδϣϒ࣮ߦ ɾCircleCI੔උ ӡ༻४උ ɾAutoscaling ruleߟ͑Δ ɾmackerel ɾͦͷଞʢlambdaͱ͔ʣ ׬શҠߦ ɾঃʑʹҠߦ ↑ ΠϚίίͷ։ൃத ↑ ίίϚσ࿩͢
32. ਐΊํ
    
    ։ൃ؀ڥͷ%PDLFSԽ %PDLFSJ[F FROM centos:6.8 FROM fluent/fluentd:v0.12.26-onbuild FROM centos:6.8
33. ਐΊํ
    
    ։ൃ؀ڥͷ%PDLFSԽ %PDLFSJ[F %PDLFS
    *NBHFͷμΠΤοτ͸͋ͱ·Θ͠
34. ਐΊํ
    
    ։ൃ؀ڥͷ%PDLFSԽ %PDLFSJ[F ૫ͤΑ͏ͱ͢Δҙࢥ͸͋Δ
35. ਐΊํ
    
    ։ൃ؀ڥͷ%PDLFSԽ EPDLFSDPNQPTFΛ࢖͏ ϩʔΧϧͰಈ࡞֬ೝ͕͠΍͍͢
36. ਐΊํ
    
    &$4؀ڥͷߏங &$4Ϋϥελ͸։ൃϝϯόʔ෼༻ҙ ։ൃϝϯόʔ༻
    
    QSPEVDUJPO༻
    
    ͭ ಈ࡞֬ೝ͕͠΍͍͢ɻؾܰʹࢼͤΔͷͰཧղ΋ਂ·Δɻ
    &$4ʹؔ͢Δݕূ͸֤ࣗΑΖ͘͠΍ͬͯ΋Β͏ɻ
37. ਐΊํ
    
    &$4؀ڥͷߏங &$4Ϋϥελ͸։ൃϝϯόʔ෼༻ҙ 6TFSEBUB͸͜Μͳײ͡ lib/tasks/aws/production.rake lib/tasks/aws/sandbox.rake
38. ਐΊํ
    
    &$4؀ڥͷߏங 5BTL
    EFGJOJUJPOͷߏ੒ CBUDI༻
39. ਐΊํ
    
    &$4؀ڥͷߏங 5BTL
    EFGJOJUJPOͷߏ੒ GMVFOUE༻
40. ਐΊํ
    
    &$4؀ڥͷߏங 5BTLͷߏ੒ ͖ͬ͞ͷ5BTL
    EFGJOJUJPO͕͜͏ͳΔ Docker container Docker container Docker container batch༻ Task fluentd༻ Task
41. ਐΊํ
    
    &$4؀ڥͷߏங &$4্Ͱ͸͜͏ͳΔ GMVFOUE༻5BTL͕4FSWJDFͱͯ͠ৗ࣌Քಇ͍ͯ͠Δ ECS Cluster Internal CLB EC2 ECS Agent Task Service EC2 ECS Agent Task Service EC2 ECS Agent EC2 ECS Agent :24224 :24224 "desiredCount": 2, "deploymentConfiguration": { "maximumPercent": 200, "minimumHealthyPercent": 100 } http://bit.ly/2clQnv5
42. ਐΊํ
    
    &$4؀ڥͷߏங &$4Ͱͷߏ੒ SVOUBTL͢ΔͱCBUDI͕૸Δ ECS Cluster Internal CLB EC2 ECS Agent Task Service EC2 ECS Agent Task Service EC2 ECS Agent EC2 ECS Agent :24224 :24224 ECS Scheduler batch Task
43. ਐΊํ
    
    &$4؀ڥͷߏங &$4Ͱͷߏ੒ &-#Λ࢖ͬͨ4FSWJDF
    EJTDPWFSZ ECS Cluster Internal CLB EC2 ECS Agent Task Service EC2 ECS Agent Task Service EC2 ECS Agent EC2 ECS Agent :24224 :24224 ECS Scheduler batch Task sandbox-wakayama.vpclocal:24224 "logDriver": "fluentd", "options": { "tag": “docker.batch-app", "fluentd-address": “sandbox-wakayama.vpclocal:24224" }
44. ਐΊํ
    
    &$4؀ڥͷߏங $JSDMF$*ܦ༝ͰͷϗοτσϓϩΠ ϚʔδΛϑοΫʹίϯςφ͕σϓϩΠ͞ΕΔ deploy aws ecs update-service Serverspec bundle exec rake serverspec RSpec bundle exec rake spec ! git merge topic branchΛϚʔδ CircleCI workflow hot deploy via CircleCI " ECR docker push
45. ؾΛ͚ͭͨ͜ͱ
46. ؾΛ͚ͭͨ͜ͱ %PDLFSGJMF͸Ͱ͖Δ͚ͩಡΈ΍͘͢͢Δ ൿ఻ͷλϨԽΛ๷͙ɻ
47. 
    lazͰฒ΂Δ΍ͭ
    গ͠Ͱ΋ಡΈ΍͘͢ͳΔΑ͏ʹɻ
48. ґଘੑͷ͋ΔQBDLBHF͸&/7ʹࢦఆͯ͠͠·͏
    ʮ͜ΕͬͯͳΜͷͨΊʹೖΕͯΜͷʁʯͷ๷ࢭ
49. ؾΛ͚ͭͨ͜ͱ ࢖͍͍ͨϩάυϥΠό͸6TFSEBUBʹॻ͔ͳ͍ͱ࢖͑ͳ͍ &$4"HFOUͷઃఆ
50. σϑΥϧτ͸[“json-file"]ͷΈɻ໘౗ͳͷͰશ෦ॻ͍ͨɻ https://docs.aws.amazon.com/ja_jp/AmazonECS/latest/developerguide/ecs-agent-config.html
51. ͭΒ͔ͬͨ͜ͱ
52. ஌ݟ ͭΒ͔ͬͨ͜ͱ ίϯςφͷ؀ڥม਺ͷ੾Γସ͑
53. EPDLFSGJMFTGMVFOUEFYQPSU@FOWGJMFTI EPDLFSGJMFTGMVFOUE%PDLFSGJMF &OUSZLJUͷTXJUDIͰؤுΔελΠϧ ൿಗ஋ΛؚΜͰ͍ΔͷͰ4͔Βऔ͖͍ͬͯͯΔ 44&ࡁΈ
54. ஌ݟ ͭΒ͔ͬͨ͜ͱ $JSDMF$*ͰΩϟογϡ͕ޮ͔ͳ͍ dockerfileͷfull build͕ຖճ૸Δɻ load/saveํࣜͰͰ͖Δͱॻ͍ͯ͋Δ͕ɺͦΕͰ΋Ωϟογϡ͞Εͳ͍ɻ ɾhttps://circleci.com/docs/docker/#caching-docker-layers
55. ஌ݟ ͭΒ͔ͬͨ͜ͱ "-#࢖͑ͳ͔ͬͨ Service discoveryʹCLBΛ࢖ͬͯΔ͕ALBʹ͍ͨ͠ɻ ͔͠͠ɺCLB͸fluentd༻ʹ24224 (TCP)ΛListen͍ͯ͠ΔͷͰ ALBͷTCPରԠ଴ͪɻ
56. ·ͱΊ
57. ·ͱΊ Θ͔ͬͨ͜ͱ ɾDockerfile͸ಡΈ΍͘͢ॻ͍ͨ΄͏͕͍͍ɻ ɾόονͳΒECSԽ͸ͦΜͳʹϋϚΒͣεοͱͰ͖Δɻ ɾECSԽʹ൐͏Dockerize΍Autoscalingͷ෭࣍ޮՌ͕େ͖͍ɻ
58. ͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ ɾDockerfile͸ಡΈ΍͘͢ॻ͍ͨ΄͏͕͍͍ɻ ɾόονͳΒECSԽ͸ͦΜͳʹϋϚΒͣεοͱͰ͖Δɻ ɾECSԽʹ൐͏Dockerize΍Autoscalingͷ෭࣍ޮՌ͕େ͖͍ɻ 5)"/,
    :06 -FU`T
    USZ
    &$4