Lancersをコンテナへ本番移行する取り組み

https://lancersrecruit.connpass.com/event/248522/

Transcript

1. ϥϯαʔζגࣜձࣾ
   
   
   ϓϩμΫτ։ൃ෦/ٕज़ج൫ SRE ҆ୡ ྋ
   ʲLancers x dipʳEngineer Meetup #1
   LancersΛίϯςφ΁ຊ൪Ҡߦ͢ΔऔΓ૊Έ
   

   View Slide

2. ࣗݾ঺հ
   • ҆ୡ ྋ(adachin)
   
   
   • Lancers/SRE
   

   2018/5~
   
   
   • @adachin0817
   
   
   • RVIRUS0817
   
   
   • blog.adachin.me
   
   
   • wiki.adachin.me
   
   
   • ݸਓαʔϏε։ൃத(ਐḿѱ)
   ʲLancers x dipʳEngineer Meetup #1
   

   View Slide

3. ࠓճ͓࿩͍ͨ͜͠ͱ
   • 2018೥ͷγεςϜ
   
   
   • TerraformԽਪਐ
   
   
   • Amazon Linux2Խ
   
   
   • ϩάαʔόʔͷഇࢭ
   
   
   • ։ൃ؀ڥͷվળ
   • ϑϩϯτΤϯυͷվળ
   
   
   • چ։ൃ؀ڥɺEC2Ͱͷӡ༻՝୊
   
   
   • Batch/AppΛίϯςφ΁
   
   
   • ࣗલσϓϩΠγεςϜΛഇࢭ
   
   
   • ෛՙରࡦͱϨεϙϯε
   
   
   • ·ͱΊ
   ʲLancers x dipʳEngineer Meetup #1
   

   View Slide

4. ฐࣾʹ͍ͭͯ
   ʲLancers x dipʳEngineer Meetup #1
   

   View Slide

5. άϧʔϓձࣾ͸ECS/FargateʹҠߦࡁΈ
   ʲLancers x dipʳEngineer Meetup #1
   

   View Slide

6. Ҏલొஃ͍ͯͨ͠εϥΠυ͸ͪ͜Β
   ʲLancers x dipʳEngineer Meetup #1
   

   View Slide

7. ʲLancers x dipʳEngineer Meetup #1
   

   View Slide

8. 2018೥ϥϯαʔζͷαʔόʔߏ੒
   ʲLancers x dipʳEngineer Meetup #1
   

   View Slide

9. 2021೥ϥϯαʔζͷαʔόʔߏ੒
   ʲLancers x dipʳEngineer Meetup #1
   

   View Slide

10. 2021/10~(࣮ࢪ)
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

11. TerraformԽ
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

12. Ұ෦TerraformԽΛ͍ͯ͠Δ͕׬શͰ͸ͳ͍
    • άϧʔϓձࣾ͸શͯTerraformԽ
    
    
    • ຊՈ͸Ұ෦͔͠TerraformԽ͞Ε͍ͯͳ͍
    

    ɾηΩϡϦςΟʔάϧʔϓIDͳͲΛͦͷ··ࢦఆ
    

    ɾEC2Ͱͷߏங͸ಠࣗͷAWS CLIͰࣗಈԽ
    

    ɾͦΕҎ֎͸खಈͰઃఆ
    
    
    • શͯΛterraform importΛ͢Δʹ͸ݫ͍͠
    

    ɾϑΣʔζ͝ͱʹऔΓ૊Ή͜ͱʹ
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

13. Terraform Phase 1
    • VPC
    

    ɾαϒωοτ
    

    ɾϧʔτςʔϒϧ
    

    ɾΠϯλʔωοτήʔτ΢ΣΠ
    

    ɾNatήʔτ΢ΣΠ
    

    ɾΤϯυϙΠϯτ
    

    ɾηΩϡϦςΟάϧʔϓ
    
    
    • ELB
    

    ɾλʔήοτάϧʔϓ
    

    ɾϦεφʔϧʔϧ
    ʲLancers x dipʳEngineer Meetup #1
    • EC2
    
    
    • RDS
    
    
    • ACM
    
    
    • εςʔδϯάͱຊ൪؀ڥ߹Θͤͯ
    

    ɾ໿1ϲ݄൒
    

    ɾͦΕҎ֎͸ࠓظରԠ
    

    View Slide

14. Terraform ։ൃ؀ڥͱCI/CD
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

15. Amazon Linux 2Խ
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

16. EC2ΛAmazon Linux 2Խ
    • Amazon Linux1Ͱӡ༻
    

    ɾAppɺ؅ཧը໘(Admin)ɺBatchαʔόʔ
    

    ɾ2020೥6݄3೔ʹηΩϡϦςΟΞοϓσʔτͷఏڙ͕ऴྃ
    
    
    • άϧʔϓձࣾͷҠߦ͕ॏͳΓରԠͰ͖ͣ
    
    
    • ͞ΒʹArm͕ରԠ͞ΕͯύϑΥʔϚϯε্͕ঢͱίετ࡟ݮ
    
    
    • άϧʔϓձࣾͷίϯςφͷϊ΢ϋ΢͕ߴ·ΓҠߦΛܾҙ
    

    ɾಉ࣌ʹMySQL8Խ΋औΓ૊Ή͜ͱʹ
    

    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

17. ϩάαʔόʔͷഇࢭ
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

18. ϩάαʔόʔ͕୯Ұো֐఺
    • ෼ੳج൫(Digdag/Embulk/BigQuery/Redash)Λӡ༻
    

    ɾϩάج൫(Fluentd)
    

    ɾApp(ΞΫηεϩά/ΞϓϦέʔγϣϯϩά)
    

    ɹɾS3ʹू໿
    
    
    • ਖ਼ৗʹಈ࡞͍ͯ͠ͳ͍ঢ়ଶ͕౓ʑݟड͚ΒΕ
    

    ɾϒϥοΫϘοΫεԽͱ௕೥ͷ՝୊ʹ…
    
    
    • td-agentͷόʔδϣϯΞοϓΛࢼΈ͕ͨ
    

    ɾ͏·͍͔͓ͣ͘ख্͛ঢ়ଶ
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

19. Amazon KinesisʹҠߦ
    • CloudWatch Logs(awslogs)Λ࠾༻
    

    ɾ։ൃϝϯόʔͰ΋ௐࠪͰ͖ΔΑ͏ʹ
    

    ɾൺֱతֶशίετ͕௿͍awslogsΛબ୒
    
    
    • ϩάϙϦγʔΛݟ௚͢
    

    ɾΞΫηεϩά͸S3ʹू໿ͷ··
    

    ɾΞϓϦϩά͸෼ੳ༻్ͱͯ͠ར༻͠ͳ͍
    
    
    • Datadog LogsͰKinesisͷ࿈ܞ
    

    ɾΞϓϦνʔϜ΋ॠ࣌ʹରԠՄೳ
    

    ɾՄ؍ଌੑΛߴΊΒΕΔΑ͏ʹ
    
    
    • ίϯςφʹҠߦ͢Δ४උ͕੔ͬͨ
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

20. ։ൃ؀ڥͷमਖ਼
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

21. ։ൃ؀ڥͷमਖ਼
    • ։ൃ؀ڥ͸ຊ൪ಉ౳Λ࠶ݱ
    

    ɾAnsibleίϯςφ
    

    ɹɾύοέʔδΠϯετʔϧ
    
    
    • มߋ఺
    

    ɾDocker Image͸amazonlinux2
    

    ɾsupervisorͰϓϩηε؅ཧ
    

    ɾෆཁͳύοέʔδΛચ͍ग़͠࡟আ
    

    ɾamazon-linux-extrasʹΑΔPHPΠϯετʔϧ
    

    ɾtd-agentͷ࡟আͱawslogsͷΠϯετʔϧ
    
    
    • ؅ཧը໘͸ECS/FargateʹҠߦ͢ΔͨΊ
    

    ɾDocker
    fi
    leΛҰ͔Βbuild͢ΔΑ͏ʹ
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

22. 2022/03~(Ҡߦ)
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

23. Ҡߦ͢Δʹ͋ͨͬͯ
    • ؅ཧը໘(Admin)͸ECS/FargateʹҠߦ
    

    ɾCakePHP2͸EC2
    

    ɾCakePHP4͸ECS/FargateͰಈ࡞
    
    
    • Batch͸ArmʹҠߦ
    
    
    • App͸ArmʹҠߦͰ͖ͣintelͰӡ༻
    

    ɾDatadog
    

    ɹɾdd-trace-php͕ArmΛαϙʔτ❌
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

24. ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

25. ͜͜Ͱٙ໰ʹࢥ͏ํ
    ΋͍ΔͷͰ͸
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

26. App͸ίϯςφʹ
    

    Ҡߦ͠ͳ͍ͷ͔ʁ
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

27. ϑϩϯτΤϯυͷ
    

    ՝୊
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

28. ϑϩϯτΤϯυͷωοΫͱվળ
    • ιʔείʔυ͕5GBҎ্
    

    ɾyarn installͰ30෼Ҏ্͔͔Δ
    

    ɾίϯςφσϓϩΠͰ͕࣌ؒ๲େʹ͔͔Δ
    
    
    • ϑϩϯτΤϯυΛ෼ׂ͔ͯ͠ΒҠߦ
    

    ɾιʔεߏ଄্ɺύεʹΑΔॲཧ෼ׂ͕ࠔ೉Ͱ޻਺͕͔͔Δ
    
    
    • ༧ΊϩʔΧϧͰϏϧυͯ͠ੜ੒͞ΕͨϑΝΠϧΛGit؅ཧ͢Δӡ༻
    

    ɾyarn install͸ෆཁʹͳΓɺCI͕࣌ؒେ෯ʹ୹ॖ
    

    ɾAppαʔόʔͷίϯςφ΁ҠߦͰ͖Δ৚͕݅੔ͬͨ
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

29. 2022/04(࣮ࢪ)
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

30. چ։ൃ؀ڥɺEC2Ͱͷ
    ӡ༻՝୊
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

31. SREͷӡ༻՝୊
    • AnsibleίϯςφʹΑΔ։ൃ؀ڥͷ౷ҰʹΑΓ޻਺͕͔͔Δ
    

    ɾAppίϯςφΛECRͰڞ༻͢Δඞཁ͕͋Δ
    

    ɾύοέʔδͷߋ৽ͳͲίϯςφΛ࡞Γ௚ͯ͠ECRʹ౎౓push
    
    
    • Φʔτεέʔϧͷ࠶ઃఆͰAppαʔόʔΛ࡞Γ௚͞ͳ͚Ε͹ͳΒͳ͍
    

    ɾAMIΛखಈͰ࡞Γ௚͢ͱ͍͏τΠϧ͕࢒ͬͨ··
    
    
    • ࣗલͷσϓϩΠγεςϜ͕ଐਓԽ
    

    ɾෆ۩߹ͷରԠʹ޻਺͕͔͔Δ
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

32. ։ൃ؀ڥͷվળ
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

33. ৽։ൃ؀ڥ
    • Docker
    fi
    leʹΑΔҰൠతͳӡ༻
    
    
    • ผϦϙδτϦ(playbook)
    

    ɾLancersຊମͷϦϙδτϦʹ౷Ұ
    
    
    • Ansibleίϯςφ͸ഇࢭ
    
    
    • ϩάج൫Ͱར༻
    

    ɾEC2ͱಉ༷awslogsΛΠϯετʔϧ
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

34. BatchΛίϯςφ΁
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

35. Batchαʔόʔ͸ECS Scheduled TaskʹҠߦ
    • όον͸100ݸҎ্
    

    ɾ৑௕ԽͤͣEC2 1୆ͷΈͰӡ༻
    

    ɾαʔόʔμ΢ϯ΍AZো֐͕ى͖Δͱ
    

    ɹɾαʔϏεʹӨڹ͕ग़ͯ͠·͏ঢ়گ
    
    
    • CakePHP4
    

    ɾ80ݸͷΈҠߦ
    

    ɾ։ൃϝϯόʔ͕TerraformͰ؅ཧͰ͖ΔΑ͏ʹ
    
    
    • CloudWatch LogsɺSNSɺLambdaͰΤϥʔ௨஌
    
    
    • CakePHP2ͷόον͸QAνʔϜ͕Ҡߦத
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

36. ࣗલσϓϩΠγεςϜ
    ʹ͍ͭͯ
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

37. ʮDeploy-Sanʯ
    • 2015~
    

    ɾRuby on RailsͱJenkins
    

    ɾಠࣗγΣϧͰrsync
    
    
    • Φʔτεέʔϧ࣌
    

    ɾDeploy-san͔Β࠷৽ͷίʔυΛऔಘ
    

    ɾ୯Ұো֐఺
    
    
    • ϦϦʔε࣌ؒ
    

    ɾࠩ෼ߋ৽Ͱ1෼Ҏ಺
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

38. CircleCIʹҠߦ
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

39. CI/CDΛCircleCIʹҠߦ
    • MasterϚʔδ:ຊ൪
    

    APIܦ༝:εςʔδϯά/ΧφϦΞ
    
    
    • ίϯςφσϓϩΠߴ଎Խ
    

    ɾ.dockerignoreʹΑΔෆཁͳϑΝΠϧ͸ল͘
    

    ɾDocker
    fi
    leͰෆཁͳύοέʔδͷ୨Է͠
    

    ɾCircleCIͷcheckout_shallowΛར༻
    

    ɹɾ࠷ޙͷίϛοτͷΈcheckout
    

    ɾremote-docker-layer-cachingΛ༗ޮԽ
    
    
    • 5෼Ҏ಺·ͰॖΊΔ͜ͱ͕Ͱ͖ͨ
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

40. ΧφϦΞ؀ڥͷσϓϩΠΛ੍ݶ
    • ຊ൪DBΛࢀর
    

    ɾΧφϦΞ؀ڥʹ୭Ͱ΋ϦϦʔεͰ͖ͯ͠·͏
    
    
    • CircleCI
    

    ɾcontext
    

    ɹɾSecurity Group
    

    ɹɾGitHub Teamsͷݖݶ͋ΔਓͷΈঝೝ
    
    
    • https://wiki.adachin.me/archives/1313/
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

41. ෛՙରࡦͱϨεϙϯε
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

42. ෛՙରࡦ
    • EC2࣌୅
    

    ɾc5ܥΠϯελϯε
    

    ɾECSͩͱCPU͸બ୒Ͱ͖ͳ͍
    
    
    • Fargate Spot
    

    ɾطଘͷ3୆͸Fargate
    

    ɾΦʔτεέʔϧ͸Fargate Spot(ίετ࡟ݮ)
    
    
    • Φʔτεέʔϧ
    

    ɾ୆਺͸࠷খͰ3୆ɺ࠷େͰ20୆
    

    ɾCPU 20%Ͱ૿Ճ 10%Ͱݮগ
    
    
    • php-fpm΋νϡʔχϯά͠௚ͨ͠
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

43. 2022/06(Ҡߦ)
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

44. ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

45. ҠߦޙͷϨεϙϯεʹ͍ͭͯ
    • ϥϯαʔζ͸೔தʹෛՙ͕ϐʔΫ
    
    
    • αʔόʔϨεϙϯε͸਺ेmsఔ౓
    
    
    • ݱࡏ
    

    ɾCPU 60%Ͱ૿Ճ 40%Ͱݮগ
    

    ɾΦʔτεέʔϧ7୆
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

46. Ҡߦޙͷίετ
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

47. ·ͱΊ
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

48. ·ͱΊ
    • ໿1೥Ͱ΍Γ͖Δ͜ͱ͕Ͱ͖ͨ
    

    ɾάϧʔϓձࣾͰͷϊ΢ϋ΢Λ׆͔ͤͨ
    
    
    • ੜ࢈ੑ޲্͕ظ଴
    

    ɾSREͷτΠϧ࡟ݮ
    

    ɾ։ൃ෦શମͷϦϦʔεϑϩʔվળ
    
    
    • ࠓޙ
    

    ɾϑϩϯτΤϯυͷϞμϯԽ
    

    ɾPHP8.1/CakePHPͷόʔδϣϯΞοϓ
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

49. ΤϯδχΞϒϩά΋ॻ͍ͯ·͢
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide

50. ͝ਗ਼ௌ͋Γ͕ͱ͏
    
    
    ͍͟͝·ͨ͠
    ʲLancers x dipʳEngineer Meetup #1
    

    View Slide