【アソビュー×ランサーズ】AWSでのオーケストレーションツールの活用事例(EKS vs ECS)
https://lancersrecruit.connpass.com/event/187989/
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)ECS/Fargateͷ׆༻ࣄྫ(Ruby on Railsฤ)ϥϯαʔζגࣜձࣾSRE/҆ୡ ྋ
View Slide
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)ΞδΣϯμ ɾࣗݾհɾݱࡏऔΓΜͰ͍Δ͜ͱɾECS/FargateͷϝϦοτσϝϦοτɾ։ൃڥΛ࡞ΔϙΠϯτɾStg/ຊ൪ڥͷߏʹ͍ͭͯɾSSH/࡞ۀ༻ίϯςφʮdevopsʯʹ͍ͭͯɾCircleCIʹΑΔίϯςφσϓϩΠɾECS Scheduled TasksʹΑΔόον࣮ʹ͍ͭͯɾϩά/ίϯςφࢹʹ͍ͭͯɾ·ͱΊ
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)ࣗݾհ
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)ࣗݾհ ɾ҆ୡ ྋ(adachin)ɾLancers SRE 2018~ɾ@adachin0817ɾRVIRUS0817ɾblog.adachin.meɾwiki.adachin.meɾOSS :ɾCakePHPͰݸਓαʔϏε։ൃத
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)ݱࡏऔΓΜͰ͍Δ͜ͱ
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)ݱࡏऔΓΜͰ͍Δ͜ͱ ɾڈळɾάϧʔϓձࣾ(γΫϩϚʔέςΟϯά)ɾ͘͞ΒͷΫϥυͰӡ༻ɾॳΊͯECS/FargateʹҠߦνϟϨϯδɾࠓ4݄ʹҠߦྃɹɾࠓճͪ͜ΒͷΛ͠·͢!!ɾݱࡏ(5݄͔Β)ɾάϧʔϓձࣾ(Lancers Agency)ɹɾECS/FargateͰҠߦத!!
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)EC2ͱൺֱͨ͠ECS/FargateͷϝϦοτσϝϦοτ
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)EC2ͱൺֱͨ͠ECS/FargateͷϝϦοτσϝϦοτ ϝϦοτɾଐਓԽ͕ൃੜ͠ͳ͍ɾImmutableͳαʔόʔͷӡ༻ɹɾ҆ఆӡ༻ɺӡ༻ޮͷΞοϓɾϗετΠϯελϯεͷཧ͕ল͚ΔɾλεΫ୯ҐͰࣗ༝ʹϦιʔεมߋՄೳɾεέʔϧΞτͷ͢͠͞σϝϦοτɾσόοά͕ͮ͠Β͍ɾdocker execSSH͕Ͱ͖ͳ͍ɾλεΫͷىಈ͕Ίɹɾόον࣮ߦ͘ͳΔɾྉ͕͓ۚߴΊɹɾEC2ΑΓׂߴʢ1ׂʙ2ׂʣ
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)։ൃڥΛ࡞ΔϙΠϯτ
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)։ൃڥΛ࡞ΔϙΠϯτ ɾຊ൪ڥͱಉͷߏΛ࠶ݱɾELB(H2O)ίϯςφ(ϦόʔεϓϩΩγ)ɾApp(ίϯςφ)ɹɾruby:2.5.7ɹɾRails4.2.11ɹɾNginxɾMySQLίϯςφɹɾ5.7ɾ֤ΞϓϦέʔγϣϯͷىಈSupervisorͰཧɾbundle installAppίϯςφͰϩάΠϯ࣌ʹ࣮ߦɹɾsupervisorctlͰΞϓϦΛrestart͢ΔΑ͏ʹ
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)Stg/ຊ൪ڥͷߏʹ͍ͭͯ
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)Stg/ຊ൪ڥͷߏʹ͍ͭͯ ɾTerraformͰશϦιʔείʔυԽɾStgڥλεΫ1ͭɾdevopsίϯςφͷར༻ɾDBRDS AuroraʹҠߦɾը૾S3ʹอଘɾϩάCloudWatch LogsɾόονECS Scheduled Tasksɾ෦ࠪ(ϩάͷू)ɹɾAmazon KinesisɾσϓϩΠCircleCIͷΈͰ࣮ɾࢹDatadog
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)SSH/࡞ۀ༻ίϯςφʮdevopsʯʹ͍ͭͯ
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)SSH/࡞ۀ༻ίϯςφʮdevopsʯʹ͍ͭͯ ɾAppʹSSH͠ͳ͍ɾSSH/࡞ۀ༻ίϯςφ(Appͱಉ)ɹɾdevopsͱ໋໊ɾίϚϯυͰRDSଓɹɾdumprestoreɾaws cliͷར༻ɾrails consoleͰͷσόοάɾSSMͷSession Manager ͋Δ͕ɹɾखܰʹ౿Έ͔ΒSSHΛ͍ͨͨ͠ΊɾVPC PeeringͰϥϯαʔζͷωοτϫʔΫ͔ΒSSHϩάΠϯ
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)CircleCIʹΑΔίϯςφσϓϩΠʹ͍ͭͯ
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)CircleCIʹΑΔStg/ຊ൪σϓϩΠʹ͍ͭͯ(ॳظ) ɾCircleCIͷOrbsɹɾDockerfileͷbuildɹɾECRpushɹɾECSϦϏδϣϯͷߋ৽ɹɾαʔϏελεΫఆٛͷߋ৽ɹɾDBϚΠάϨʔγϣϯɹɾrspecɾgit flowɾdevelopϒϥϯνɹɾϚʔδ͢ΔͱStgʹϦϦʔεɾmasterϒϥϯνɹɾϚʔδ͢Δͱຊ൪ʹϦϦʔε
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)APIܦ༝ͰͷStgڥͷσϓϩΠ(ݱࡏ) ɾStgڥͷσϓϩΠɹɾgithub flowɹɾγΣϧεΫϦϓτɹɹɾҙͷϒϥϯνΛࢦఆɾૉૣ͘σϓϩΠ͕Մೳɹɾ։ൃޮ͕ΞοϓɾTerraform CIڥΛ࣮$ sh deploy.shԼهͷΑ͏ʹϒϥϯνΛࢦఆ࣮ͯ͠ߦ͍ͯͩ͘͠͞ɻ(ྫ) sh deploy.sh ϒϥϯν໊$ sh deploy.sh fix-nginx{"number" : 74,"state" : "pending","id" : "xxxxxx-xxxxx-xxxxx-xxx-xxxxxxxx","created_at" : "2020-08-27T06:14:43.973Z"}ϒϥϯν໊/ fix-nginx ͷStgσϓϩΠΛ։࢝͠·ͨ͠!
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)όονͷ࣮ʹ͍ͭͯ
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)ECS Scheduled TasksʹΑΔόον࣮ʹ͍ͭͯ ɾࠓ·Ͱɹɾgem wheneverɹɾschedule.rbɾݱࡏɹɾECS Scheduled TasksɹɹɾCloudWatchͷΠϕϯτϧʔϧɹɹɾTerraformԽ{"containerOverrides": [{"name": "HogeWorker","command": ["bundle","exec","rails","runner","HogeWorker.new.execute"]}]}
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)ϩάʗίϯςφࢹʹ͍ͭͯ
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)ϩά/ίϯςφࢹʹ͍ͭͯ ɾΞΫηε/ΞϓϦϩάɺRDSࠪ/ΤϥʔϩάɹɾCloudWatch Logsʹɹɾࠓ·ͰɹɹɾLambdaͰS3ʹγϯΫɹɹɹɹɾPythonͰ࣮͍ͯͨͨ͠Ίӡ༻ίετΞοϓɹɾࠓճɹɹɾAmazon Kinesis Data FirehoseͰS3ʹूɹɹɾTerraformԽɾࢹɹɾMackerel→DatadogʹҠߦɹɾλεΫఆٛʹdatadog-agentΛಈ࡞ͤ͞ΔΑ͏ʹɹɾcpu:10ɺmemory:256 TerraformԽ
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)·ͱΊ
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)·ͱΊ ɾγϯϓϧͳߏͰ࣮ݱͰ͖ͨɾখ͍͞αʔϏε͔ΒECS/FargateԽதɾDockerΠϝʔδͱλεΫఆٛͰӡ༻ՄೳɾॳΊͯίϯςφຊ൪ӡ༻͢Δʹ͔ͳΓϋϚΔɹɾϔϧενΣοΫ͕௨Βͳ͍ͱίϯςφ͕མͪͨΓɹɾσϓϩΠͷ࣮͕େมɹɾLinuxྗ͕ࢼ͞ΕΔɾࠓޙɹɾRuby2.7.1/Rails6ͷόʔδϣϯΞοϓɹɾίϯςφͷ੬ऑੑݕ (Trivy)ɹɾσϓϩΠ͕10͔͔ΔͷͰվળɹɾChatOpsͰͷϦϦʔεɹɾDatadogͰϩά࿈ܞɹɾϥϯαʔζຊՈίϯςφҠߦʹνϟϨϯδ
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)SREٻΉʂʂ ECS/Fargate/PHPͰͷڥΛ͓͠·͢ʂ
2020/9/18 ʲΞιϏϡʔ×ϥϯαʔζʳAWSͰͷΦʔέετϨʔγϣϯπʔϧͷ׆༻ࣄྫ(EKS vs ECS)