2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE MENTAΛAWSʹҠߦͯ͠ৼΓฦΔ (ECS/Fargate + Laravelฤ) ϥϯαʔζגࣜձࣾ 
 SRE/҆ୡ ྋ

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ΞδΣϯμ ɾࣗݾ঺հ ɾ3೥લͷγεςϜߏ੒/͘͞ΒͷΫϥ΢υʹ͍ͭͯ(ۤ࿑࿩) ɾ։ൃ؀ڥΛ࡞ΔϙΠϯτ ɾStg/ຊ൪؀ڥͷߏ੒ʹ͍ͭͯ ɾCircleCIʹΑΔίϯςφσϓϩΠ ɾECS Scheduled TasksʹΑΔόον࣮૷ʹ͍ͭͯ ɾϩά/ίϯςφ؂ࢹʹ͍ͭͯ ɾຊ൪DBϚεΩϯάࣗಈԽʹ͍ͭͯ ɾECSΦʔτεέʔϧʹ͍ͭͯ ɾ෼ੳج൫ͷ঺հ(Digdag,Embulk,BigQuery,Redash) ɾҠߦલͷ՝୊ͱҠߦޙͷτϥϒϧ ɾ·ͱΊ

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ࣗݾ঺հ

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ࣗݾ঺հ ɾ҆ୡ ྋ(adachin) ɾLancers SRE 2018/5~ ɾ@adachin0817 ɾRVIRUS0817 ɾblog.adachin.me ɾwiki.adachin.me ɾݸਓ։ൃ(famii) ɾOSS

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ͪͳΈʹ…

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ͪͳΈʹ… MENTAೖΕΔͱ3ͭίϯςφʹҠߦ͠·ͨ͠ʂ PROSheet 
 Lancers Agent Lancers Assistant 
 Creative

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ͪͳΈʹMENTAͰ͸..?

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ͪͳΈʹMENTAͰ͸..? ɾΠϯϑϥ/αʔόʔ ίʔνϯά ɾMENTAϥϯΩϯά ɹɾ4Ґʂ ɾྦྷܭ184ਓʂ 
 ɹɾۦ͚ग़͠ΤϯδχΞϑΥϩʔ ɹɾస৬࣮੷ଟ਺༗Γ

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE 3೥લͷγεςϜߏ੒

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE 3೥લͷγεςϜߏ੒ ɾ2018೥ϦϦʔεɹ ɾ෭ۀͰࣗ෼͕αʔόʔอक͍ͯͨ͠ 
 ɾ͘͞ΒͷΫϥ΢υ ɾPHP 7.2 / Laravel 5.5 ɹɾNginx / MariaDB /Redis / Ansible ɹɾSendGrid/Redash 
 ɹɾ؂ࢹ/Mackerel ɹɾ΄ͱΜͲVPSతͳӡ༻ ɾ։ൃ؀ڥ ɹɾDocker ɾϦϦʔεํ๏ ɹɾSSHʹΑΔγΣϧεΫϦϓτ

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE 3೥લͷγεςϜߏ੒ 
 (ۤ࿑࿩)

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE 3೥લͷγεςϜߏ੒(ۤ࿑࿩) ɾσϓϩΠ͕͠೉͍ 
 ɹɾγΣϧεΫϦϓτ͕ͩɺ΄΅खಈͰϦϦʔε ɾΞΫηεෛՙʹΑΔϨΠςϯγʔͷѱ͞ 
 ɹɾϩʔυΞϕϨʔδ΍ϝϞϦʔ͕ΧπΧπ… ɾ৑௕ੑ΍ෛՙ෼ࢄͳͲ͸୲อ͍ͯ͠ͳ͍ঢ়ଶ ɹɾ1୆Ͱࡹ͘ͷ͸ͱʹ͔͘͠ΜͲ͍…౔೔͕ϐʔΫ ɾͳΔ΂͘ίετΛ཈͑ͨΠϯϑϥ؀ڥ 
 ɹɾStgͰςετ͍͚ͨ͠ͲεϖοΫ͕଍ΓͶ͑..!

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ։ൃ؀ڥΛ࡞Δ 
 ϙΠϯτ

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ։ൃ؀ڥΛ࡞ΔϙΠϯτ ɾຊ൪؀ڥͱಉ౳ͷߏ੒Λ࠶ݱ ɾELB(Nginx)ίϯςφ(ϦόʔεϓϩΩγ) ɾApp(PHP7.2-alpine/Nginx) ɾMySQL 5.7 ɾSendGrid/phpMyAdmin/Redis ɾStripe/Selenium/node/Digdag(Embulk) ɾWordPress(PHP7.3-alpine/Nginx) ɾ֤ΞϓϦέʔγϣϯͷىಈ͸SupervisorͰ؅ཧ ɾcomposer install͸AppίϯςφͰϩάΠϯ࣌ʹ࣮ߦ ɹɾsupervisorctlͰΞϓϦΛrestart͢ΔΑ͏ʹ 
 ɾM1 MacͰ΋ಈ࡞ݕূࡁΈ

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE Stg/ຊ൪؀ڥͷ 
 ߏ੒ʹ͍ͭͯ

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE Stg/ຊ൪؀ڥͷߏ੒ʹ͍ͭͯ TerraformͰશϦιʔείʔυԽ ɾStaging؀ڥ͸ۃྗίετΛ཈͑λεΫ਺͸1ͭ ɾࣾ಺༻͸ϥϯαʔζRProxyͰΞΫηε(phpMyAdmin,SendGrid,Digdag,Redash) ɾWordPressίϯςφʹΑΔϝσΟΞӡ༻ ɾphpMyAdminίϯςφʹΑΔσʔλग़͠ ɾΦʔτεέʔϧʹΑΔ࠷େ6୆ͷAppίϯςφͰӡ༻ ɾdevops/SSHίϯςφͷར༻Ͱaws cli.MySQLͷૢ࡞ ɹɾݱࡏ͸ഇࢭ͕ͨ͠EC2ຊ൪DBϚεΩϯά༻ͷ؀ڥͱͯ͠ར༻ ɾDigdag/Embulk/BigQueryʹΑΔ෼ੳج൫ͷߏங ɾDB͸RDS AuroraʹҠߦ ɾElastiCacheʹΑΔϩάΠϯΩϟογϡԽ ɾը૾౳S3Խ ɾϩά͸CloudWatch Logs ɾόον͸ECS Scheduled Tasks ɾ಺෦؂ࠪ(ϩάͷू໿) ɹɾAmazon Kinesis ɾAWS KMSʹΑΔ҉߸Խ ɾAWS WAFͰͷDDoSΞλοΫ੍ޚ ɾ؂ࢹ͸DatadogͰίϯςφϦιʔεͱAPMͰͷΤϥʔݕ஌ ɾຊ൪؀ڥ͸ΧφϦΞ؀ڥ΋

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE SSH/࡞ۀ༻ίϯςφ 
 ʮdevopsʯʹ͍ͭͯ

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE SSH/࡞ۀ༻ίϯςφʮdevopsʯʹ͍ͭͯ ɾAppʹ͸SSH͠ͳ͍ 
 ɾSSH/࡞ۀ༻ίϯςφ(Appͱಉ౳) ɹɾdevopsͱ໋໊ ɾίϚϯυͰRDS΁઀ଓ ɹɾdump΍restore ɾaws cliͷར༻ ɾSSMͷSession Manager ΋͋Δ͕ ɹɾखܰʹ౿Έ୆͔ΒSSHΛ͍ͨͨ͠Ί ɾVPC PeeringͰϥϯαʔζͷωοτϫʔΫ͔ΒSSHϩάΠϯ

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE SSH/࡞ۀ༻ίϯςφʮdevopsʯͷݱࡏ ɾdevopsίϯςφΛഇࢭ(2021/04~) ɾECS Execͷొ৔ ɾAppίϯςφʹ௚઀ϩάΠϯ ɾγΣϧεΫϦϓτͰ࣮૷ ɹɾݱࡏ͸ecspressoͰίϯςφ؅ཧ ↑ݸਓϒϩάΛνΣοΫ👍 $ ls /etc/ecspresso/menta/ README.md app.yaml digdag.yaml $ export AWS_PROFILE=menta $ ecspresso --config /etc/ecspresso/menta/app.yaml exec --command bash $ ecspresso --config /etc/ecspresso/menta/digdag.yaml exec --command bash

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE CircleCIʹΑΔ 
 ίϯςφσϓϩΠʹ͍ͭͯ

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE APIܦ༝ͰͷStg؀ڥͷσϓϩΠ ɾStg؀ڥͷσϓϩΠ ɹɾgithub flow ɹɾγΣϧεΫϦϓτ 
 ɹɹɾ೚ҙͷϒϥϯνΛࢦఆ ɾૉૣ͘σϓϩΠ͕Մೳ ɹɾ։ൃޮ཰͕Ξοϓ ɾMasterϚʔδͰຊ൪؀ڥϦϦʔε ɾTerraform CI؀ڥΛ࣮૷ $ sh deploy.sh ԼهͷΑ͏ʹϒϥϯνΛࢦఆ࣮ͯ͠ߦ͍ͯͩ͘͠͞ɻ (ྫ) sh deploy.sh ϒϥϯν໊ $ sh deploy.sh fix-nginx { "number" : 74, "state" : "pending", "id" : "xxxxxx-xxxxx-xxxxx-xxx-xxxxxxxx", "created_at" : “2021-06-27T06:14:43.973Z" } ϒϥϯν໊/ fix-nginx ͷStgσϓϩΠΛ։࢝͠·ͨ͠!

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE όονͷ࣮૷ʹ͍ͭͯ

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ECS Scheduled TasksʹΑΔόον࣮૷ʹ͍ͭͯ ɾࠓ·Ͱ 
 ɹɾLaravelͷλεΫεέδϡʔϧ ɾݱࡏ ɹɾECS Scheduled Tasks 
 ɹɹɾCloudWatchͷΠϕϯτϧʔϧ 
 ɹɹɾTerraformԽ { "containerOverrides": [ { "name": "menta-app", "command": [ "php", "artisan", "admin:coupon-mail:send" ] } ] }

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ϩάʗίϯςφ؂ࢹ 
 ʹ͍ͭͯ

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ϩά/ίϯςφ؂ࢹʹ͍ͭͯ ɾΞΫηε/ΞϓϦϩάɺRDS؂ࠪ/Τϥʔϩά 
 ɹɾCloudWatch Logsʹ ɹɾࠓ·Ͱ ɹɹɾLambdaͰS3ʹγϯΫɹɹ ɹɹɾPythonͰ࣮૷͍ͯͨͨ͠Ίӡ༻ίετΞοϓ ɹɾࠓճ ɹɹɾAmazon Kinesis Data FirehoseͰS3ʹू໿ ɹɹɾTerraformԽ 
 ɾ؂ࢹ ɹɾMackerel→DatadogʹҠߦ ɹɾλεΫఆٛʹdatadog-agentΛಈ࡞ͤ͞ΔΑ͏ʹ 
 ɹɾcpu:10ɺmemory:256 TerraformԽ ɹɾAPMͰͷΞϓϦΤϥʔݕ஌

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ຊ൪DBϚεΩϯά 
 ࣗಈԽ

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ຊ൪DBϚεΩϯάࣗಈԽ 
 ɾStg؀ڥͷDB ɹɾຖि1ճຊ൪DBͱಉ͡σʔλΛೖΕͯ΄͍͠ʂ ɹɾݸਓ৘ใ͸΋ͪΖΜϚεΩϯά ɾdevops(EC2)ͷ෮׆ ɾECS/FargateͰ΍Ζ͏ͱࢥͬͨ ɹɾσΟεΫྔ͕গͳ͍ͨΊஅ೦ ɾauto-masking ɹɾprd-masking.sh ຖि౔༵೔0࣌ʙ ɹɹɾDump ɹɾstg-masking-restore.sh ຖि౔༵೔5࣌ʙ ɹɹɾRestore ɾpost_slack.sh ݁ՌΛSlack௨஌ ɾ։ൃ؀ڥ(MySQLίϯςφ) ɾauto-renew-mysql-container ຖि౔༵೔9࣌ʙ ɾCodeBuild ɹɾMySQLίϯςφΛbuild͠ϚεΩϯάσʔλΛRestore

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ECSͰॳͷ 
 Φʔτεέʔϧ࠾༻

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ECSॳͷΦʔτεέʔϧ࠾༻ ɾECS/FargateͰॳͷΦʔτεέʔϧ࠾༻ ɾTerraformԽ ɹɾCloudWatch Alarmઃఆ(CPU 40%ͰλεΫ૿Ճ) ɹɾECS/Service AutoScaling ઃఆ (λεΫͷ࠷େ਺4) ɹɾAuto Scaling Policyઃఆ ɾEC2ͷΑ͏ʹAMIͳͲͷ࡞ΓࠐΈ͕Ұ੾ͳ͍ˠָ

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ෼ੳج൫ͷ঺հ 
 (Digdag/Embulk/ 
 BigQuery/Redash)

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ෼ੳج൫ͷ঺հ ɾLancersͷ෼ੳج൫ΛԠ༻ ɾ෼ੳج൫ ɹɾDigdag/Embulk/Redash/BigQuery 
 ɹɾຖ೔ਂ໷ʹRDSͷσʔλΛBigQueryʹγϯΫ ɹɾLancersͱಉ͡ϓϩδΣΫτͰ؅ཧ 
 ɾઌ݄·ͰEC2(Digdagαʔόʔ)Ͱӡ༻͍ͯͨ͠ ɹɾECS/FargateʹҠߦ(։ൃ؀ڥ΋ߏங) 
 ɹɾຊՈϥϯαʔζ΋Ҡߦ ɾৄࡉ͸ϥϯαʔζΤϯδχΞϒϩάͰ👍

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE Ҡߦ࣌ͷ՝୊

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE Ҡߦ࣌ͷ՝୊ͱΑ͔ͬͨ͜ͱ ɾϩάΛίϯςφ಺Ͱ؅ཧͯ͠͠·͏ͱσϓϩΠ͢Δͨͼফ͑ͯ͠·͏ ɹɾCloudWatch Logsʹඪ४ग़ྗ͠ͳ͚Ε͹ͳΒͳ͍ ɹɾLaravelΞϓϦϩάͷstdout͕5.8Ҏ্͡Όͳ͍ͱରԠͰ͖ͳ͍ ɹɾόʔδϣϯΞοϓΛߦͳͬͨ ɾը૾ܥ ɹɾS3 ը૾Ξοϓϩʔυ͕Ұ෦ͷػೳ͔͠ରԠ͍ͯ͠ͳ͔ͬͨ 
 ɹɹɾ1ϲ݄Ҏ্͔͚ͯҠߦ ɾόον΍աڈͷاըͷιʔεΛ୨Է ɹɾෛ࠴ͱͳ͍ͬͯͨιʔεΛ੔ཧͰ͖ͨ ɾUnit/Feature/E2Eͷࣗಈςετ੔උ ɹɾPHPCSΛ࢖ͬͨن໿౷Ұ ɹɾPHPStanʢLarastanʣ࢖ͬͨ੩తղੳͰͷ඼࣭޲্όά࡟ݮ ɹɾϦϑΝΫλϦϯά 💦

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE Ҡߦޙͷτϥϒϧ

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE Ҡߦޙͷτϥϒϧ ɾECS Scheduled Tasks ɹɾຖ෼ͷόονͰNAT Gatewayͷ௨৴අ͕ංେԽ 
 ɹɾECR΍CloudWatch Logsͷ޲͖ઌΛมߋ 
 ɹɹɾVPCΤϯυϙΠϯτܦ༝ʹ 
 ɾଞͷάϧʔϓձࣾ΋10ສԁίετΞοϓ͍ͯͨ͠ 
 ɹɾ೥ؒ120ສ࡟ݮʹվળͨ͠ ɾݸਓϒϩάॻ͍͍ͯΔͷͰࢀߟʹʂ 💦

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ·ͱΊ

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ɾࠓޙ ɹɾ։ൃ؀ڥͷDB΋ϚεΩϯά͞ΕͨDBΛར༻ 
 ɹɹɾcodebuild ɹɾPHP/LaravelͷόʔδϣϯΞοϓ ɹɾίϯςφͷ੬ऑੑݕ஌ (Trivy) ɹɾDatadogͰΞΫηεϩά࿈ܞ ɹɾϥϯαʔζຊՈ΋ίϯςφҠߦʹνϟϨϯδ 
 ɹɹɾࣾ಺Ͱӡ༻͍ͯ͠Δ΋ͷ͸ҠߦࡁΈ ·ͱΊ ɾ൒೥ͰҠߦ͢Δ͜ͱ͕Ͱ͖ͨ(2020/10~2021/03) 
 ɹɾٕज़తͳෛ࠴ͷ੔ཧΛߦ͍ҠߦΛ࣮ࢪͰ͖ͨ ɾاըɾΞΠσΟΞϕʔεͷϓϩτλΠϓ։ൃ͔Β 
 ɹɾকདྷΛݟਾ͑ӡ༻ɾอक΋ҙࣝͨ͠։ൃελΠϧʹ 
 ɹɾϥϯαʔζͱಉ͡։ൃελΠϧʹมΘͬͨ ɾImmutableͳαʔόʔͷӡ༻ ɹɾ҆ఆӡ༻ɺӡ༻ޮ཰ͷΞοϓ ɹɾ৑௕Խ΍ΦʔτεέʔϦϯάʹΑΔߴ͍Մ༻ੑΛ࣮ݱͨ͠ߏ੒ ɾDatadogͷAPMͰΞϓϦͷΤϥʔ௨஌͕Ͱ͖ΔΑ͏ʹ ɾϗετΠϯελϯεͷ؅ཧ͕ল͚Δ ɾλεΫ୯ҐͰࣗ༝ʹϦιʔεมߋՄೳ 
 ɾλεΫͷىಈ଎౓͕஗Ί ɹɾόον࣮ߦ΋਺෼஗͘ͳΔ 
 ɾ͘͞ΒͷΫϥ΢υ࣌୅ֹ݄Ͱ਺ेສԁˢ 


2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ΤϯδχΞϒϩάͰ΋ެ։ͯ͠·͢ https://engineer.blog.lancers.jp/2021/03/replace-menta/

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE SREٻΉʂʂ SRE͕଍Γ·ͤΜ😇

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ࠂ஌ʂ https://lancersrecruit.connpass.com/event/221994/

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE