https://lancersrecruit.connpass.com/event/219434/
【SPACEMARKET×Lancers】シェアリングエコノミーを支えるインフラ/SREでのスライドとなります。
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREMENTAΛAWSʹҠߦͯ͠ৼΓฦΔ(ECS/Fargate + Laravelฤ)ϥϯαʔζגࣜձࣾ SRE/҆ୡ ྋ
View Slide
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREΞδΣϯμ ɾࣗݾհɾ3લͷγεςϜߏ/͘͞ΒͷΫϥυʹ͍ͭͯ(ۤ࿑)ɾ։ൃڥΛ࡞ΔϙΠϯτɾStg/ຊ൪ڥͷߏʹ͍ͭͯɾCircleCIʹΑΔίϯςφσϓϩΠɾECS Scheduled TasksʹΑΔόον࣮ʹ͍ͭͯɾϩά/ίϯςφࢹʹ͍ͭͯɾຊ൪DBϚεΩϯάࣗಈԽʹ͍ͭͯɾECSΦʔτεέʔϧʹ͍ͭͯɾੳج൫ͷհ(Digdag,Embulk,BigQuery,Redash)ɾҠߦલͷ՝ͱҠߦޙͷτϥϒϧɾ·ͱΊ
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREࣗݾհ
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREࣗݾհ ɾ҆ୡ ྋ(adachin)ɾLancers SRE 2018/5~ɾ@adachin0817ɾRVIRUS0817ɾblog.adachin.meɾwiki.adachin.meɾݸਓ։ൃ(famii)ɾOSS
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREͪͳΈʹ…
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREͪͳΈʹ… MENTAೖΕΔͱ3ͭίϯςφʹҠߦ͠·ͨ͠ʂPROSheet Lancers AgentLancers Assistant Creative
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREͪͳΈʹMENTAͰ..?
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREͪͳΈʹMENTAͰ..? ɾΠϯϑϥ/αʔόʔ ίʔνϯάɾMENTAϥϯΩϯάɹɾ4Ґʂɾྦྷܭ184ਓʂ ɹɾۦ͚ग़͠ΤϯδχΞϑΥϩʔɹɾస৬࣮ଟ༗Γ
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE3લͷγεςϜߏ
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE3લͷγεςϜߏ ɾ2018ϦϦʔεɹɾ෭ۀͰ͕ࣗαʔόʔอक͍ͯͨ͠ ɾ͘͞ΒͷΫϥυɾPHP 7.2 / Laravel 5.5ɹɾNginx / MariaDB /Redis / AnsibleɹɾSendGrid/Redash ɹɾࢹ/Mackerelɹɾ΄ͱΜͲVPSతͳӡ༻ɾ։ൃڥɹɾDockerɾϦϦʔεํ๏ɹɾSSHʹΑΔγΣϧεΫϦϓτ
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE3લͷγεςϜߏ (ۤ࿑)
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE3લͷγεςϜߏ(ۤ࿑) ɾσϓϩΠ͕͍͠ ɹɾγΣϧεΫϦϓτ͕ͩɺ΄΅खಈͰϦϦʔεɾΞΫηεෛՙʹΑΔϨΠςϯγʔͷѱ͞ ɹɾϩʔυΞϕϨʔδϝϞϦʔ͕ΧπΧπ…ɾੑෛՙࢄͳͲ୲อ͍ͯ͠ͳ͍ঢ়ଶɹɾ1Ͱࡹ͘ͷͱʹ͔͘͠ΜͲ͍…͕ϐʔΫɾͳΔ͘ίετΛ͑ͨΠϯϑϥڥ ɹɾStgͰςετ͍͚ͨ͠ͲεϖοΫ͕ΓͶ͑..!
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE։ൃڥΛ࡞Δ ϙΠϯτ
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE։ൃڥΛ࡞ΔϙΠϯτ ɾຊ൪ڥͱಉͷߏΛ࠶ݱɾELB(Nginx)ίϯςφ(ϦόʔεϓϩΩγ)ɾApp(PHP7.2-alpine/Nginx)ɾMySQL 5.7ɾSendGrid/phpMyAdmin/RedisɾStripe/Selenium/node/Digdag(Embulk)ɾWordPress(PHP7.3-alpine/Nginx)ɾ֤ΞϓϦέʔγϣϯͷىಈSupervisorͰཧɾcomposer installAppίϯςφͰϩάΠϯ࣌ʹ࣮ߦɹɾsupervisorctlͰΞϓϦΛrestart͢ΔΑ͏ʹ ɾM1 MacͰಈ࡞ݕূࡁΈ
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREStg/ຊ൪ڥͷ ߏʹ͍ͭͯ
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREStg/ຊ൪ڥͷߏʹ͍ͭͯ TerraformͰશϦιʔείʔυԽɾStagingڥۃྗίετΛ͑λεΫ1ͭɾࣾ༻ϥϯαʔζRProxyͰΞΫηε(phpMyAdmin,SendGrid,Digdag,Redash)ɾWordPressίϯςφʹΑΔϝσΟΞӡ༻ɾphpMyAdminίϯςφʹΑΔσʔλग़͠ɾΦʔτεέʔϧʹΑΔ࠷େ6ͷAppίϯςφͰӡ༻ɾdevops/SSHίϯςφͷར༻Ͱaws cli.MySQLͷૢ࡞ɹɾݱࡏഇࢭ͕ͨ͠EC2ຊ൪DBϚεΩϯά༻ͷڥͱͯ͠ར༻ɾDigdag/Embulk/BigQueryʹΑΔੳج൫ͷߏஙɾDBRDS AuroraʹҠߦɾElastiCacheʹΑΔϩάΠϯΩϟογϡԽɾը૾S3ԽɾϩάCloudWatch LogsɾόονECS Scheduled Tasksɾ෦ࠪ(ϩάͷू)ɹɾAmazon KinesisɾAWS KMSʹΑΔ҉߸ԽɾAWS WAFͰͷDDoSΞλοΫ੍ޚɾࢹDatadogͰίϯςφϦιʔεͱAPMͰͷΤϥʔݕɾຊ൪ڥΧφϦΞڥ
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRESSH/࡞ۀ༻ίϯςφ ʮdevopsʯʹ͍ͭͯ
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRESSH/࡞ۀ༻ίϯςφʮdevopsʯʹ͍ͭͯɾAppʹSSH͠ͳ͍ ɾSSH/࡞ۀ༻ίϯςφ(Appͱಉ)ɹɾdevopsͱ໋໊ɾίϚϯυͰRDSଓɹɾdumprestoreɾaws cliͷར༻ɾSSMͷSession Manager ͋Δ͕ɹɾखܰʹ౿Έ͔ΒSSHΛ͍ͨͨ͠ΊɾVPC PeeringͰϥϯαʔζͷωοτϫʔΫ͔ΒSSHϩάΠϯ
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRESSH/࡞ۀ༻ίϯςφʮdevopsʯͷݱࡏɾdevopsίϯςφΛഇࢭ(2021/04~)ɾECS ExecͷొɾAppίϯςφʹϩάΠϯɾγΣϧεΫϦϓτͰ࣮ɹɾݱࡏecspressoͰίϯςφཧ ↑ݸਓϒϩάΛνΣοΫ👍$ ls /etc/ecspresso/menta/README.md app.yaml digdag.yaml$ export AWS_PROFILE=menta$ ecspresso --config /etc/ecspresso/menta/app.yaml exec --command bash$ ecspresso --config /etc/ecspresso/menta/digdag.yaml exec --command bash
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRECircleCIʹΑΔ ίϯςφσϓϩΠʹ͍ͭͯ
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREAPIܦ༝ͰͷStgڥͷσϓϩΠ ɾStgڥͷσϓϩΠɹɾgithub flowɹɾγΣϧεΫϦϓτ ɹɹɾҙͷϒϥϯνΛࢦఆɾૉૣ͘σϓϩΠ͕Մೳɹɾ։ൃޮ͕ΞοϓɾMasterϚʔδͰຊ൪ڥϦϦʔεɾTerraform CIڥΛ࣮$ sh deploy.shԼهͷΑ͏ʹϒϥϯνΛࢦఆ࣮ͯ͠ߦ͍ͯͩ͘͠͞ɻ(ྫ) sh deploy.sh ϒϥϯν໊$ sh deploy.sh fix-nginx{"number" : 74,"state" : "pending","id" : "xxxxxx-xxxxx-xxxxx-xxx-xxxxxxxx","created_at" : “2021-06-27T06:14:43.973Z"}ϒϥϯν໊/ fix-nginx ͷStgσϓϩΠΛ։࢝͠·ͨ͠!
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREόονͷ࣮ʹ͍ͭͯ
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREECS Scheduled TasksʹΑΔόον࣮ʹ͍ͭͯɾࠓ·Ͱ ɹɾLaravelͷλεΫεέδϡʔϧɾݱࡏɹɾECS Scheduled Tasks ɹɹɾCloudWatchͷΠϕϯτϧʔϧ ɹɹɾTerraformԽ{"containerOverrides": [{"name": "menta-app","command": ["php","artisan","admin:coupon-mail:send"]}]}
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREϩάʗίϯςφࢹ ʹ͍ͭͯ
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREϩά/ίϯςφࢹʹ͍ͭͯɾΞΫηε/ΞϓϦϩάɺRDSࠪ/Τϥʔϩά ɹɾCloudWatch Logsʹɹɾࠓ·ͰɹɹɾLambdaͰS3ʹγϯΫɹɹɹɹɾPythonͰ࣮͍ͯͨͨ͠Ίӡ༻ίετΞοϓɹɾࠓճɹɹɾAmazon Kinesis Data FirehoseͰS3ʹूɹɹɾTerraformԽ ɾࢹɹɾMackerel→DatadogʹҠߦɹɾλεΫఆٛʹdatadog-agentΛಈ࡞ͤ͞ΔΑ͏ʹ ɹɾcpu:10ɺmemory:256 TerraformԽɹɾAPMͰͷΞϓϦΤϥʔݕ
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREຊ൪DBϚεΩϯά ࣗಈԽ
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREຊ൪DBϚεΩϯάࣗಈԽ ɾStgڥͷDBɹɾຖि1ճຊ൪DBͱಉ͡σʔλΛೖΕͯ΄͍͠ʂɹɾݸਓใͪΖΜϚεΩϯάɾdevops(EC2)ͷ෮׆ɾECS/FargateͰΖ͏ͱࢥͬͨɹɾσΟεΫྔ͕গͳ͍ͨΊஅ೦ɾauto-maskingɹɾprd-masking.sh ຖि༵0࣌ʙɹɹɾDumpɹɾstg-masking-restore.sh ຖि༵5࣌ʙɹɹɾRestoreɾpost_slack.sh ݁ՌΛSlack௨ɾ։ൃڥ(MySQLίϯςφ)ɾauto-renew-mysql-container ຖि༵9࣌ʙɾCodeBuildɹɾMySQLίϯςφΛbuild͠ϚεΩϯάσʔλΛRestore
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREECSͰॳͷ Φʔτεέʔϧ࠾༻
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREECSॳͷΦʔτεέʔϧ࠾༻ɾECS/FargateͰॳͷΦʔτεέʔϧ࠾༻ɾTerraformԽɹɾCloudWatch Alarmઃఆ(CPU 40%ͰλεΫ૿Ճ)ɹɾECS/Service AutoScaling ઃఆ (λεΫͷ࠷େ4)ɹɾAuto Scaling PolicyઃఆɾEC2ͷΑ͏ʹAMIͳͲͷ࡞ΓࠐΈ͕Ұͳ͍ˠָ
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREੳج൫ͷհ (Digdag/Embulk/ BigQuery/Redash)
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREੳج൫ͷհɾLancersͷੳج൫ΛԠ༻ɾੳج൫ɹɾDigdag/Embulk/Redash/BigQuery ɹɾຖਂʹRDSͷσʔλΛBigQueryʹγϯΫɹɾLancersͱಉ͡ϓϩδΣΫτͰཧ ɾઌ݄·ͰEC2(Digdagαʔόʔ)Ͱӡ༻͍ͯͨ͠ɹɾECS/FargateʹҠߦ(։ൃڥߏங) ɹɾຊՈϥϯαʔζҠߦɾৄࡉϥϯαʔζΤϯδχΞϒϩάͰ👍
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREҠߦ࣌ͷ՝
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREҠߦ࣌ͷ՝ͱΑ͔ͬͨ͜ͱ ɾϩάΛίϯςφͰཧͯ͠͠·͏ͱσϓϩΠ͢Δͨͼফ͑ͯ͠·͏ɹɾCloudWatch Logsʹඪ४ग़ྗ͠ͳ͚ΕͳΒͳ͍ɹɾLaravelΞϓϦϩάͷstdout͕5.8Ҏ্͡Όͳ͍ͱରԠͰ͖ͳ͍ɹɾόʔδϣϯΞοϓΛߦͳͬͨɾը૾ܥɹɾS3 ը૾Ξοϓϩʔυ͕Ұ෦ͷػೳ͔͠ରԠ͍ͯ͠ͳ͔ͬͨ ɹɹɾ1ϲ݄Ҏ্͔͚ͯҠߦɾόονաڈͷاըͷιʔεΛ୨Էɹɾෛ࠴ͱͳ͍ͬͯͨιʔεΛཧͰ͖ͨɾUnit/Feature/E2EͷࣗಈςετඋɹɾPHPCSΛͬͨن౷ҰɹɾPHPStanʢLarastanʣͬͨ੩తղੳͰͷ্࣭όάݮɹɾϦϑΝΫλϦϯά💦
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREҠߦޙͷτϥϒϧ
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREҠߦޙͷτϥϒϧɾECS Scheduled TasksɹɾຖͷόονͰNAT Gatewayͷ௨৴අ͕ංେԽ ɹɾECRCloudWatch Logsͷ͖ઌΛมߋ ɹɹɾVPCΤϯυϙΠϯτܦ༝ʹ ɾଞͷάϧʔϓձࣾ10ສԁίετΞοϓ͍ͯͨ͠ ɹɾؒ120ສݮʹվળͨ͠ɾݸਓϒϩάॻ͍͍ͯΔͷͰࢀߟʹʂ💦
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE·ͱΊ
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREɾࠓޙɹɾ։ൃڥͷDBϚεΩϯά͞ΕͨDBΛར༻ ɹɹɾcodebuildɹɾPHP/LaravelͷόʔδϣϯΞοϓɹɾίϯςφͷ੬ऑੑݕ (Trivy)ɹɾDatadogͰΞΫηεϩά࿈ܞɹɾϥϯαʔζຊՈίϯςφҠߦʹνϟϨϯδ ɹɹɾࣾͰӡ༻͍ͯ͠ΔͷҠߦࡁΈ·ͱΊɾͰҠߦ͢Δ͜ͱ͕Ͱ͖ͨ(2020/10~2021/03) ɹɾٕज़తͳෛ࠴ͷཧΛߦ͍ҠߦΛ࣮ࢪͰ͖ͨɾاըɾΞΠσΟΞϕʔεͷϓϩτλΠϓ։ൃ͔Β ɹɾকདྷΛݟਾ͑ӡ༻ɾอकҙࣝͨ͠։ൃελΠϧʹ ɹɾϥϯαʔζͱಉ͡։ൃελΠϧʹมΘͬͨɾImmutableͳαʔόʔͷӡ༻ɹɾ҆ఆӡ༻ɺӡ༻ޮͷΞοϓɹɾԽΦʔτεέʔϦϯάʹΑΔߴ͍Մ༻ੑΛ࣮ݱͨ͠ߏɾDatadogͷAPMͰΞϓϦͷΤϥʔ௨͕Ͱ͖ΔΑ͏ʹɾϗετΠϯελϯεͷཧ͕ল͚ΔɾλεΫ୯ҐͰࣗ༝ʹϦιʔεมߋՄೳ ɾλεΫͷىಈ͕Ίɹɾόον࣮ߦ͘ͳΔ ɾ͘͞ΒͷΫϥυֹ݄࣌Ͱेສԁˢ
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREΤϯδχΞϒϩάͰެ։ͯ͠·͢https://engineer.blog.lancers.jp/2021/03/replace-menta/
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRESREٻΉʂʂSRE͕Γ·ͤΜ😇
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SREࠂʂ https://lancersrecruit.connpass.com/event/221994/
2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE