MENTAをAWSに移行して振り返る(ECS/Fargate + Laravel編)

https://lancersrecruit.connpass.com/event/219434/

【SPACEMARKET×Lancers】シェアリングエコノミーを支えるインフラ/SREでのスライドとなります。

Transcript

1. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE MENTAΛAWSʹҠߦͯ͠ৼΓฦΔ (ECS/Fargate + Laravelฤ) ϥϯαʔζגࣜձࣾ 
 SRE/҆ୡ ྋ
2. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ΞδΣϯμ  ɾࣗݾ঺հ ɾ3೥લͷγεςϜߏ੒/͘͞ΒͷΫϥ΢υʹ͍ͭͯ(ۤ࿑࿩) ɾ։ൃ؀ڥΛ࡞ΔϙΠϯτ ɾStg/ຊ൪؀ڥͷߏ੒ʹ͍ͭͯ ɾCircleCIʹΑΔίϯςφσϓϩΠ ɾECS Scheduled TasksʹΑΔόον࣮૷ʹ͍ͭͯ ɾϩά/ίϯςφ؂ࢹʹ͍ͭͯ ɾຊ൪DBϚεΩϯάࣗಈԽʹ͍ͭͯ ɾECSΦʔτεέʔϧʹ͍ͭͯ ɾ෼ੳج൫ͷ঺հ(Digdag,Embulk,BigQuery,Redash) ɾҠߦલͷ՝୊ͱҠߦޙͷτϥϒϧ ɾ·ͱΊ
3. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ࣗݾ঺հ
4. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ࣗݾ঺հ  ɾ҆ୡ ྋ(adachin) ɾLancers SRE 2018/5~ ɾ@adachin0817 ɾRVIRUS0817 ɾblog.adachin.me ɾwiki.adachin.me ɾݸਓ։ൃ(famii) ɾOSS
5. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ͪͳΈʹ…
6. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ͪͳΈʹ…  MENTAೖΕΔͱ3ͭίϯςφʹҠߦ͠·ͨ͠ʂ PROSheet 
 Lancers Agent Lancers Assistant 
 Creative
7. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ͪͳΈʹMENTAͰ͸..?
8. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ͪͳΈʹMENTAͰ͸..?  ɾΠϯϑϥ/αʔόʔ ίʔνϯά ɾMENTAϥϯΩϯά ɹɾ4Ґʂ ɾྦྷܭ184ਓʂ 
 ɹɾۦ͚ग़͠ΤϯδχΞϑΥϩʔ ɹɾస৬࣮੷ଟ਺༗Γ
9. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE 3೥લͷγεςϜߏ੒
10. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE 3೥લͷγεςϜߏ੒  ɾ2018೥ϦϦʔεɹ ɾ෭ۀͰࣗ෼͕αʔόʔอक͍ͯͨ͠ 
 ɾ͘͞ΒͷΫϥ΢υ ɾPHP 7.2 / Laravel 5.5 ɹɾNginx / MariaDB /Redis / Ansible ɹɾSendGrid/Redash 
 ɹɾ؂ࢹ/Mackerel ɹɾ΄ͱΜͲVPSతͳӡ༻ ɾ։ൃ؀ڥ ɹɾDocker ɾϦϦʔεํ๏ ɹɾSSHʹΑΔγΣϧεΫϦϓτ
11. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE 3೥લͷγεςϜߏ੒ 
 (ۤ࿑࿩)
12. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE 3೥લͷγεςϜߏ੒(ۤ࿑࿩)  ɾσϓϩΠ͕͠೉͍ 
 ɹɾγΣϧεΫϦϓτ͕ͩɺ΄΅खಈͰϦϦʔε ɾΞΫηεෛՙʹΑΔϨΠςϯγʔͷѱ͞ 
 ɹɾϩʔυΞϕϨʔδ΍ϝϞϦʔ͕ΧπΧπ… ɾ৑௕ੑ΍ෛՙ෼ࢄͳͲ͸୲อ͍ͯ͠ͳ͍ঢ়ଶ ɹɾ1୆Ͱࡹ͘ͷ͸ͱʹ͔͘͠ΜͲ͍…౔೔͕ϐʔΫ ɾͳΔ΂͘ίετΛ཈͑ͨΠϯϑϥ؀ڥ 
 ɹɾStgͰςετ͍͚ͨ͠ͲεϖοΫ͕଍ΓͶ͑..!
13. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ։ൃ؀ڥΛ࡞Δ 
 ϙΠϯτ
14. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ։ൃ؀ڥΛ࡞ΔϙΠϯτ  ɾຊ൪؀ڥͱಉ౳ͷߏ੒Λ࠶ݱ ɾELB(Nginx)ίϯςφ(ϦόʔεϓϩΩγ) ɾApp(PHP7.2-alpine/Nginx) ɾMySQL 5.7 ɾSendGrid/phpMyAdmin/Redis ɾStripe/Selenium/node/Digdag(Embulk) ɾWordPress(PHP7.3-alpine/Nginx) ɾ֤ΞϓϦέʔγϣϯͷىಈ͸SupervisorͰ؅ཧ ɾcomposer install͸AppίϯςφͰϩάΠϯ࣌ʹ࣮ߦ ɹɾsupervisorctlͰΞϓϦΛrestart͢ΔΑ͏ʹ 
 ɾM1 MacͰ΋ಈ࡞ݕূࡁΈ
15. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE Stg/ຊ൪؀ڥͷ 
 ߏ੒ʹ͍ͭͯ
16. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE Stg/ຊ൪؀ڥͷߏ੒ʹ͍ͭͯ  TerraformͰશϦιʔείʔυԽ ɾStaging؀ڥ͸ۃྗίετΛ཈͑λεΫ਺͸1ͭ ɾࣾ಺༻͸ϥϯαʔζRProxyͰΞΫηε(phpMyAdmin,SendGrid,Digdag,Redash) ɾWordPressίϯςφʹΑΔϝσΟΞӡ༻ ɾphpMyAdminίϯςφʹΑΔσʔλग़͠ ɾΦʔτεέʔϧʹΑΔ࠷େ6୆ͷAppίϯςφͰӡ༻ ɾdevops/SSHίϯςφͷར༻Ͱaws cli.MySQLͷૢ࡞ ɹɾݱࡏ͸ഇࢭ͕ͨ͠EC2ຊ൪DBϚεΩϯά༻ͷ؀ڥͱͯ͠ར༻ ɾDigdag/Embulk/BigQueryʹΑΔ෼ੳج൫ͷߏங ɾDB͸RDS AuroraʹҠߦ ɾElastiCacheʹΑΔϩάΠϯΩϟογϡԽ ɾը૾౳S3Խ ɾϩά͸CloudWatch Logs ɾόον͸ECS Scheduled Tasks ɾ಺෦؂ࠪ(ϩάͷू໿) ɹɾAmazon Kinesis ɾAWS KMSʹΑΔ҉߸Խ ɾAWS WAFͰͷDDoSΞλοΫ੍ޚ ɾ؂ࢹ͸DatadogͰίϯςφϦιʔεͱAPMͰͷΤϥʔݕ஌ ɾຊ൪؀ڥ͸ΧφϦΞ؀ڥ΋
17. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE SSH/࡞ۀ༻ίϯςφ 
 ʮdevopsʯʹ͍ͭͯ
18. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE SSH/࡞ۀ༻ίϯςφʮdevopsʯʹ͍ͭͯ  ɾAppʹ͸SSH͠ͳ͍ 
 ɾSSH/࡞ۀ༻ίϯςφ(Appͱಉ౳) ɹɾdevopsͱ໋໊ ɾίϚϯυͰRDS΁઀ଓ ɹɾdump΍restore ɾaws cliͷར༻ ɾSSMͷSession Manager ΋͋Δ͕ ɹɾखܰʹ౿Έ୆͔ΒSSHΛ͍ͨͨ͠Ί ɾVPC PeeringͰϥϯαʔζͷωοτϫʔΫ͔ΒSSHϩάΠϯ
19. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE SSH/࡞ۀ༻ίϯςφʮdevopsʯͷݱࡏ  ɾdevopsίϯςφΛഇࢭ(2021/04~) ɾECS Execͷొ৔ ɾAppίϯςφʹ௚઀ϩάΠϯ ɾγΣϧεΫϦϓτͰ࣮૷ ɹɾݱࡏ͸ecspressoͰίϯςφ؅ཧ ↑ݸਓϒϩάΛνΣοΫ👍 $ ls /etc/ecspresso/menta/ README.md app.yaml digdag.yaml $ export AWS_PROFILE=menta $ ecspresso --config /etc/ecspresso/menta/app.yaml exec --command bash $ ecspresso --config /etc/ecspresso/menta/digdag.yaml exec --command bash
20. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE CircleCIʹΑΔ 
 ίϯςφσϓϩΠʹ͍ͭͯ
21. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE APIܦ༝ͰͷStg؀ڥͷσϓϩΠ  ɾStg؀ڥͷσϓϩΠ ɹɾgithub flow ɹɾγΣϧεΫϦϓτ 
 ɹɹɾ೚ҙͷϒϥϯνΛࢦఆ ɾૉૣ͘σϓϩΠ͕Մೳ ɹɾ։ൃޮ཰͕Ξοϓ ɾMasterϚʔδͰຊ൪؀ڥϦϦʔε ɾTerraform CI؀ڥΛ࣮૷ $ sh deploy.sh ԼهͷΑ͏ʹϒϥϯνΛࢦఆ࣮ͯ͠ߦ͍ͯͩ͘͠͞ɻ (ྫ) sh deploy.sh ϒϥϯν໊ $ sh deploy.sh fix-nginx { "number" : 74, "state" : "pending", "id" : "xxxxxx-xxxxx-xxxxx-xxx-xxxxxxxx", "created_at" : “2021-06-27T06:14:43.973Z" } ϒϥϯν໊/ fix-nginx ͷStgσϓϩΠΛ։࢝͠·ͨ͠!
22. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE όονͷ࣮૷ʹ͍ͭͯ
23. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ECS Scheduled TasksʹΑΔόον࣮૷ʹ͍ͭͯ  ɾࠓ·Ͱ 
 ɹɾLaravelͷλεΫεέδϡʔϧ ɾݱࡏ ɹɾECS Scheduled Tasks 
 ɹɹɾCloudWatchͷΠϕϯτϧʔϧ 
 ɹɹɾTerraformԽ { "containerOverrides": [ { "name": "menta-app", "command": [ "php", "artisan", "admin:coupon-mail:send" ] } ] }
24. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ϩάʗίϯςφ؂ࢹ 
 ʹ͍ͭͯ
25. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ϩά/ίϯςφ؂ࢹʹ͍ͭͯ  ɾΞΫηε/ΞϓϦϩάɺRDS؂ࠪ/Τϥʔϩά 
 ɹɾCloudWatch Logsʹ ɹɾࠓ·Ͱ ɹɹɾLambdaͰS3ʹγϯΫɹɹ ɹɹɾPythonͰ࣮૷͍ͯͨͨ͠Ίӡ༻ίετΞοϓ ɹɾࠓճ ɹɹɾAmazon Kinesis Data FirehoseͰS3ʹू໿ ɹɹɾTerraformԽ 
 ɾ؂ࢹ ɹɾMackerel→DatadogʹҠߦ ɹɾλεΫఆٛʹdatadog-agentΛಈ࡞ͤ͞ΔΑ͏ʹ 
 ɹɾcpu:10ɺmemory:256 TerraformԽ ɹɾAPMͰͷΞϓϦΤϥʔݕ஌
26. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ຊ൪DBϚεΩϯά 
 ࣗಈԽ
27. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ຊ൪DBϚεΩϯάࣗಈԽ 
  ɾStg؀ڥͷDB ɹɾຖि1ճຊ൪DBͱಉ͡σʔλΛೖΕͯ΄͍͠ʂ ɹɾݸਓ৘ใ͸΋ͪΖΜϚεΩϯά ɾdevops(EC2)ͷ෮׆ ɾECS/FargateͰ΍Ζ͏ͱࢥͬͨ ɹɾσΟεΫྔ͕গͳ͍ͨΊஅ೦ ɾauto-masking ɹɾprd-masking.sh ຖि౔༵೔0࣌ʙ ɹɹɾDump ɹɾstg-masking-restore.sh ຖि౔༵೔5࣌ʙ ɹɹɾRestore ɾpost_slack.sh ݁ՌΛSlack௨஌ ɾ։ൃ؀ڥ(MySQLίϯςφ) ɾauto-renew-mysql-container ຖि౔༵೔9࣌ʙ ɾCodeBuild ɹɾMySQLίϯςφΛbuild͠ϚεΩϯάσʔλΛRestore
28. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ECSͰॳͷ 
 Φʔτεέʔϧ࠾༻
29. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ECSॳͷΦʔτεέʔϧ࠾༻  ɾECS/FargateͰॳͷΦʔτεέʔϧ࠾༻ ɾTerraformԽ ɹɾCloudWatch Alarmઃఆ(CPU 40%ͰλεΫ૿Ճ) ɹɾECS/Service AutoScaling ઃఆ (λεΫͷ࠷େ਺4) ɹɾAuto Scaling Policyઃఆ ɾEC2ͷΑ͏ʹAMIͳͲͷ࡞ΓࠐΈ͕Ұ੾ͳ͍ˠָ
30. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ෼ੳج൫ͷ঺հ 
 (Digdag/Embulk/ 
 BigQuery/Redash)
31. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ෼ੳج൫ͷ঺հ  ɾLancersͷ෼ੳج൫ΛԠ༻ ɾ෼ੳج൫ ɹɾDigdag/Embulk/Redash/BigQuery 
 ɹɾຖ೔ਂ໷ʹRDSͷσʔλΛBigQueryʹγϯΫ ɹɾLancersͱಉ͡ϓϩδΣΫτͰ؅ཧ 
 ɾઌ݄·ͰEC2(Digdagαʔόʔ)Ͱӡ༻͍ͯͨ͠ ɹɾECS/FargateʹҠߦ(։ൃ؀ڥ΋ߏங) 
 ɹɾຊՈϥϯαʔζ΋Ҡߦ ɾৄࡉ͸ϥϯαʔζΤϯδχΞϒϩάͰ👍
32. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE Ҡߦ࣌ͷ՝୊
33. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE Ҡߦ࣌ͷ՝୊ͱΑ͔ͬͨ͜ͱ  ɾϩάΛίϯςφ಺Ͱ؅ཧͯ͠͠·͏ͱσϓϩΠ͢Δͨͼফ͑ͯ͠·͏ ɹɾCloudWatch Logsʹඪ४ग़ྗ͠ͳ͚Ε͹ͳΒͳ͍ ɹɾLaravelΞϓϦϩάͷstdout͕5.8Ҏ্͡Όͳ͍ͱରԠͰ͖ͳ͍ ɹɾόʔδϣϯΞοϓΛߦͳͬͨ ɾը૾ܥ ɹɾS3 ը૾Ξοϓϩʔυ͕Ұ෦ͷػೳ͔͠ରԠ͍ͯ͠ͳ͔ͬͨ 
 ɹɹɾ1ϲ݄Ҏ্͔͚ͯҠߦ ɾόον΍աڈͷاըͷιʔεΛ୨Է ɹɾෛ࠴ͱͳ͍ͬͯͨιʔεΛ੔ཧͰ͖ͨ ɾUnit/Feature/E2Eͷࣗಈςετ੔උ ɹɾPHPCSΛ࢖ͬͨن໿౷Ұ ɹɾPHPStanʢLarastanʣ࢖ͬͨ੩తղੳͰͷ඼࣭޲্όά࡟ݮ ɹɾϦϑΝΫλϦϯά 💦
34. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE Ҡߦޙͷτϥϒϧ
35. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE Ҡߦޙͷτϥϒϧ  ɾECS Scheduled Tasks ɹɾຖ෼ͷόονͰNAT Gatewayͷ௨৴අ͕ංେԽ 
 ɹɾECR΍CloudWatch Logsͷ޲͖ઌΛมߋ 
 ɹɹɾVPCΤϯυϙΠϯτܦ༝ʹ 
 ɾଞͷάϧʔϓձࣾ΋10ສԁίετΞοϓ͍ͯͨ͠ 
 ɹɾ೥ؒ120ສ࡟ݮʹվળͨ͠ ɾݸਓϒϩάॻ͍͍ͯΔͷͰࢀߟʹʂ 💦
36. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ·ͱΊ
37. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ɾࠓޙ ɹɾ։ൃ؀ڥͷDB΋ϚεΩϯά͞ΕͨDBΛར༻ 
 ɹɹɾcodebuild ɹɾPHP/LaravelͷόʔδϣϯΞοϓ ɹɾίϯςφͷ੬ऑੑݕ஌ (Trivy) ɹɾDatadogͰΞΫηεϩά࿈ܞ ɹɾϥϯαʔζຊՈ΋ίϯςφҠߦʹνϟϨϯδ 
 ɹɹɾࣾ಺Ͱӡ༻͍ͯ͠Δ΋ͷ͸ҠߦࡁΈ ·ͱΊ  ɾ൒೥ͰҠߦ͢Δ͜ͱ͕Ͱ͖ͨ(2020/10~2021/03) 
 ɹɾٕज़తͳෛ࠴ͷ੔ཧΛߦ͍ҠߦΛ࣮ࢪͰ͖ͨ ɾاըɾΞΠσΟΞϕʔεͷϓϩτλΠϓ։ൃ͔Β 
 ɹɾকདྷΛݟਾ͑ӡ༻ɾอक΋ҙࣝͨ͠։ൃελΠϧʹ 
 ɹɾϥϯαʔζͱಉ͡։ൃελΠϧʹมΘͬͨ ɾImmutableͳαʔόʔͷӡ༻ ɹɾ҆ఆӡ༻ɺӡ༻ޮ཰ͷΞοϓ ɹɾ৑௕Խ΍ΦʔτεέʔϦϯάʹΑΔߴ͍Մ༻ੑΛ࣮ݱͨ͠ߏ੒ ɾDatadogͷAPMͰΞϓϦͷΤϥʔ௨஌͕Ͱ͖ΔΑ͏ʹ ɾϗετΠϯελϯεͷ؅ཧ͕ল͚Δ ɾλεΫ୯ҐͰࣗ༝ʹϦιʔεมߋՄೳ 
 ɾλεΫͷىಈ଎౓͕஗Ί ɹɾόον࣮ߦ΋਺෼஗͘ͳΔ 
 ɾ͘͞ΒͷΫϥ΢υ࣌୅ֹ݄Ͱ਺ेສԁˢ 

38. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ΤϯδχΞϒϩάͰ΋ެ։ͯ͠·͢  https://engineer.blog.lancers.jp/2021/03/replace-menta/
39. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE SREٻΉʂʂ  SRE͕଍Γ·ͤΜ😇
40. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ࠂ஌ʂ  https://lancersrecruit.connpass.com/event/221994/
41. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE