Upgrade to Pro — share decks privately, control downloads, hide ads and more …

MENTAをAWSに移行して振り返る(ECS/Fargate + Laravel編)

MENTAをAWSに移行して振り返る(ECS/Fargate + Laravel編)

https://lancersrecruit.connpass.com/event/219434/

【SPACEMARKET×Lancers】シェアリングエコノミーを支えるインフラ/SREでのスライドとなります。

5059d3f370ad7e1f4d8de4be79ae1a2c?s=128

adachin0817

August 25, 2021
Tweet

Transcript

  1. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE MENTAΛAWSʹҠߦͯ͠ৼΓฦΔ (ECS/Fargate + Laravelฤ) ϥϯαʔζגࣜձࣾ 
 SRE/҆ୡ ྋ

  2. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ΞδΣϯμ  ɾࣗݾ঺հ ɾ3೥લͷγεςϜߏ੒/͘͞ΒͷΫϥ΢υʹ͍ͭͯ(ۤ࿑࿩) ɾ։ൃ؀ڥΛ࡞ΔϙΠϯτ ɾStg/ຊ൪؀ڥͷߏ੒ʹ͍ͭͯ ɾCircleCIʹΑΔίϯςφσϓϩΠ ɾECS Scheduled

    TasksʹΑΔόον࣮૷ʹ͍ͭͯ ɾϩά/ίϯςφ؂ࢹʹ͍ͭͯ ɾຊ൪DBϚεΩϯάࣗಈԽʹ͍ͭͯ ɾECSΦʔτεέʔϧʹ͍ͭͯ ɾ෼ੳج൫ͷ঺հ(Digdag,Embulk,BigQuery,Redash) ɾҠߦલͷ՝୊ͱҠߦޙͷτϥϒϧ ɾ·ͱΊ
  3. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ࣗݾ঺հ

  4. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ࣗݾ঺հ  ɾ҆ୡ ྋ(adachin) ɾLancers SRE 2018/5~ ɾ@adachin0817/@adachin_channel ɾRVIRUS0817

    ɾblog.adachin.me ɾwiki.adachin.me ɾݸਓ։ൃ΋ ɾήʔϜ࣮گYouTuber
  5. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ͪͳΈʹ…

  6. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ͪͳΈʹ…  MENTAೖΕΔͱ3ͭίϯςφʹҠߦ͠·ͨ͠ʂ PROSheet 
 Lancers Agent Lancers Assistant

    
 Creative
  7. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ͪͳΈʹMENTAͰ͸..?

  8. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ͪͳΈʹMENTAͰ͸..?  ɾΠϯϑϥ/αʔόʔ ίʔνϯά ɾMENTAϥϯΩϯά ɹɾ4Ґʂ ɾྦྷܭ184ਓʂ 
 ɹɾۦ͚ग़͠ΤϯδχΞϑΥϩʔ

    ɹɾస৬࣮੷ଟ਺༗Γ
  9. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE 3೥લͷγεςϜߏ੒

  10. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE 3೥લͷγεςϜߏ੒  ɾ2018೥ϦϦʔεɹ ɾ෭ۀͰࣗ෼͕αʔόʔอक͍ͯͨ͠ 
 ɾ͘͞ΒͷΫϥ΢υ ɾPHP 7.2 /

    Laravel 5.5 ɹɾNginx / MariaDB /Redis / Ansible ɹɾSendGrid/Redash 
 ɹɾ؂ࢹ/Mackerel ɹɾ΄ͱΜͲVPSతͳӡ༻ ɾ։ൃ؀ڥ ɹɾDocker ɾϦϦʔεํ๏ ɹɾSSHʹΑΔγΣϧεΫϦϓτ
  11. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE 3೥લͷγεςϜߏ੒ 
 (ۤ࿑࿩)

  12. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE 3೥લͷγεςϜߏ੒(ۤ࿑࿩)  ɾσϓϩΠ͕͠೉͍ 
 ɹɾγΣϧεΫϦϓτ͕ͩɺ΄΅खಈͰϦϦʔε ɾΞΫηεෛՙʹΑΔϨΠςϯγʔͷѱ͞ 
 ɹɾϩʔυΞϕϨʔδ΍ϝϞϦʔ͕ΧπΧπ… ɾ৑௕ੑ΍ෛՙ෼ࢄͳͲ͸୲อ͍ͯ͠ͳ͍ঢ়ଶ

    ɹɾ1୆Ͱࡹ͘ͷ͸ͱʹ͔͘͠ΜͲ͍…౔೔͕ϐʔΫ ɾͳΔ΂͘ίετΛ཈͑ͨΠϯϑϥ؀ڥ 
 ɹɾStgͰςετ͍͚ͨ͠ͲεϖοΫ͕଍ΓͶ͑..!
  13. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ։ൃ؀ڥΛ࡞Δ 
 ϙΠϯτ

  14. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ։ൃ؀ڥΛ࡞ΔϙΠϯτ  ɾຊ൪؀ڥͱಉ౳ͷߏ੒Λ࠶ݱ ɾELB(Nginx)ίϯςφ(ϦόʔεϓϩΩγ) ɾApp(PHP7.2-alpine/Nginx) ɾMySQL 5.7 ɾSendGrid/phpMyAdmin/Redis ɾStripe/Selenium/node/Digdag(Embulk)

    ɾWordPress(PHP7.3-alpine/Nginx) ɾ֤ΞϓϦέʔγϣϯͷىಈ͸SupervisorͰ؅ཧ ɾcomposer install͸AppίϯςφͰϩάΠϯ࣌ʹ࣮ߦ ɹɾsupervisorctlͰΞϓϦΛrestart͢ΔΑ͏ʹ 
 ɾM1 MacͰ΋ಈ࡞ݕূࡁΈ
  15. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE Stg/ຊ൪؀ڥͷ 
 ߏ੒ʹ͍ͭͯ

  16. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE Stg/ຊ൪؀ڥͷߏ੒ʹ͍ͭͯ  TerraformͰશϦιʔείʔυԽ ɾStaging؀ڥ͸ۃྗίετΛ཈͑λεΫ਺͸1ͭ ɾࣾ಺༻͸ϥϯαʔζRProxyͰΞΫηε(phpMyAdmin,SendGrid,Digdag,Redash) ɾWordPressίϯςφʹΑΔϝσΟΞӡ༻ ɾphpMyAdminίϯςφʹΑΔσʔλग़͠ ɾΦʔτεέʔϧʹΑΔ࠷େ6୆ͷAppίϯςφͰӡ༻ ɾdevops/SSHίϯςφͷར༻Ͱaws

    cli.MySQLͷૢ࡞ ɹɾݱࡏ͸ഇࢭ͕ͨ͠EC2ຊ൪DBϚεΩϯά༻ͷ؀ڥͱͯ͠ར༻ ɾDigdag/Embulk/BigQueryʹΑΔ෼ੳج൫ͷߏங ɾDB͸RDS AuroraʹҠߦ ɾElastiCacheʹΑΔϩάΠϯΩϟογϡԽ ɾը૾౳S3Խ ɾϩά͸CloudWatch Logs ɾόον͸ECS Scheduled Tasks ɾ಺෦؂ࠪ(ϩάͷू໿) ɹɾAmazon Kinesis ɾAWS KMSʹΑΔ҉߸Խ ɾAWS WAFͰͷDDoSΞλοΫ੍ޚ ɾ؂ࢹ͸DatadogͰίϯςφϦιʔεͱAPMͰͷΤϥʔݕ஌ ɾຊ൪؀ڥ͸ΧφϦΞ؀ڥ΋
  17. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE SSH/࡞ۀ༻ίϯςφ 
 ʮdevopsʯʹ͍ͭͯ

  18. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE SSH/࡞ۀ༻ίϯςφʮdevopsʯʹ͍ͭͯ  ɾAppʹ͸SSH͠ͳ͍ 
 ɾSSH/࡞ۀ༻ίϯςφ(Appͱಉ౳) ɹɾdevopsͱ໋໊ ɾίϚϯυͰRDS΁઀ଓ ɹɾdump΍restore ɾaws

    cliͷར༻ ɾSSMͷSession Manager ΋͋Δ͕ ɹɾखܰʹ౿Έ୆͔ΒSSHΛ͍ͨͨ͠Ί ɾVPC PeeringͰϥϯαʔζͷωοτϫʔΫ͔ΒSSHϩάΠϯ
  19. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE SSH/࡞ۀ༻ίϯςφʮdevopsʯͷݱࡏ  ɾdevopsίϯςφΛഇࢭ(2021/04~) ɾECS Execͷొ৔ ɾAppίϯςφʹ௚઀ϩάΠϯ ɾγΣϧεΫϦϓτͰ࣮૷ ɹɾݱࡏ͸ecspressoͰίϯςφ؅ཧ ↑ݸਓϒϩάΛνΣοΫ👍

    $ ls /etc/ecspresso/menta / README.md app.yaml digdag.yam l $ export AWS_PROFILE=ment a $ ecspresso --config /etc/ecspresso/menta/app.yaml exec --command bas h $ ecspresso --config /etc/ecspresso/menta/digdag.yaml exec --command bash
  20. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE CircleCIʹΑΔ 
 ίϯςφσϓϩΠʹ͍ͭͯ

  21. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE APIܦ༝ͰͷStg؀ڥͷσϓϩΠ  ɾStg؀ڥͷσϓϩΠ ɹɾgithub flow ɹɾγΣϧεΫϦϓτ 
 ɹɹɾ೚ҙͷϒϥϯνΛࢦఆ ɾૉૣ͘σϓϩΠ͕Մೳ

    ɹɾ։ൃޮ཰͕Ξοϓ ɾMasterϚʔδͰຊ൪؀ڥϦϦʔε ɾTerraform CI؀ڥΛ࣮૷ $ sh deploy.s h ԼهͷΑ͏ʹϒϥϯνΛࢦఆ࣮ͯ͠ߦ͍ͯͩ͘͠͞ɻ (ྫ) sh deploy.sh ϒϥϯν໊ $ sh deploy.sh fix-ngin x { "number" : 74 , "state" : "pending" , "id" : "xxxxxx-xxxxx-xxxxx-xxx-xxxxxxxx" , "created_at" : “2021-06-27T06:14:43.973Z " } ϒϥϯν໊/ fix-nginx ͷStgσϓϩΠΛ։࢝͠·ͨ͠!
  22. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE όονͷ࣮૷ʹ͍ͭͯ

  23. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ECS Scheduled TasksʹΑΔόον࣮૷ʹ͍ͭͯ  ɾࠓ·Ͱ 
 ɹɾLaravelͷλεΫεέδϡʔϧ ɾݱࡏ ɹɾECS

    Scheduled Tasks 
 ɹɹɾCloudWatchͷΠϕϯτϧʔϧ 
 ɹɹɾTerraformԽ { "containerOverrides": [ { "name": "menta-app" , "command": [ "php" , "artisan" , "admin:coupon-mail:send " ] } ] }
  24. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ϩάʗίϯςφ؂ࢹ 
 ʹ͍ͭͯ

  25. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ϩά/ίϯςφ؂ࢹʹ͍ͭͯ  ɾΞΫηε/ΞϓϦϩάɺRDS؂ࠪ/Τϥʔϩά 
 ɹɾCloudWatch Logsʹ ɹɾࠓ·Ͱ ɹɹɾLambdaͰS3ʹγϯΫɹɹ ɹɹɾPythonͰ࣮૷͍ͯͨͨ͠Ίӡ༻ίετΞοϓ

    ɹɾࠓճ ɹɹɾAmazon Kinesis Data FirehoseͰS3ʹू໿ ɹɹɾTerraformԽ 
 ɾ؂ࢹ ɹɾMackerel→DatadogʹҠߦ ɹɾλεΫఆٛʹdatadog-agentΛಈ࡞ͤ͞ΔΑ͏ʹ 
 ɹɾcpu:10ɺmemory:256 TerraformԽ ɹɾAPMͰͷΞϓϦΤϥʔݕ஌
  26. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ຊ൪DBϚεΩϯά 
 ࣗಈԽ

  27. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ຊ൪DBϚεΩϯάࣗಈԽ 
  ɾStg؀ڥͷDB ɹɾຖि1ճຊ൪DBͱಉ͡σʔλΛೖΕͯ΄͍͠ʂ ɹɾݸਓ৘ใ͸΋ͪΖΜϚεΩϯά ɾdevops(EC2)ͷ෮׆ ɾECS/FargateͰ΍Ζ͏ͱࢥͬͨ ɹɾσΟεΫྔ͕গͳ͍ͨΊஅ೦

    ɾauto-masking ɹɾprd-masking.sh ຖि౔༵೔0࣌ʙ ɹɹɾDump ɹɾstg-masking-restore.sh ຖि౔༵೔5࣌ʙ ɹɹɾRestore ɾpost_slack.sh ݁ՌΛSlack௨஌ ɾ։ൃ؀ڥ(MySQLίϯςφ) ɾauto-renew-mysql-container ຖि౔༵೔9࣌ʙ ɾCodeBuild ɹɾMySQLίϯςφΛbuild͠ϚεΩϯάσʔλΛRestore
  28. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ECSͰॳͷ 
 Φʔτεέʔϧ࠾༻

  29. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ECSॳͷΦʔτεέʔϧ࠾༻  ɾECS/FargateͰॳͷΦʔτεέʔϧ࠾༻ ɾTerraformԽ ɹɾCloudWatch Alarmઃఆ(CPU 40%ͰλεΫ૿Ճ) ɹɾECS/Service AutoScaling

    ઃఆ (λεΫͷ࠷େ਺4) ɹɾAuto Scaling Policyઃఆ ɾEC2ͷΑ͏ʹAMIͳͲͷ࡞ΓࠐΈ͕Ұ੾ͳ͍ˠָ
  30. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ෼ੳج൫ͷ঺հ 
 (Digdag/Embulk/ 
 BigQuery/Redash)

  31. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ෼ੳج൫ͷ঺հ  ɾLancersͷ෼ੳج൫ΛԠ༻ ɾ෼ੳج൫ ɹɾDigdag/Embulk/Redash/BigQuery 
 ɹɾຖ೔ਂ໷ʹRDSͷσʔλΛBigQueryʹγϯΫ ɹɾLancersͱಉ͡ϓϩδΣΫτͰ؅ཧ 


    ɾઌ݄·ͰEC2(Digdagαʔόʔ)Ͱӡ༻͍ͯͨ͠ ɹɾECS/FargateʹҠߦ(։ൃ؀ڥ΋ߏங) 
 ɹɾຊՈϥϯαʔζ΋Ҡߦ ɾৄࡉ͸ϥϯαʔζΤϯδχΞϒϩάͰ👍
  32. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE Ҡߦ࣌ͷ՝୊

  33. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE Ҡߦ࣌ͷ՝୊ͱΑ͔ͬͨ͜ͱ  ɾϩάΛίϯςφ಺Ͱ؅ཧͯ͠͠·͏ͱσϓϩΠ͢Δͨͼফ͑ͯ͠·͏ ɹɾCloudWatch Logsʹඪ४ग़ྗ͠ͳ͚Ε͹ͳΒͳ͍ ɹɾLaravelΞϓϦϩάͷstdout͕5.8Ҏ্͡Όͳ͍ͱରԠͰ͖ͳ͍ ɹɾόʔδϣϯΞοϓΛߦͳͬͨ ɾը૾ܥ ɹɾS3

    ը૾Ξοϓϩʔυ͕Ұ෦ͷػೳ͔͠ରԠ͍ͯ͠ͳ͔ͬͨ 
 ɹɹɾ1ϲ݄Ҏ্͔͚ͯҠߦ ɾόον΍աڈͷاըͷιʔεΛ୨Է ɹɾෛ࠴ͱͳ͍ͬͯͨιʔεΛ੔ཧͰ͖ͨ ɾUnit/Feature/E2Eͷࣗಈςετ੔උ ɹɾPHPCSΛ࢖ͬͨن໿౷Ұ ɹɾPHPStanʢLarastanʣ࢖ͬͨ੩తղੳͰͷ඼࣭޲্όά࡟ݮ ɹɾϦϑΝΫλϦϯά 💦
  34. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE Ҡߦޙͷτϥϒϧ

  35. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE Ҡߦޙͷτϥϒϧ  ɾECS Scheduled Tasks ɹɾຖ෼ͷόονͰNAT Gatewayͷ௨৴අ͕ංେԽ 
 ɹɾECR΍CloudWatch

    Logsͷ޲͖ઌΛมߋ 
 ɹɹɾVPCΤϯυϙΠϯτܦ༝ʹ 
 ɾଞͷάϧʔϓձࣾ΋10ສԁίετΞοϓ͍ͯͨ͠ 
 ɹɾ೥ؒ120ສ࡟ݮʹվળͨ͠ ɾݸਓϒϩάॻ͍͍ͯΔͷͰࢀߟʹʂ 💦
  36. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ·ͱΊ

  37. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ɾࠓޙ ɹɾ։ൃ؀ڥͷDB΋ϚεΩϯά͞ΕͨDBΛར༻ 
 ɹɹɾcodebuild ɹɾPHP/LaravelͷόʔδϣϯΞοϓ ɹɾίϯςφͷ੬ऑੑݕ஌ (Trivy) ɹɾDatadogͰΞΫηεϩά࿈ܞ ɹɾϥϯαʔζຊՈ΋ίϯςφҠߦʹνϟϨϯδ

    
 ɹɹɾࣾ಺Ͱӡ༻͍ͯ͠Δ΋ͷ͸ҠߦࡁΈ ·ͱΊ  ɾ൒೥ͰҠߦ͢Δ͜ͱ͕Ͱ͖ͨ(2020/10~2021/03) 
 ɹɾٕज़తͳෛ࠴ͷ੔ཧΛߦ͍ҠߦΛ࣮ࢪͰ͖ͨ ɾاըɾΞΠσΟΞϕʔεͷϓϩτλΠϓ։ൃ͔Β 
 ɹɾকདྷΛݟਾ͑ӡ༻ɾอक΋ҙࣝͨ͠։ൃελΠϧʹ 
 ɹɾϥϯαʔζͱಉ͡։ൃελΠϧʹมΘͬͨ ɾImmutableͳαʔόʔͷӡ༻ ɹɾ҆ఆӡ༻ɺӡ༻ޮ཰ͷΞοϓ ɹɾ৑௕Խ΍ΦʔτεέʔϦϯάʹΑΔߴ͍Մ༻ੑΛ࣮ݱͨ͠ߏ੒ ɾDatadogͷAPMͰΞϓϦͷΤϥʔ௨஌͕Ͱ͖ΔΑ͏ʹ ɾϗετΠϯελϯεͷ؅ཧ͕ল͚Δ ɾλεΫ୯ҐͰࣗ༝ʹϦιʔεมߋՄೳ 
 ɾλεΫͷىಈ଎౓͕஗Ί ɹɾόον࣮ߦ΋਺෼஗͘ͳΔ 
 ɾ͘͞ΒͷΫϥ΢υ࣌୅ֹ݄Ͱ਺ेສԁˢ 

  38. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ΤϯδχΞϒϩάͰ΋ެ։ͯ͠·͢  https://engineer.blog.lancers.jp/2021/03/replace-menta/

  39. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE SREٻΉʂʂ  SRE͕଍Γ·ͤΜ😇

  40. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ࠂ஌ʂ  https://lancersrecruit.connpass.com/event/221994/

  41. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE