MENTAをAWSに移行して振り返る(ECS/Fargate + Laravel編)

2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE
MENTAΛAWSʹҠߦͯ͠ৼΓฦΔ

(ECS/Fargate + Laravelฤ)

ϥϯαʔζגࣜձࣾ
 
SRE/҆ୡ ྋ

View Slide

ΞδΣϯμ
ɾࣗݾ঺հ

ɾ3೥લͷγεςϜߏ੒/͘͞ΒͷΫϥ΢υʹ͍ͭͯ(ۤ࿑࿩)

ɾ։ൃ؀ڥΛ࡞ΔϙΠϯτ

ɾStg/ຊ൪؀ڥͷߏ੒ʹ͍ͭͯ

ɾCircleCIʹΑΔίϯςφσϓϩΠ

ɾECS Scheduled TasksʹΑΔόον࣮૷ʹ͍ͭͯ

ɾϩά/ίϯςφ؂ࢹʹ͍ͭͯ

ɾຊ൪DBϚεΩϯάࣗಈԽʹ͍ͭͯ

ɾECSΦʔτεέʔϧʹ͍ͭͯ

ɾ෼ੳج൫ͷ঺հ(Digdag,Embulk,BigQuery,Redash)

ɾҠߦલͷ՝୊ͱҠߦޙͷτϥϒϧ

ɾ·ͱΊ

View Slide

ࣗݾ঺հ

View Slide

ࣗݾ঺հ
ɾ҆ୡ ྋ(adachin)

ɾLancers SRE 2018/5~

ɾ@adachin0817

ɾRVIRUS0817

ɾblog.adachin.me

ɾwiki.adachin.me

ɾݸਓ։ൃ(famii)

ɾOSS

View Slide

ͪͳΈʹ…

View Slide

ͪͳΈʹ…
MENTAೖΕΔͱ3ͭίϯςφʹҠߦ͠·ͨ͠ʂ
PROSheet
 
Lancers Agent
Lancers Assistant
 
Creative

View Slide

ͪͳΈʹMENTAͰ͸..?

View Slide

ͪͳΈʹMENTAͰ͸..?
ɾΠϯϑϥ/αʔόʔ ίʔνϯά

ɾMENTAϥϯΩϯά

ɹɾ4Ґʂ

ɾྦྷܭ184ਓʂ
 
ɹɾۦ͚ग़͠ΤϯδχΞϑΥϩʔ

ɹɾస৬࣮੷ଟ਺༗Γ

View Slide

3೥લͷγεςϜߏ੒

View Slide

ɾ2018೥ϦϦʔεɹ

ɾ෭ۀͰࣗ෼͕αʔόʔอक͍ͯͨ͠
 
ɾ͘͞ΒͷΫϥ΢υ

ɾPHP 7.2 / Laravel 5.5

ɹɾNginx / MariaDB /Redis / Ansible

ɹɾSendGrid/Redash
 
ɹɾ؂ࢹ/Mackerel

ɹɾ΄ͱΜͲVPSతͳӡ༻

ɾ։ൃ؀ڥ

ɹɾDocker

ɾϦϦʔεํ๏

ɹɾSSHʹΑΔγΣϧεΫϦϓτ

View Slide

 
(ۤ࿑࿩)

View Slide

3೥લͷγεςϜߏ੒(ۤ࿑࿩)
ɾσϓϩΠ͕͠೉͍
 
ɹɾγΣϧεΫϦϓτ͕ͩɺ΄΅खಈͰϦϦʔε

ɾΞΫηεෛՙʹΑΔϨΠςϯγʔͷѱ͞
 
ɹɾϩʔυΞϕϨʔδ΍ϝϞϦʔ͕ΧπΧπ…

ɾ৑௕ੑ΍ෛՙ෼ࢄͳͲ͸୲อ͍ͯ͠ͳ͍ঢ়ଶ

ɹɾ1୆Ͱࡹ͘ͷ͸ͱʹ͔͘͠ΜͲ͍…౔೔͕ϐʔΫ

ɾͳΔ΂͘ίετΛ཈͑ͨΠϯϑϥ؀ڥ
 
ɹɾStgͰςετ͍͚ͨ͠ͲεϖοΫ͕଍ΓͶ͑..!

View Slide

։ൃ؀ڥΛ࡞Δ
 
ϙΠϯτ

View Slide

։ൃ؀ڥΛ࡞ΔϙΠϯτ
ɾຊ൪؀ڥͱಉ౳ͷߏ੒Λ࠶ݱ

ɾELB(Nginx)ίϯςφ(ϦόʔεϓϩΩγ)

ɾApp(PHP7.2-alpine/Nginx)

ɾMySQL 5.7

ɾSendGrid/phpMyAdmin/Redis

ɾStripe/Selenium/node/Digdag(Embulk)

ɾWordPress(PHP7.3-alpine/Nginx)

ɾ֤ΞϓϦέʔγϣϯͷىಈ͸SupervisorͰ؅ཧ

ɾcomposer install͸AppίϯςφͰϩάΠϯ࣌ʹ࣮ߦ

ɹɾsupervisorctlͰΞϓϦΛrestart͢ΔΑ͏ʹ
 
ɾM1 MacͰ΋ಈ࡞ݕূࡁΈ

View Slide

Stg/ຊ൪؀ڥͷ
 
ߏ੒ʹ͍ͭͯ

View Slide

Stg/ຊ൪؀ڥͷߏ੒ʹ͍ͭͯ
TerraformͰશϦιʔείʔυԽ

ɾStaging؀ڥ͸ۃྗίετΛ཈͑λεΫ਺͸1ͭ

ɾࣾ಺༻͸ϥϯαʔζRProxyͰΞΫηε(phpMyAdmin,SendGrid,Digdag,Redash)

ɾWordPressίϯςφʹΑΔϝσΟΞӡ༻

ɾphpMyAdminίϯςφʹΑΔσʔλग़͠

ɾΦʔτεέʔϧʹΑΔ࠷େ6୆ͷAppίϯςφͰӡ༻

ɾdevops/SSHίϯςφͷར༻Ͱaws cli.MySQLͷૢ࡞

ɹɾݱࡏ͸ഇࢭ͕ͨ͠EC2ຊ൪DBϚεΩϯά༻ͷ؀ڥͱͯ͠ར༻

ɾDigdag/Embulk/BigQueryʹΑΔ෼ੳج൫ͷߏங

ɾDB͸RDS AuroraʹҠߦ

ɾElastiCacheʹΑΔϩάΠϯΩϟογϡԽ

ɾը૾౳S3Խ

ɾϩά͸CloudWatch Logs

ɾόον͸ECS Scheduled Tasks

ɾ಺෦؂ࠪ(ϩάͷू໿)

ɹɾAmazon Kinesis

ɾAWS KMSʹΑΔ҉߸Խ

ɾAWS WAFͰͷDDoSΞλοΫ੍ޚ

ɾ؂ࢹ͸DatadogͰίϯςφϦιʔεͱAPMͰͷΤϥʔݕ஌

ɾຊ൪؀ڥ͸ΧφϦΞ؀ڥ΋

View Slide

SSH/࡞ۀ༻ίϯςφ
 
ʮdevopsʯʹ͍ͭͯ

View Slide

SSH/࡞ۀ༻ίϯςφʮdevopsʯʹ͍ͭͯ

ɾAppʹ͸SSH͠ͳ͍
 
ɾSSH/࡞ۀ༻ίϯςφ(Appͱಉ౳)

ɹɾdevopsͱ໋໊

ɾίϚϯυͰRDS΁઀ଓ

ɹɾdump΍restore

ɾaws cliͷར༻

ɾSSMͷSession Manager ΋͋Δ͕

ɹɾखܰʹ౿Έ୆͔ΒSSHΛ͍ͨͨ͠Ί

ɾVPC PeeringͰϥϯαʔζͷωοτϫʔΫ͔ΒSSHϩάΠϯ

View Slide

SSH/࡞ۀ༻ίϯςφʮdevopsʯͷݱࡏ

ɾdevopsίϯςφΛഇࢭ(2021/04~)

ɾECS Execͷొ৔

ɾAppίϯςφʹ௚઀ϩάΠϯ

ɾγΣϧεΫϦϓτͰ࣮૷

ɹɾݱࡏ͸ecspressoͰίϯςφ؅ཧ ↑ݸਓϒϩάΛνΣοΫ👍

$ ls /etc/ecspresso/menta/
README.md app.yaml digdag.yaml
$ export AWS_PROFILE=menta
$ ecspresso --config /etc/ecspresso/menta/app.yaml exec --command bash
$ ecspresso --config /etc/ecspresso/menta/digdag.yaml exec --command bash

View Slide

CircleCIʹΑΔ
 
ίϯςφσϓϩΠʹ͍ͭͯ

View Slide

APIܦ༝ͰͷStg؀ڥͷσϓϩΠ
ɾStg؀ڥͷσϓϩΠ

ɹɾgithub flow

ɹɾγΣϧεΫϦϓτ
 
ɹɹɾ೚ҙͷϒϥϯνΛࢦఆ

ɾૉૣ͘σϓϩΠ͕Մೳ

ɹɾ։ൃޮ཰͕Ξοϓ

ɾMasterϚʔδͰຊ൪؀ڥϦϦʔε

ɾTerraform CI؀ڥΛ࣮૷

$ sh deploy.sh
ԼهͷΑ͏ʹϒϥϯνΛࢦఆ࣮ͯ͠ߦ͍ͯͩ͘͠͞ɻ
(ྫ) sh deploy.sh ϒϥϯν໊
$ sh deploy.sh fix-nginx
{
"number" : 74,
"state" : "pending",
"id" : "xxxxxx-xxxxx-xxxxx-xxx-xxxxxxxx",
"created_at" : “2021-06-27T06:14:43.973Z"
}
ϒϥϯν໊/ fix-nginx ͷStgσϓϩΠΛ։࢝͠·ͨ͠!

View Slide

όονͷ࣮૷ʹ͍ͭͯ

View Slide

ECS Scheduled TasksʹΑΔόον࣮૷ʹ͍ͭͯ

ɾࠓ·Ͱ
 
ɹɾLaravelͷλεΫεέδϡʔϧ

ɾݱࡏ

ɹɾECS Scheduled Tasks
 
ɹɹɾCloudWatchͷΠϕϯτϧʔϧ
 
ɹɹɾTerraformԽ
{
"containerOverrides": [
{
"name": "menta-app",
"command": [
"php",
"artisan",
"admin:coupon-mail:send"
]
}
]
}

View Slide

ϩάʗίϯςφ؂ࢹ
 
ʹ͍ͭͯ

View Slide

ϩά/ίϯςφ؂ࢹʹ͍ͭͯ

ɾΞΫηε/ΞϓϦϩάɺRDS؂ࠪ/Τϥʔϩά
 
ɹɾCloudWatch Logsʹ

ɹɾࠓ·Ͱ

ɹɹɾLambdaͰS3ʹγϯΫɹɹ

ɹɹɾPythonͰ࣮૷͍ͯͨͨ͠Ίӡ༻ίετΞοϓ

ɹɾࠓճ

ɹɹɾAmazon Kinesis Data FirehoseͰS3ʹू໿

ɹɹɾTerraformԽ

 
ɾ؂ࢹ

ɹɾMackerel→DatadogʹҠߦ

ɹɾλεΫఆٛʹdatadog-agentΛಈ࡞ͤ͞ΔΑ͏ʹ
 
ɹɾcpu:10ɺmemory:256 TerraformԽ

ɹɾAPMͰͷΞϓϦΤϥʔݕ஌

View Slide

ຊ൪DBϚεΩϯά
 
ࣗಈԽ

View Slide

ຊ൪DBϚεΩϯάࣗಈԽ
 

ɾStg؀ڥͷDB

ɹɾຖि1ճຊ൪DBͱಉ͡σʔλΛೖΕͯ΄͍͠ʂ

ɹɾݸਓ৘ใ͸΋ͪΖΜϚεΩϯά

ɾdevops(EC2)ͷ෮׆

ɾECS/FargateͰ΍Ζ͏ͱࢥͬͨ

ɹɾσΟεΫྔ͕গͳ͍ͨΊஅ೦

ɾauto-masking

ɹɾprd-masking.sh ຖि౔༵೔0࣌ʙ

ɹɹɾDump

ɹɾstg-masking-restore.sh ຖि౔༵೔5࣌ʙ

ɹɹɾRestore

ɾpost_slack.sh ݁ՌΛSlack௨஌

ɾ։ൃ؀ڥ(MySQLίϯςφ)

ɾauto-renew-mysql-container ຖि౔༵೔9࣌ʙ

ɾCodeBuild

ɹɾMySQLίϯςφΛbuild͠ϚεΩϯάσʔλΛRestore

View Slide

ECSͰॳͷ
 
Φʔτεέʔϧ࠾༻

View Slide

ECSॳͷΦʔτεέʔϧ࠾༻

ɾECS/FargateͰॳͷΦʔτεέʔϧ࠾༻

ɾTerraformԽ

ɹɾCloudWatch Alarmઃఆ(CPU 40%ͰλεΫ૿Ճ)

ɹɾECS/Service AutoScaling ઃఆ (λεΫͷ࠷େ਺4)

ɹɾAuto Scaling Policyઃఆ

ɾEC2ͷΑ͏ʹAMIͳͲͷ࡞ΓࠐΈ͕Ұ੾ͳ͍ˠָ

View Slide

෼ੳج൫ͷ঺հ
 
(Digdag/Embulk/
 
BigQuery/Redash)

View Slide

෼ੳج൫ͷ঺հ

ɾLancersͷ෼ੳج൫ΛԠ༻

ɾ෼ੳج൫

ɹɾDigdag/Embulk/Redash/BigQuery
 
ɹɾຖ೔ਂ໷ʹRDSͷσʔλΛBigQueryʹγϯΫ

ɹɾLancersͱಉ͡ϓϩδΣΫτͰ؅ཧ
 
ɾઌ݄·ͰEC2(Digdagαʔόʔ)Ͱӡ༻͍ͯͨ͠

ɹɾECS/FargateʹҠߦ(։ൃ؀ڥ΋ߏங)
 
ɹɾຊՈϥϯαʔζ΋Ҡߦ

ɾৄࡉ͸ϥϯαʔζΤϯδχΞϒϩάͰ👍

View Slide

Ҡߦ࣌ͷ՝୊

View Slide

Ҡߦ࣌ͷ՝୊ͱΑ͔ͬͨ͜ͱ
ɾϩάΛίϯςφ಺Ͱ؅ཧͯ͠͠·͏ͱσϓϩΠ͢Δͨͼফ͑ͯ͠·͏

ɹɾCloudWatch Logsʹඪ४ग़ྗ͠ͳ͚Ε͹ͳΒͳ͍

ɹɾLaravelΞϓϦϩάͷstdout͕5.8Ҏ্͡Όͳ͍ͱରԠͰ͖ͳ͍

ɹɾόʔδϣϯΞοϓΛߦͳͬͨ

ɾը૾ܥ

ɹɾS3 ը૾Ξοϓϩʔυ͕Ұ෦ͷػೳ͔͠ରԠ͍ͯ͠ͳ͔ͬͨ
 
ɹɹɾ1ϲ݄Ҏ্͔͚ͯҠߦ

ɾόον΍աڈͷاըͷιʔεΛ୨Է

ɹɾෛ࠴ͱͳ͍ͬͯͨιʔεΛ੔ཧͰ͖ͨ

ɾUnit/Feature/E2Eͷࣗಈςετ੔උ

ɹɾPHPCSΛ࢖ͬͨن໿౷Ұ

ɹɾPHPStanʢLarastanʣ࢖ͬͨ੩తղੳͰͷ඼࣭޲্όά࡟ݮ

ɹɾϦϑΝΫλϦϯά
💦

View Slide

Ҡߦޙͷτϥϒϧ

View Slide

Ҡߦޙͷτϥϒϧ

ɾECS Scheduled Tasks

ɹɾຖ෼ͷόονͰNAT Gatewayͷ௨৴අ͕ංେԽ
 
ɹɾECR΍CloudWatch Logsͷ޲͖ઌΛมߋ
 
ɹɹɾVPCΤϯυϙΠϯτܦ༝ʹ
 
ɾଞͷάϧʔϓձࣾ΋10ສԁίετΞοϓ͍ͯͨ͠
 
ɹɾ೥ؒ120ສ࡟ݮʹվળͨ͠

ɾݸਓϒϩάॻ͍͍ͯΔͷͰࢀߟʹʂ
💦

View Slide

·ͱΊ

View Slide

ɾࠓޙ

ɹɾ։ൃ؀ڥͷDB΋ϚεΩϯά͞ΕͨDBΛར༻
 
ɹɹɾcodebuild

ɹɾPHP/LaravelͷόʔδϣϯΞοϓ

ɹɾίϯςφͷ੬ऑੑݕ஌ (Trivy)

ɹɾDatadogͰΞΫηεϩά࿈ܞ

ɹɾϥϯαʔζຊՈ΋ίϯςφҠߦʹνϟϨϯδ
 
ɹɹɾࣾ಺Ͱӡ༻͍ͯ͠Δ΋ͷ͸ҠߦࡁΈ
·ͱΊ

ɾ൒೥ͰҠߦ͢Δ͜ͱ͕Ͱ͖ͨ(2020/10~2021/03)
 
ɹɾٕज़తͳෛ࠴ͷ੔ཧΛߦ͍ҠߦΛ࣮ࢪͰ͖ͨ

ɾاըɾΞΠσΟΞϕʔεͷϓϩτλΠϓ։ൃ͔Β
 
ɹɾকདྷΛݟਾ͑ӡ༻ɾอक΋ҙࣝͨ͠։ൃελΠϧʹ
 
ɹɾϥϯαʔζͱಉ͡։ൃελΠϧʹมΘͬͨ

ɾImmutableͳαʔόʔͷӡ༻

ɹɾ҆ఆӡ༻ɺӡ༻ޮ཰ͷΞοϓ

ɹɾ৑௕Խ΍ΦʔτεέʔϦϯάʹΑΔߴ͍Մ༻ੑΛ࣮ݱͨ͠ߏ੒

ɾDatadogͷAPMͰΞϓϦͷΤϥʔ௨஌͕Ͱ͖ΔΑ͏ʹ

ɾϗετΠϯελϯεͷ؅ཧ͕ল͚Δ

ɾλεΫ୯ҐͰࣗ༝ʹϦιʔεมߋՄೳ

 
ɾλεΫͷىಈ଎౓͕஗Ί

ɹɾόον࣮ߦ΋਺෼஗͘ͳΔ
 
ɾ͘͞ΒͷΫϥ΢υ࣌୅ֹ݄Ͱ਺ेສԁˢ

 

View Slide

ΤϯδχΞϒϩάͰ΋ެ։ͯ͠·͢

https://engineer.blog.lancers.jp/2021/03/replace-menta/

View Slide

SREٻΉʂʂ

SRE͕଍Γ·ͤΜ😇

View Slide

ࠂ஌ʂ
https://lancersrecruit.connpass.com/event/221994/

View Slide


View Slide

MENTAをAWSに移行して振り返る(ECS/Fargate + Laravel編)

MENTAをAWSに移行して振り返る(ECS/Fargate + Laravel編)

adachin0817

More Decks by adachin0817

Other Decks in Programming

Featured

Transcript