޿ࠂ഑৴Λࢧ͑Δόονج൫Λ αʔόʔϨεҠߦͨ͠࿩ ECS Fargate, Step Functions Serverless Meetup Tokyo #16 2020/02/27

ۙ৿३ฏ pei0804 Zucks(VOYAGE GROUP)ΤϯδχΞ 2018೥ʹ৽ଔೖࣾ͠ɺӡ༻͔Β։ൃΛͯ͠·͢ɻ ࠷ۙ͸νʔϜΛԣஅͨ͠ηΩϡϦςΟؔ࿈ͷ ੔උ΍DSP։ൃΛ΍ͬͯ·͢ɻ ޷͖ͳαʔϏε͸ɺϑϧϚωʔδυαʔϏεͰ͢ɻ

ΞδΣϯμ • Ҡߦݩͷόοναʔόʔͷ՝୊ • αʔόʔϨεҠߦ • ࣮૷ • ؂ࢹ • ӡ༻ͯ͠ΈͯͲ͏͔ͩͬͨ

Ҡߦݩͷόοναʔόʔͷ՝୊

Ҡߦݩͷόοναʔόʔ DSPOVTFSUJNFPVUqPDLXIPHFMPDLNBLFSVOIPHF DSPOVTFSUJNFPVUqPDLXGVHBMPDLNBLFSVOGVHB DSPOVTFSUJNFPVUqPDLXGVHBMPDLNBLFSVOGPP ࡉ͔͍࣮૷ͷ࢓ํ͸ɺνʔϜʹΑͬͯҧ͏ͱࢥ͍·͢ɻ

ӡ༻͕೉͘͠ͳΓ͕ͪ

ӡ༻͕೉͍͠ཁҼ • ໰୊ൃੜ࣌ʹߟ͑Δ͜ͱ͕ଟ͍ɻ • ෮ݩͷ࢓ํ஌ͬͯΔਓډΔʁ

໰୊ൃੜ࣌ʹߟ͑Δ͜ͱ͕ଟ͍ • όονॲཧ͕མͪͨ࣌ʹɺϗετͷ໰୊͔ɺ ͦΕͱ΋ॻ͔Ε͍ͯΔίʔυ͕ո͍͠ʁ ͱ͔ߟ͑ͳ͍ͱ͍͚ͳ͍ɻ • ৭Μͳcron͕ಈ͍͍ͯΔέʔε͕͋ͬͯɺ Πϯελϯε্ཱͪ͛௚͚ͩ͢Ͱ͸ͩΊͩͬͨΓ͢Δɻ ඍົʹಈ͍͍ͯΔόονॲཧ͕ډͨΓ

෮ݩͷ࢓ํ஌ͬͯΔਓډΔʁ • Ͳ͏΍ͬͯಈ͍͍ͯΔ͔ͷ஌͕ࣝɺ ଐਓԽ͍ͯ͠Δέʔε͕ଟ͍ɻ • ্ཱͪ͛௚͢ΦϖϨʔγϣϯΛɺ ීஈ΍Βͳ͍ͷͰɺૉૣ͘ग़དྷͳ͍ɻ • ίʔυԽ͞Ε͍ͯͳ͍෦෼͕͋ͬͨΓ͢Δɻ

όονॲཧͰ΍Γ͍ͨ͜ͱ ఆظతʹಈ͘ॲཧΛɺ ͨͩॻ͖͍͚ͨͩͳΜ͡Όɻ ཉΛݴ͏ͱϗετΛؾʹͨ͘͠ͳ͍ɻ

खܰʹѻ͑Δج൫ʹ͍ͨ͠

όοναʔόʔ͸ Ͳ͏͍͏࢓ࣄΛ͍ͯ͠Δͷ͔

ௐ΂͍ͯ͘ͱόονॲཧ͸3ͭʹɺ ෼ྨग़དྷΔ͜ͱ͕Θ͔ͬͨɻ

৭Μͳόονॲཧ • ஞ࣮࣍ߦ • ఆظ࣮ߦ • ࣮͸ґଘؔ܎͕͋Δ ͦΕͧΕͷόονॲཧ͕ɺ ॏෳͯ͠ಈ͔ͳ͍͜ͱΛظ଴͍ͯ͠Δɻ

৭Μͳόονॲཧ • ஞ࣮࣍ߦ • ఆظ࣮ߦ • ࣮͸ґଘؔ܎͕͋Δ ͦΕͧΕͷόονॲཧ͕ɺ ॏෳͯ͠ಈ͔ͳ͍͜ͱΛظ଴͍ͯ͠Δɻ ͍·"84Ͱఏڙ͞ΕͯΔαʔϏεͳΒɺ ΋ͬͱѻ͍΍͍͢ج൫ʹग़དྷΔͷͰ͸ʁ

৽͍͠ج൫ʹ࢖͑ͦ͏ͳαʔϏε • ECS(+Fargate) • Step Functions

ECS • Amazon Elastic Container Service (Amazon ECS) ͸ɺ ׬શϚωʔδυܕͷίϯςφΦʔέετϨʔγϣϯ αʔϏεͰ͢ɻ https://aws.amazon.com/jp/ecs/ • Fargateͱ૊Έ߹ΘͤΔͱɺ ϗετͳ͠Ͱར༻ՄೳʹͳΔɻʢαʔόʔϨεʣ

ECSͷಈ࡞Πϝʔδ

{ "executionRoleArn": "arn:aws:iam::000000:role/execRole", "containerDefinitions": [ { "logConfiguration": { "logDriver": "awslogs", "options": {...} }, "command": [ "make", "run" ], "image": "00000.dkr.ecr.ap-northeast-1.amazonaws.com/batch:production", "name": "hoge" } ], "memory": "2048", "taskRoleArn": "arn:aws:iam::895849419934:role/taskRole", "family": "hoge", "requiresCompatibilities": [ "FARGATE" ], "networkMode": "awsvpc", "cpu": "512", "volumes": [] } taskఆٛྫ aws ecs register-task-definition --cli-input-json file://task.json

Fargateʹ͍ͭͯ

Fargateͷྑ͍ͱ͜Ζ 2018೥12݄ࠒ͔Βɺஈ֊తʹόονॲཧΛFargate ʹҠߦͯ͠·͕ͨ͠ɺ FargateͰࠔͬͨͱ͍͏έʔε͸͍·ͷͱ͜Ζͳ͍ɻ αʔόʔϨεͳͷͰɺϗετͷෆௐͰ೰·͞ΕΔ έʔε͕ͳ͘ͳΓɺτϥϒϧγϡʔςΟϯά͸ ҎલΑΓ΋γϯϓϧʹͳΓ·ͨ͠ɻ

Fargate͕߹Θͳ͍έʔε ىಈ͕࣌ؒEC2্Ͱಈ͔͢ECSΑΓ͸஗͘ͳΔͷͰɺ ىಈ͕࣌ؒॏཁͳόονॲཧͰ͸͓͢͢Ί͠·ͤΜɻ τʔλϧͷ஗͘ͳΔ౓߹͍͸ΠϝʔδαΠζʹ΋ ࠨӈ͞ΕΔͷͰɺ֤؀ڥͰݕূͯ͠Έ͍ͯͩ͘͞ɻ

Step Functions • AWS LambdaɺAWS Fargate ͓Αͼ Amazon SageMaker ͳͲͷαʔϏεΛͭͳ͛ͯ ػೳ๛෋ͳΞϓϦέʔγϣϯʹ·ͱΊΔ ϫʔΫϑϩʔΛઃܭ࣮ͯ͠ߦͰ͖·͢ɻ https://aws.amazon.com/jp/step-functions/

Step Functionsͷಈ࡞Πϝʔδ

͜Ε͍͚ΔͷͰ͸ʁ

αʔόʔϨεҠߦ

৭Μͳόονॲཧʢ࠶ܝʣ • ஞ࣮࣍ߦ • ఆظ࣮ߦ • ࣮͸ґଘؔ܎͕͋Δ ͦΕͧΕͷόονॲཧ͕ɺ ॏෳͯ͠ಈ͔ͳ͍͜ͱΛظ଴͍ͯ͠Δɻ

ஞ࣮࣍ߦ

ஞ࣮࣍ߦͱ͸ όονॲཧΛͳΔ͸΍Ͱ࣮ߦ͢Δɻ cron࣮૷ྫ * * * * * * cron-user flock -w 1 hoge.lock make run

No content

ECS Service • ࢦఆͨ͠਺ͷΠϯελϯεΛಉ࣌ʹ࣮ߦͯ͠ ҡ࣋Ͱ͖·͢ɻ https://docs.aws.amazon.com/ja_jp/ AmazonECS/latest/developerguide/ ecs_services.html

ECS ServiceͷྲྀΕɿඞཁ਺1ͷྫ 1. λεΫ਺͕0ͳͨΊɺAλεΫ͕౤ೖ͞Εɺ λεΫ਺͕1ʹͳΔɻ 2. AλεΫ͕ॲཧΛऴ͑Δɻ λεΫ਺͕0ʹͳΔɻ 3. λεΫ਺͕0ͳͨΊɺAλεΫ͕౤ೖ͞ΕΔry ͜Ε͕܁Γฦ͞ΕΔɻ

{ "cluster": "batch", "taskDefinition": "arn:aws:ecs:ap-northeast-1:000000:task-definition/hoge", "networkConfiguration": { "awsvpcConfiguration": { "assignPublicIp": "ENABLED", "securityGroups": [ "sg-hoge" ], "subnets": [ "subnet-hoge" ] } }, "desiredCount": 1ɹඞཁ਺ } Serviceͷఆٛྫ aws ecs create-service --service-name hoge \ --launch-type FARGATE --cli-input-json file://service.json

͜ͷύλʔϯͷϝϦοτ • ͳΔ΂͘ૣ࣮͘ߦ͢Δ͕γϯϓϧʹදݱͰ͖Δɻ

ఆظ࣮ߦ

ఆظ࣮ߦͱ͸ ࢦఆ࣌ؒʹόονॲཧΛ࣮ߦ͍ͨ͠ɻ cron࣮૷ྫ 45 * * * * * cron-user flock -w 1 hoge.lock make run

No content

Cloud Watch Events ECS Task Schedule • cron ϥΠΫͳεέδϡʔϧͰͷ λεΫͷ࣮ߦ͕Մೳɻ https://docs.aws.amazon.com/ja_jp/ AmazonECS/latest/developerguide/ scheduling_tasks.html

εέδϡʔϥʔͷઃఆ 3VMF ͍࣮ͭߦ͢Δ͔ 5BSHFUT ԿΛ͢Δ͔ 3VMFʹ5BSHFUTΛඥ෇͚Δ

{ "Name": "hoge", "ScheduleExpression": "cron(35 * * * ? *)", "State": "ENABLED", "Description": "ίϝϯτ" } Rule aws events put-rule --name hoge --cli-input-json file://rule.json

{ "Targets": [ { "Id": "hoge", "Arn": "arn:aws:ecs:ap-northeast-1:00000:cluster/batch", "RoleArn": "arn:aws:iam::895849419934:role/ecsEventsRole", "Input": "{}", "EcsParameters": { "TaskDefinitionArn": "arn:aws:ecs:ap-northeast-1:000000:task-definition/hoge", "TaskCount": 1, "LaunchType": "FARGATE", "NetworkConfiguration": { "awsvpcConfiguration": { "Subnets": [ "subnet-hoge" ], "SecurityGroups": [ "sg-hoge" ], "AssignPublicIp": "ENABLED" } }, "PlatformVersion": "LATEST" } } ] } Targets aws events put-targets --rule hoge —cli-input-json file://targets.json

IUUQTEPDTBXTBNB[PODPNKB@KQ"NB[PO$MPVE8BUDIMBUFTUFWFOUT$8&@5SPVCMFTIPPUJOHIUNM3VMF5SJHHFSFE.PSF5IBO0ODF ෳ਺ճτϦΨʔ͞ΕΔ͜ͱ͕͋Δ

No content

dlock Go੡෼ࢄγεςϜ޲͚flockϥΠΫπʔϧ ҠߦݩͷόοναʔόʔͷॲཧΛͦͷ··ECSͳͲʹɺ ࣋ͬͯ͜ΕΔ༻ʹ։ൃ͞Εͨࣾ಺πʔϧɻ Lock؅ཧʹDynamoDBͷςʔϒϧΛ࢖༻͍ͯ͠Δɻ ςʔϒϧͷΩϟύγςΟͱ͔͸ؾʹ͠ͳͯ͘ྑ͍Α͏ʹΦϯσϚϯυΛ ࢖͍ͬͯ·͢ɻ ※ഉଞ੍ޚΛؾʹ͠ͳ͍͍࣮ͯ͘૷ʹ͢Δͷ͕ϕετͰ͢ɻ OSSʹ͸ͯ͠·ͤΜ͕ɺधཁ͕͋Γͦ͏ͳΒެ։΋ߟ͑ͯ΋ྑͦ͞͏ɻ

dlock λεΫͷίϚϯυͰҎԼΛ࣮ߦ͢Δ EMPDLŠSFHJPOBQOPSUIFBTUSVOEMPDLIPHFMPDLNBLFSVO IPHF

͜ͷύλʔϯͷϝϦοτ • cronΛ࢖ͬͨόονॲཧΛɺͦͷ··࣋ͬͯ͜ΕΔɻ • dlockͱ૊Έ߹ΘͤΕ͹ɺഉଞॲཧ΋Մೳɻ

࣮͸ґଘؔ܎͕͋Δ

࣮͸ґଘؔ܎͕͋Δ ॻ͔Ε͍ͯΔcron Aόον͸ຖ࣌20෼։࢝ Bόον͸ຖ࣌40෼։࢝ "όονॲཧ #όονॲཧ "όονॲཧͷ݁ՌΛݩʹॲཧ͢Δ DSPOͰ͸ݟ͑ͳ͍͚ͲɺཪͰ͸ґଘ͍ͯ͠Δ

࣮͸ґଘؔ܎͕͋Δ ૉ௚ʹcronͰґଘؔ܎Λදݱ͢Δͱɺ ґଘ͍ͯ͠Δόονॲཧ͕མͪΔͱɺޙஈͷόονॲཧ ΋མͪͯɺͦͷޙஈ΋མͪͯͱͳΔɻ ·ͨɺόονॲཧͷґଘؔ܎ʹৄ͘͠ͳ͍ͱɺ ॲཧ͠௚͠खॱ͕Θ͔Βͳ͍ɻ ࣗಈͰ͍͍ײ͡ʹ͢Δʹͯ͠΋ɺ ϦτϥΠͷ࢓૊ΈΛ࡞Δͱ͔͠ͳ͍ͱ͍͚ͳ͍ɻ

No content

͜ͷύλʔϯͷϝϦοτ • δϣϒͷґଘؔ܎Λૉ௚ʹදݱͰ͖Δɻ • ϦτϥΠ΍Τϥʔॲཧ͸StepFucntionsͷඪ४ػೳ Λ࢖͑͹؆୯ʹ࣮૷Մೳɻ https://docs.aws.amazon.com/ja_jp/step-functions/latest/dg/concepts-error-handling.html "Retry": [ { "ErrorEquals": [ "States.Timeout" ], "IntervalSeconds": 3, ࠷ॳͷ࠶ࢼߦલͷඵ਺ "MaxAttempts": 2,ɹ࠶ࢼߦͷ࠷େճ਺ "BackoffRate": 1.5ɹ֤ࢼߦؒͰ࠶ࢼߦִ͕ؒ૿Ճ͢Δ৐਺ } ]

৭Μͳόονॲཧʢ࠶ܝʣ • ஞ࣮࣍ߦ • ఆظ࣮ߦ • ࣮͸ґଘؔ܎͕͋Δ ͦΕͧΕͷόονॲཧ͕ɺ ॏෳͯ͠ಈ͔ͳ͍͜ͱΛظ଴͍ͯ͠Δɻ

αʔόʔϨεͰ͍͚Δ΍Μ

؂ࢹ

ؾʹͳΔͱ͜Ζ • ECSͷ࣮ߦΤϥʔൃੜ͍ͯ͠ͳ͍͔ɻ • StepFunctionsͷεςʔτϚγϯͷ ϫʔΫϑϩʔ్͕தͰࣦഊ͍ͯ͠Δ͔ɻ • CloudWatch Events͕ECS TaskΛ invokeग़དྷ͍ͯΔ͔ɻ

ECSͷ࣮ߦΤϥʔൃੜ͍ͯ͠ͳ͍͔

ECSͷ؂ࢹ

Resources: Func: Type: AWS::Serverless::Function Properties: Runtime: python3.8 MemorySize: 128 Timeout: 60 Handler: main.handle Role: !GetAtt FuncRole.Arn CodeUri: ./ Environment: Variables: SLACK_SUCCESS_CHANNEL: !Ref SlackSuccessChannel SLACK_FAILURE_CHANNEL: !Ref SlackFailureChannel SLACK_HOOK_URL: !Ref SlackHookUrl Events: ECSTask: Type: CloudWatchEvent Properties: Pattern: source: - "aws.ecs" detail-type: - "ECS Task State Change" detail: lastStatus: - "STOPPED" ECSͷSTOPPEDΠϕϯτʢSAMʣ

{ ... "detail": { "clusterArn": "arn:aws:ecs:ap-northeast-1:951472794671:cluster/name", "containerInstanceArn": "arn:aws:ecs:ap-northeast-1:951472794671:container-instance/aaaaaaaa-bbbb-cccc-dddd- eeeeeeeeeeee", "containers": [ { "containerArn": "arn:aws:ecs:ap-northeast-1:951472794671:container/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", "exitCode": 0, codeΛݟΕ͹੒ޭ͔ͨ֬͠ೝՄೳ "lastStatus": "STOPPED", "name": "container_one", "taskArn": "arn:aws:ecs:ap-northeast-1:951472794671:task/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee" }, { "containerArn": "arn:aws:ecs:ap-northeast-1:951472794671:container/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", "lastStatus": "STOPPED", "name": “container_two", "reason": "CannotStartContainerError: API error (500): cannot start a stopped process: unknown\n",ɹࣦഊཧ༝ "taskArn": "arn:aws:ecs:ap-northeast-1:951472794671:task/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee" } ] } ... } ΠϕϯτͰඈΜͰ͘Δ৘ใ

ฐࣾͷྫ

StepFunctionsͷεςʔτϚγϯͷ ϫʔΫϑϩʔ్͕தͰࣦഊ͍ͯ͠Δ͔

StepFunctionsͷεςʔτϚγϯͷ ϫʔΫϑϩʔ్͕தͰࣦഊ͍ͯ͠Δ͔ ExecutionsFailedΛݟΕ͹ɺ్தͰࣦഊ͍ͯ͠Δ͜ͱ͕෼͔Δɻ ฐࣾͰ͸εςʔτϚγϯͷࡉ͔͍ঢ়ଶભҠ͸ݟ͍ͯ·ͤΜ͕ɺ CloudWatch Eventsܦ༝Ͱऔಘ͢Δ͜ͱ΋ՄೳͰ͢ɻ

CloudWatch Events͕ECS TaskΛ invokeग़དྷ͍ͯΔ͔

CloudWatch Events͕ECS TaskΛ invokeग़དྷ͍ͯΔ͔ FailedInvocationͰInvokeͷࣦഊʹؾͮ͘͜ͱ͕ग़དྷ·͢ɻ Ͳ͏͍͏έʔεͰinvokeࣦഊ͢Δ͔ͱ͍͏ͱɺ TargetsͷRole ArnͷRoleͷݖݶ͕଍Γͯͳ͍ͱ͔ɺ Role͕ͳ͘ͳͬͯΔͱ͔Ͱ͢ɻ Ξϥʔτແ͍͔Β҆ఆͯ͠ΔͳͬͯࢥͬͨΒɺInvokeࣦഊͯͨ͠ ͱ͔স͑ͳ͍Ͱ͔͢ΒͶʢࣗ෼͸΍ͬͯ͠·ͬͨʣɻ

ӡ༻ͯ͠ΈͯͲ͏͔ͩͬͨ

ಘΒΕͨϝϦοτ

ಘΒΕͨϝϦοτ • ৗʹ࢖͍ࣺͯ͢Δ࣮ߦ؀ڥΛ࣮ݱͰ͖ͨɻ 2019೥8݄23೔ͷEC2ো֐࣌ʹҰ࣌తʹ ΤϥʔʹͳΓͭͭ΋ɺࣗಈͰ෮چͨ͠ɻ https://www.itmedia.co.jp/news/articles/1908/28/news127_2.html • ECSɺStepFunctions͑͞஌͍ͬͯΕ͹ڍಈ͕Θ͔Δɻ

஍ຯخ͍͠ • 1όονॲཧɺ1ϩʔϧʹͳͬͨͷͰɺݖݶ͕࠷௿ݶ෇༩ग़དྷΔɻ • όονॲཧʹ߹ΘͤͯɺϚγϯύϫʔͷνϡʔχϯά͕ग़དྷΔɻ • ϑϧϚωʔδυ͔ͩΒɺԿ΋͠ͳͯ͘΋ศརʹͳ͍ͬͯ͘ɻ • ଞͷόονॲཧ͕Ͳ͏ͱ͔ؾʹ͠ͳͯ͘Α͍ɻ

όονॲཧ͸αʔόʔϨεͰ ग़དྷΔ࣌୅Ͱ͢ʂ

·ͱΊ

αʔόʔϨε࠷ߴ

ฐࣾͰ͸ ΤϯδχΞ࠾༻ͯ͠·͢ʂ

https://techlog.voyagegroup.com/entry/2019/02/04/171325

pei0804ͷ࠷ۙͷ࢓ࣄ • DSPͷػೳ։ൃ Ͳ͏࡞Δ͔ΒɺίʔσΟϯάɺӡ༻·Ͱ Ұؾ௨؏ͯ͠΍ͬͯ·͢ɻ • ηΩϡϦςΟ੔උ ηΩϡϦςΟΞΧ΢ϯτ࡞੒ͨ͠Γɺ GuardDutyಋೖ΍CISϕϯνϚʔΫʹ߹Θͤͨ ؀ڥ੔උ΍ࣾ಺޲͚ηΩϡϦςΟษڧձͷ։࠵ɻ

https://note.com/ryosuke_kawamura/n/nb5fc4d34a7c8