Zucks inc. (VOYAGE GROUP inc.) https://serverless.connpass.com/event/165352/ https://youtu.be/dSoIQhobDb8?t=8021
ࠂ৴Λࢧ͑Δόονج൫ΛαʔόʔϨεҠߦͨ͠ECS Fargate, Step FunctionsServerless Meetup Tokyo #162020/02/27
View Slide
ۙ३ฏ pei0804Zucks(VOYAGE GROUP)ΤϯδχΞ2018ʹ৽ଔೖࣾ͠ɺӡ༻͔Β։ൃΛͯ͠·͢ɻ࠷ۙνʔϜΛԣஅͨ͠ηΩϡϦςΟؔ࿈ͷඋDSP։ൃΛͬͯ·͢ɻ͖ͳαʔϏεɺϑϧϚωʔδυαʔϏεͰ͢ɻ
ΞδΣϯμ• Ҡߦݩͷόοναʔόʔͷ՝• αʔόʔϨεҠߦ• ࣮• ࢹ• ӡ༻ͯ͠ΈͯͲ͏͔ͩͬͨ
Ҡߦݩͷόοναʔόʔͷ՝
ҠߦݩͷόοναʔόʔDSPOVTFSUJNFPVUqPDLXIPHFMPDLNBLFSVOIPHFDSPOVTFSUJNFPVUqPDLXGVHBMPDLNBLFSVOGVHBDSPOVTFSUJNFPVUqPDLXGVHBMPDLNBLFSVOGPPࡉ͔͍࣮ͷํɺνʔϜʹΑͬͯҧ͏ͱࢥ͍·͢ɻ
ӡ༻͕͘͠ͳΓ͕ͪ
ӡ༻͕͍͠ཁҼ• ൃੜ࣌ʹߟ͑Δ͜ͱ͕ଟ͍ɻ• ෮ݩͷํͬͯΔਓډΔʁ
ൃੜ࣌ʹߟ͑Δ͜ͱ͕ଟ͍• όονॲཧ͕མͪͨ࣌ʹɺϗετͷ͔ɺͦΕͱॻ͔Ε͍ͯΔίʔυ͕ո͍͠ʁͱ͔ߟ͑ͳ͍ͱ͍͚ͳ͍ɻ• ৭Μͳcron͕ಈ͍͍ͯΔέʔε͕͋ͬͯɺΠϯελϯε্ཱ͚ͪ͛ͩ͢ͰͩΊͩͬͨΓ͢Δɻඍົʹಈ͍͍ͯΔόονॲཧ͕ډͨΓ
෮ݩͷํͬͯΔਓډΔʁ• Ͳ͏ͬͯಈ͍͍ͯΔ͔ͷ͕ࣝɺଐਓԽ͍ͯ͠Δέʔε͕ଟ͍ɻ• ্ཱͪ͛͢ΦϖϨʔγϣϯΛɺීஈΒͳ͍ͷͰɺૉૣ͘ग़དྷͳ͍ɻ• ίʔυԽ͞Ε͍ͯͳ͍෦͕͋ͬͨΓ͢Δɻ
όονॲཧͰΓ͍ͨ͜ͱఆظతʹಈ͘ॲཧΛɺͨͩॻ͖͍͚ͨͩͳΜ͡ΌɻཉΛݴ͏ͱϗετΛؾʹͨ͘͠ͳ͍ɻ
खܰʹѻ͑Δج൫ʹ͍ͨ͠
όοναʔόʔͲ͏͍͏ࣄΛ͍ͯ͠Δͷ͔
ௐ͍ͯ͘ͱόονॲཧ3ͭʹɺྨग़དྷΔ͜ͱ͕Θ͔ͬͨɻ
৭Μͳόονॲཧ• ஞ࣮࣍ߦ• ఆظ࣮ߦ• ࣮ґଘ͕ؔ͋ΔͦΕͧΕͷόονॲཧ͕ɺॏෳͯ͠ಈ͔ͳ͍͜ͱΛظ͍ͯ͠Δɻ
৭Μͳόονॲཧ• ஞ࣮࣍ߦ• ఆظ࣮ߦ• ࣮ґଘ͕ؔ͋ΔͦΕͧΕͷόονॲཧ͕ɺॏෳͯ͠ಈ͔ͳ͍͜ͱΛظ͍ͯ͠Δɻ͍·"84Ͱఏڙ͞ΕͯΔαʔϏεͳΒɺͬͱѻ͍͍͢ج൫ʹग़དྷΔͷͰʁ
৽͍͠ج൫ʹ͑ͦ͏ͳαʔϏε• ECS(+Fargate)• Step Functions
ECS• Amazon Elastic Container Service (Amazon ECS) ɺશϚωʔδυܕͷίϯςφΦʔέετϨʔγϣϯαʔϏεͰ͢ɻhttps://aws.amazon.com/jp/ecs/• FargateͱΈ߹ΘͤΔͱɺϗετͳ͠Ͱར༻ՄೳʹͳΔɻʢαʔόʔϨεʣ
ECSͷಈ࡞Πϝʔδ
{"executionRoleArn": "arn:aws:iam::000000:role/execRole","containerDefinitions": [{"logConfiguration": {"logDriver": "awslogs","options": {...}},"command": ["make","run"],"image": "00000.dkr.ecr.ap-northeast-1.amazonaws.com/batch:production","name": "hoge"}],"memory": "2048","taskRoleArn": "arn:aws:iam::895849419934:role/taskRole","family": "hoge","requiresCompatibilities": ["FARGATE"],"networkMode": "awsvpc","cpu": "512","volumes": []}taskఆٛྫaws ecs register-task-definition --cli-input-json file://task.json
Fargateʹ͍ͭͯ
Fargateͷྑ͍ͱ͜Ζ201812݄ࠒ͔Βɺஈ֊తʹόονॲཧΛFargateʹҠߦͯ͠·͕ͨ͠ɺFargateͰࠔͬͨͱ͍͏έʔε͍·ͷͱ͜Ζͳ͍ɻαʔόʔϨεͳͷͰɺϗετͷෆௐͰ·͞ΕΔέʔε͕ͳ͘ͳΓɺτϥϒϧγϡʔςΟϯάҎલΑΓγϯϓϧʹͳΓ·ͨ͠ɻ
Fargate͕߹Θͳ͍έʔεىಈ͕࣌ؒEC2্Ͱಈ͔͢ECSΑΓ͘ͳΔͷͰɺىಈ͕࣌ؒॏཁͳόονॲཧͰ͓͢͢Ί͠·ͤΜɻτʔλϧͷ͘ͳΔ߹͍ΠϝʔδαΠζʹࠨӈ͞ΕΔͷͰɺ֤ڥͰݕূͯ͠Έ͍ͯͩ͘͞ɻ
Step Functions• AWS LambdaɺAWS Fargate ͓Αͼ AmazonSageMaker ͳͲͷαʔϏεΛͭͳ͛ͯػೳ๛ͳΞϓϦέʔγϣϯʹ·ͱΊΔϫʔΫϑϩʔΛઃܭ࣮ͯ͠ߦͰ͖·͢ɻhttps://aws.amazon.com/jp/step-functions/
Step Functionsͷಈ࡞Πϝʔδ
͜Ε͍͚ΔͷͰʁ
αʔόʔϨεҠߦ
৭Μͳόονॲཧʢ࠶ܝʣ• ஞ࣮࣍ߦ• ఆظ࣮ߦ• ࣮ґଘ͕ؔ͋ΔͦΕͧΕͷόονॲཧ͕ɺॏෳͯ͠ಈ͔ͳ͍͜ͱΛظ͍ͯ͠Δɻ
ஞ࣮࣍ߦ
ஞ࣮࣍ߦͱόονॲཧΛͳΔͰ࣮ߦ͢Δɻcron࣮ྫ* * * * * * cron-user flock -w 1 hoge.lock make run
ECS Service• ࢦఆͨ͠ͷΠϯελϯεΛಉ࣌ʹ࣮ߦͯ͠ҡ࣋Ͱ͖·͢ɻhttps://docs.aws.amazon.com/ja_jp/AmazonECS/latest/developerguide/ecs_services.html
ECS ServiceͷྲྀΕɿඞཁ1ͷྫ1. λεΫ͕0ͳͨΊɺAλεΫ͕ೖ͞ΕɺλεΫ͕1ʹͳΔɻ2. AλεΫ͕ॲཧΛऴ͑ΔɻλεΫ͕0ʹͳΔɻ3. λεΫ͕0ͳͨΊɺAλεΫ͕ೖ͞ΕΔry͜Ε͕܁Γฦ͞ΕΔɻ
{"cluster": "batch","taskDefinition": "arn:aws:ecs:ap-northeast-1:000000:task-definition/hoge","networkConfiguration": {"awsvpcConfiguration": {"assignPublicIp": "ENABLED","securityGroups": ["sg-hoge"],"subnets": ["subnet-hoge"]}},"desiredCount": 1ɹඞཁ}Serviceͷఆٛྫaws ecs create-service --service-name hoge \--launch-type FARGATE --cli-input-json file://service.json
͜ͷύλʔϯͷϝϦοτ• ͳΔ͘ૣ࣮͘ߦ͢Δ͕γϯϓϧʹදݱͰ͖Δɻ
ఆظ࣮ߦ
ఆظ࣮ߦͱࢦఆ࣌ؒʹόονॲཧΛ࣮ߦ͍ͨ͠ɻcron࣮ྫ45 * * * * * cron-user flock -w 1 hoge.lock make run
Cloud Watch EventsECS Task Schedule• cron ϥΠΫͳεέδϡʔϧͰͷλεΫͷ࣮ߦ͕Մೳɻhttps://docs.aws.amazon.com/ja_jp/AmazonECS/latest/developerguide/scheduling_tasks.html
εέδϡʔϥʔͷઃఆ3VMF͍࣮ͭߦ͢Δ͔5BSHFUTԿΛ͢Δ͔3VMFʹ5BSHFUTΛඥ͚Δ
{"Name": "hoge","ScheduleExpression": "cron(35 * * * ? *)","State": "ENABLED","Description": "ίϝϯτ"}Ruleaws events put-rule --name hoge --cli-input-json file://rule.json
{"Targets": [{"Id": "hoge","Arn": "arn:aws:ecs:ap-northeast-1:00000:cluster/batch","RoleArn": "arn:aws:iam::895849419934:role/ecsEventsRole","Input": "{}","EcsParameters": {"TaskDefinitionArn": "arn:aws:ecs:ap-northeast-1:000000:task-definition/hoge","TaskCount": 1,"LaunchType": "FARGATE","NetworkConfiguration": {"awsvpcConfiguration": {"Subnets": ["subnet-hoge"],"SecurityGroups": ["sg-hoge"],"AssignPublicIp": "ENABLED"}},"PlatformVersion": "LATEST"}}]}Targetsaws events put-targets --rule hoge —cli-input-json file://targets.json
IUUQTEPDTBXTBNB[[email protected]"NB[PO$MPVE8BUDIMBUFTUFWFOUT$8&@5SPVCMFTIPPUJOHIUNM3VMF5SJHHFSFE.PSF5IBO0ODFෳճτϦΨʔ͞ΕΔ͜ͱ͕͋Δ
dlockGoࢄγεςϜ͚flockϥΠΫπʔϧҠߦݩͷόοναʔόʔͷॲཧΛͦͷ··ECSͳͲʹɺ࣋ͬͯ͜ΕΔ༻ʹ։ൃ͞ΕͨࣾπʔϧɻLockཧʹDynamoDBͷςʔϒϧΛ༻͍ͯ͠ΔɻςʔϒϧͷΩϟύγςΟͱ͔ؾʹ͠ͳͯ͘ྑ͍Α͏ʹΦϯσϚϯυΛ͍ͬͯ·͢ɻ※ഉଞ੍ޚΛؾʹ͠ͳ͍͍࣮ͯ͘ʹ͢Δͷ͕ϕετͰ͢ɻOSSʹͯ͠·ͤΜ͕ɺधཁ͕͋Γͦ͏ͳΒެ։ߟ͑ͯྑͦ͞͏ɻ
dlockλεΫͷίϚϯυͰҎԼΛ࣮ߦ͢ΔEMPDLSFHJPOBQOPSUIFBTUSVOEMPDLIPHFMPDLNBLFSVOIPHF
͜ͷύλʔϯͷϝϦοτ• cronΛͬͨόονॲཧΛɺͦͷ··࣋ͬͯ͜ΕΔɻ• dlockͱΈ߹ΘͤΕɺഉଞॲཧՄೳɻ
࣮ґଘ͕ؔ͋Δ
࣮ґଘ͕ؔ͋Δॻ͔Ε͍ͯΔcronAόονຖ࣌20։࢝Bόονຖ࣌40։࢝"όονॲཧ#όονॲཧ"όονॲཧͷ݁ՌΛݩʹॲཧ͢ΔDSPOͰݟ͑ͳ͍͚ͲɺཪͰґଘ͍ͯ͠Δ
࣮ґଘ͕ؔ͋ΔૉʹcronͰґଘؔΛදݱ͢Δͱɺґଘ͍ͯ͠Δόονॲཧ͕མͪΔͱɺޙஈͷόονॲཧམͪͯɺͦͷޙஈམͪͯͱͳΔɻ·ͨɺόονॲཧͷґଘؔʹৄ͘͠ͳ͍ͱɺॲཧ͠͠खॱ͕Θ͔Βͳ͍ɻࣗಈͰ͍͍ײ͡ʹ͢Δʹͯ͠ɺϦτϥΠͷΈΛ࡞Δͱ͔͠ͳ͍ͱ͍͚ͳ͍ɻ
͜ͷύλʔϯͷϝϦοτ• δϣϒͷґଘؔΛૉʹදݱͰ͖Δɻ• ϦτϥΠΤϥʔॲཧStepFucntionsͷඪ४ػೳΛ͑؆୯ʹ࣮Մೳɻhttps://docs.aws.amazon.com/ja_jp/step-functions/latest/dg/concepts-error-handling.html"Retry": [ {"ErrorEquals": [ "States.Timeout" ],"IntervalSeconds": 3, ࠷ॳͷ࠶ࢼߦલͷඵ"MaxAttempts": 2,ɹ࠶ࢼߦͷ࠷େճ"BackoffRate": 1.5ɹ֤ࢼߦؒͰ࠶ࢼߦִ͕ؒ૿Ճ͢Δ} ]
αʔόʔϨεͰ͍͚ΔΜ
ࢹ
ؾʹͳΔͱ͜Ζ• ECSͷ࣮ߦΤϥʔൃੜ͍ͯ͠ͳ͍͔ɻ• StepFunctionsͷεςʔτϚγϯͷϫʔΫϑϩʔ్͕தͰࣦഊ͍ͯ͠Δ͔ɻ• CloudWatch Events͕ECS TaskΛinvokeग़དྷ͍ͯΔ͔ɻ
ECSͷ࣮ߦΤϥʔൃੜ͍ͯ͠ͳ͍͔
ECSͷࢹ
Resources:Func:Type: AWS::Serverless::FunctionProperties:Runtime: python3.8MemorySize: 128Timeout: 60Handler: main.handleRole: !GetAtt FuncRole.ArnCodeUri: ./Environment:Variables:SLACK_SUCCESS_CHANNEL: !Ref SlackSuccessChannelSLACK_FAILURE_CHANNEL: !Ref SlackFailureChannelSLACK_HOOK_URL: !Ref SlackHookUrlEvents:ECSTask:Type: CloudWatchEventProperties:Pattern:source:- "aws.ecs"detail-type:- "ECS Task State Change"detail:lastStatus:- "STOPPED"ECSͷSTOPPEDΠϕϯτʢSAMʣ
{..."detail": {"clusterArn": "arn:aws:ecs:ap-northeast-1:951472794671:cluster/name","containerInstanceArn": "arn:aws:ecs:ap-northeast-1:951472794671:container-instance/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee","containers": [{"containerArn": "arn:aws:ecs:ap-northeast-1:951472794671:container/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee","exitCode": 0, codeΛݟΕޭ͔ͨ֬͠ೝՄೳ"lastStatus": "STOPPED","name": "container_one","taskArn": "arn:aws:ecs:ap-northeast-1:951472794671:task/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"},{"containerArn": "arn:aws:ecs:ap-northeast-1:951472794671:container/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee","lastStatus": "STOPPED","name": “container_two","reason": "CannotStartContainerError: API error (500): cannot start a stopped process: unknown\n",ɹࣦഊཧ༝"taskArn": "arn:aws:ecs:ap-northeast-1:951472794671:task/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"}]} ...}ΠϕϯτͰඈΜͰ͘Δใ
ฐࣾͷྫ
StepFunctionsͷεςʔτϚγϯͷϫʔΫϑϩʔ్͕தͰࣦഊ͍ͯ͠Δ͔
StepFunctionsͷεςʔτϚγϯͷϫʔΫϑϩʔ్͕தͰࣦഊ͍ͯ͠Δ͔ExecutionsFailedΛݟΕɺ్தͰࣦഊ͍ͯ͠Δ͜ͱ͕͔ΔɻฐࣾͰεςʔτϚγϯͷࡉ͔͍ঢ়ଶભҠݟ͍ͯ·ͤΜ͕ɺCloudWatch Eventsܦ༝Ͱऔಘ͢Δ͜ͱՄೳͰ͢ɻ
CloudWatch Events͕ECS TaskΛinvokeग़དྷ͍ͯΔ͔
CloudWatch Events͕ECS TaskΛinvokeग़དྷ͍ͯΔ͔FailedInvocationͰInvokeͷࣦഊʹؾͮ͘͜ͱ͕ग़དྷ·͢ɻͲ͏͍͏έʔεͰinvokeࣦഊ͢Δ͔ͱ͍͏ͱɺTargetsͷRole ArnͷRoleͷݖݶ͕Γͯͳ͍ͱ͔ɺRole͕ͳ͘ͳͬͯΔͱ͔Ͱ͢ɻΞϥʔτແ͍͔Β҆ఆͯ͠ΔͳͬͯࢥͬͨΒɺInvokeࣦഊͯͨ͠ͱ͔স͑ͳ͍Ͱ͔͢ΒͶʢࣗͬͯ͠·ͬͨʣɻ
ӡ༻ͯ͠ΈͯͲ͏͔ͩͬͨ
ಘΒΕͨϝϦοτ
ಘΒΕͨϝϦοτ• ৗʹ͍ࣺͯ͢Δ࣮ߦڥΛ࣮ݱͰ͖ͨɻ20198݄23ͷEC2ো࣌ʹҰ࣌తʹΤϥʔʹͳΓͭͭɺࣗಈͰ෮چͨ͠ɻhttps://www.itmedia.co.jp/news/articles/1908/28/news127_2.html• ECSɺStepFunctions͍͑ͬͯ͞Εڍಈ͕Θ͔Δɻ
ຯخ͍͠• 1όονॲཧɺ1ϩʔϧʹͳͬͨͷͰɺݖݶ͕࠷ݶ༩ग़དྷΔɻ• όονॲཧʹ߹ΘͤͯɺϚγϯύϫʔͷνϡʔχϯά͕ग़དྷΔɻ• ϑϧϚωʔδυ͔ͩΒɺԿ͠ͳͯ͘ศརʹͳ͍ͬͯ͘ɻ• ଞͷόονॲཧ͕Ͳ͏ͱ͔ؾʹ͠ͳͯ͘Α͍ɻ
όονॲཧαʔόʔϨεͰग़དྷΔ࣌Ͱ͢ʂ
·ͱΊ
αʔόʔϨε࠷ߴ
ฐࣾͰΤϯδχΞ࠾༻ͯ͠·͢ʂ
https://techlog.voyagegroup.com/entry/2019/02/04/171325
pei0804ͷ࠷ۙͷࣄ• DSPͷػೳ։ൃͲ͏࡞Δ͔ΒɺίʔσΟϯάɺӡ༻·ͰҰؾ௨؏ͯͬͯ͠·͢ɻ• ηΩϡϦςΟඋηΩϡϦςΟΞΧϯτ࡞ͨ͠ΓɺGuardDutyಋೖCISϕϯνϚʔΫʹ߹Θͤͨڥඋ͚ࣾηΩϡϦςΟษڧձͷ։࠵ɻ
https://note.com/ryosuke_kawamura/n/nb5fc4d34a7c8
pei0804
hirokik56
takahashikazutaro
mako5656
oracle4engineer
kenmatsu4
oekazuma
eliasnogueira
ybrliiu
lhazy
soachr
lmi
oogfranz
marcduiker
mthomps
samanthasiow
samlambert
eileencodes
chriscoyier
sugarenia
tammielis
edds
paulrobertlloyd
deanohume
jponch