Upgrade to Pro — share decks privately, control downloads, hide ads and more …

広告配信を支えるバッチ基盤をサーバーレス移行した話(ECS Fargate, Step Fun...

pei0804
February 27, 2020

広告配信を支えるバッチ基盤をサーバーレス移行した話(ECS Fargate, Step Functions)@ Serverless Meetup Tokyo #16

pei0804

February 27, 2020
Tweet

More Decks by pei0804

Other Decks in Technology

Transcript

  1. Ҡߦݩͷόοναʔόʔ      DSPOVTFSUJNFPVUqPDLXIPHFMPDLNBLFSVOIPHF   

      DSPOVTFSUJNFPVUqPDLXGVHBMPDLNBLFSVOGVHB      DSPOVTFSUJNFPVUqPDLXGVHBMPDLNBLFSVOGPP ࡉ͔͍࣮૷ͷ࢓ํ͸ɺνʔϜʹΑͬͯҧ͏ͱࢥ͍·͢ɻ
  2. ECS • Amazon Elastic Container Service (Amazon ECS) ͸ɺ ׬શϚωʔδυܕͷίϯςφΦʔέετϨʔγϣϯ

    αʔϏεͰ͢ɻ https://aws.amazon.com/jp/ecs/ • Fargateͱ૊Έ߹ΘͤΔͱɺ ϗετͳ͠Ͱར༻ՄೳʹͳΔɻʢαʔόʔϨεʣ
  3. { "executionRoleArn": "arn:aws:iam::000000:role/execRole", "containerDefinitions": [ { "logConfiguration": { "logDriver": "awslogs",

    "options": {...} }, "command": [ "make", "run" ], "image": "00000.dkr.ecr.ap-northeast-1.amazonaws.com/batch:production", "name": "hoge" } ], "memory": "2048", "taskRoleArn": "arn:aws:iam::895849419934:role/taskRole", "family": "hoge", "requiresCompatibilities": [ "FARGATE" ], "networkMode": "awsvpc", "cpu": "512", "volumes": [] } taskఆٛྫ aws ecs register-task-definition --cli-input-json file://task.json
  4. Step Functions • AWS LambdaɺAWS Fargate ͓Αͼ Amazon SageMaker ͳͲͷαʔϏεΛͭͳ͛ͯ

    ػೳ๛෋ͳΞϓϦέʔγϣϯʹ·ͱΊΔ ϫʔΫϑϩʔΛઃܭ࣮ͯ͠ߦͰ͖·͢ɻ https://aws.amazon.com/jp/step-functions/
  5. { "cluster": "batch", "taskDefinition": "arn:aws:ecs:ap-northeast-1:000000:task-definition/hoge", "networkConfiguration": { "awsvpcConfiguration": { "assignPublicIp":

    "ENABLED", "securityGroups": [ "sg-hoge" ], "subnets": [ "subnet-hoge" ] } }, "desiredCount": 1ɹඞཁ਺ } Serviceͷఆٛྫ aws ecs create-service --service-name hoge \ --launch-type FARGATE --cli-input-json file://service.json
  6. Cloud Watch Events ECS Task Schedule • cron ϥΠΫͳεέδϡʔϧͰͷ λεΫͷ࣮ߦ͕Մೳɻ

    https://docs.aws.amazon.com/ja_jp/ AmazonECS/latest/developerguide/ scheduling_tasks.html
  7. { "Name": "hoge", "ScheduleExpression": "cron(35 * * * ? *)",

    "State": "ENABLED", "Description": "ίϝϯτ" } Rule aws events put-rule --name hoge --cli-input-json file://rule.json
  8. { "Targets": [ { "Id": "hoge", "Arn": "arn:aws:ecs:ap-northeast-1:00000:cluster/batch", "RoleArn": "arn:aws:iam::895849419934:role/ecsEventsRole",

    "Input": "{}", "EcsParameters": { "TaskDefinitionArn": "arn:aws:ecs:ap-northeast-1:000000:task-definition/hoge", "TaskCount": 1, "LaunchType": "FARGATE", "NetworkConfiguration": { "awsvpcConfiguration": { "Subnets": [ "subnet-hoge" ], "SecurityGroups": [ "sg-hoge" ], "AssignPublicIp": "ENABLED" } }, "PlatformVersion": "LATEST" } } ] } Targets aws events put-targets --rule hoge —cli-input-json file://targets.json
  9. ͜ͷύλʔϯͷϝϦοτ • δϣϒͷґଘؔ܎Λૉ௚ʹදݱͰ͖Δɻ • ϦτϥΠ΍Τϥʔॲཧ͸StepFucntionsͷඪ४ػೳ Λ࢖͑͹؆୯ʹ࣮૷Մೳɻ https://docs.aws.amazon.com/ja_jp/step-functions/latest/dg/concepts-error-handling.html "Retry": [ {

    "ErrorEquals": [ "States.Timeout" ], "IntervalSeconds": 3, ࠷ॳͷ࠶ࢼߦલͷඵ਺ "MaxAttempts": 2,ɹ࠶ࢼߦͷ࠷େճ਺ "BackoffRate": 1.5ɹ֤ࢼߦؒͰ࠶ࢼߦִ͕ؒ૿Ճ͢Δ৐਺ } ]
  10. Resources: Func: Type: AWS::Serverless::Function Properties: Runtime: python3.8 MemorySize: 128 Timeout:

    60 Handler: main.handle Role: !GetAtt FuncRole.Arn CodeUri: ./ Environment: Variables: SLACK_SUCCESS_CHANNEL: !Ref SlackSuccessChannel SLACK_FAILURE_CHANNEL: !Ref SlackFailureChannel SLACK_HOOK_URL: !Ref SlackHookUrl Events: ECSTask: Type: CloudWatchEvent Properties: Pattern: source: - "aws.ecs" detail-type: - "ECS Task State Change" detail: lastStatus: - "STOPPED" ECSͷSTOPPEDΠϕϯτʢSAMʣ
  11. { ... "detail": { "clusterArn": "arn:aws:ecs:ap-northeast-1:951472794671:cluster/name", "containerInstanceArn": "arn:aws:ecs:ap-northeast-1:951472794671:container-instance/aaaaaaaa-bbbb-cccc-dddd- eeeeeeeeeeee", "containers":

    [ { "containerArn": "arn:aws:ecs:ap-northeast-1:951472794671:container/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", "exitCode": 0, codeΛݟΕ͹੒ޭ͔ͨ֬͠ೝՄೳ "lastStatus": "STOPPED", "name": "container_one", "taskArn": "arn:aws:ecs:ap-northeast-1:951472794671:task/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee" }, { "containerArn": "arn:aws:ecs:ap-northeast-1:951472794671:container/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", "lastStatus": "STOPPED", "name": “container_two", "reason": "CannotStartContainerError: API error (500): cannot start a stopped process: unknown\n",ɹࣦഊཧ༝ "taskArn": "arn:aws:ecs:ap-northeast-1:951472794671:task/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee" } ] } ... } ΠϕϯτͰඈΜͰ͘Δ৘ใ