Orchestrating your Lambda Workloads with AWS Step Functions

Slide 1

Slide 1 text

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Who is Datadog SaaS-based Monitoring & Analytics Infrastructure, APM, Logs Open Source Agent Trillions of data points per day Booth 109 in Aria Booth 1021 in Venetian We are hiring!!

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Other things learned over the last year Serverless is about Automation Serverless is about event-driven computing Scalability is different

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Slide 29

Slide 29 text

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • State Machine • Transitions • Executions • States • Task • Choice • Succeed/Fail • Pass • Wait • Parallel You define the State Machine with the Amazon States Language A diagram will be generated based on that structure AWS Step Function Concepts @technovangelist

Slide 30

Slide 30 text

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Resource * ResultPath Retry Catch TimeoutSeconds HeartbeatSeconds A task can be a Lambda or an ‘activity’ Type * Next End Comment InputPath OutputPath AWS Step Function States: Task @technovangelist

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Seconds Timestamp SecondsPath TimestampPath Type Next End Comment InputPath OutputPath AWS Step Function States: Wait @technovangelist

Slide 39

Slide 39 text

Slide 40

Slide 40 text

Slide 41

Slide 41 text

Slide 42

Slide 42 text

Slide 43

Slide 43 text

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Benefits of Serverless Framework Define everything in a single file: • IAM Role Statements • Lambda Function Handlers • Lambda Function Triggers • Step Function States • CloudFormation for other things

Slide 44

Slide 44 text

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon ECR AWS Lambda Amazon S3 Amazon DynamoDB Amazon CloudFront AWS CodeBuild AWS CodeCommit AWS CloudFormation IAM Amazon API Gateway* AWS Step Functions Amazon SNS

Slide 45

Slide 45 text

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. serverless.yml service: name: mattw-reinvent2017-build-website plugins: - serverless-pseudo-parameters - serverless-step-functions - serverless-webpack custom: repoName: reinvent2017-website

Slide 46

Slide 46 text

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. serverless.yml cont’d provider: name: aws runtime: nodejs6.10 stage: ${opt:stage, 'dev'} region: ${opt:region, 'us-east-1'} iamRoleStatements: - Effect: "Allow" Action: - "states:StartExecution" Resource: - ${self:resources.Outputs.MyStateMachine.Value} - Effect: "Allow" Action: - "codecommit:GetCommit" - "codecommit:GetDifferences" Resource: - arn:aws:codecommit:${self:provider.region}:#{AWS::AccountId}:${…custom.repoName}

Slide 47

Slide 47 text

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. serverless.yml cont’d functions: buildSite: handler: handler.build whatChanged: handler: handler.whatChanged startStepFunction: handler: handler.startStepFunction events: - sns: arn:aws:sns:${…provider.region}:#{AWS::AccountId}:mattw-reinvent-websitebuild environment: statemachine_arn: ${self:resources.Outputs.MyStateMachine.Value}

Slide 48

Slide 48 text

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. serverless.yml cont’d stepFunctions: stateMachines: FirstStateMachine: name: myStateMachine definition: StartAt: WhatChanged States: WhatChanged: Type: Task Resource: arn:aws:lambda:#{AWS::Region}:#{AWS::AccountId}\ :function:${self:service}-${self:provider.stage}-whatChanged Next: BuildDockerOrNot

Slide 49

Slide 49 text

Slide 50

Slide 50 text

Slide 51

Slide 51 text

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. startStepFunction: handler: handler.startStepFunction events: - sns: arn:aws:sns…mattw-reinvent-websitebuild environment: statemachine_arn: ${self:reso…puts.MyStateMachine.Value}

Slide 52

Slide 52 text

Slide 53

Slide 53 text

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 1. What changed in the repo? 2. If docker-related, build docker 1. Tell CodeBuild to rebuild docker image 2. Wait for it 3. Push to ECS repo 3. Use the docker image to build the website 4. Wait for it 5. Send Matt an SMS

Slide 54

Slide 54 text

Slide 55

Slide 55 text

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. WhatChanged: Type: Task Resource: arn:aws:lambda:... Next: BuildDockerOrNot Retry : - ErrorEquals: - Lambda.Unknown States.ALL IntervalSeconds: 3 MaxAttempts: 4

Slide 56

Slide 56 text

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. WhatChanged Get the commit that triggered the sf Get the previous commit Whats different If it was docker related return hasDocker = true else return hasDocker = false

Slide 57

Slide 57 text

Slide 58

Slide 58 text

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. BuildDockerOrNot: Type: Choice Choices: - Variable: "$.hasDocker" BooleanEquals: true Next: BuildDocker - Variable: "$.hasDocker" BooleanEquals: false Next: BuildSite

Slide 59

Slide 59 text

Slide 60

Slide 60 text

Slide 61

Slide 61 text

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. phases: install: commands: - nohup /usr/local/bin/dockerd -G dockremap --host=unix:///… - timeout -t 15 sh -c "until docker info; do echo .; sleep 1; done" pre_build: commands: - echo Logging in to Amazon ECR... - $(aws ecr get-login --no-include-email --region us-east-1) build: commands: - echo Build started on `date` - echo Building the Docker image... - docker build -t mattw-stepfunction-demo:latest . - docker tag mattw-stepfunction-demo:latest ...dkr.ecr.us-east-1.amazona... post_build: commands: - echo Build completed on `date` - echo Pushing the Docker image... - docker push ...dkr.ecr.us-east-1.amazonaws.com/mattw-stepfunc...

Slide 62

Slide 62 text

Slide 63

Slide 63 text

Slide 64

Slide 64 text

Slide 65

Slide 65 text

Slide 66

Slide 66 text

Slide 67

Slide 67 text

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ContainerBuiltOrNot: Type: Choice Choices: - Variable: "$.containerBuildDone" BooleanEquals: true Next: BuildSite - Variable: "$.containerBuildDone" BooleanEquals: false Next: WaitForDockerBuild

Slide 68

Slide 68 text

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. phases: build: commands: - echo "**************** In Build ******************" - aws s3 sync s3://mattw-reinvent2017-rawsitepages src/posts - export GATSBY_BUILDTIME=$(tail -1 .git/logs/HEAD | awk '{print $5}') - ln -s /backupmod/node_modules ./node_modules - gatsby build - echo "************* Build Complete ****************" post_build: commands: - echo "************** In Post Build *****************" - aws s3 sync public s3://mybucket --delete --acl public-read - echo "*********** Post Build Complete **************" # - command artifacts: files: - public/**/*

Slide 69

Slide 69 text

Slide 70

Slide 70 text

Slide 71

Slide 71 text

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What are the Lambda Metrics aws.lambda.duration, min, max, sum aws.lambda.errors aws.lambda.invocations aws.lambda.throttles aws.lambda.iterator_age

Slide 72

Slide 72 text

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What are the Step Function Metrics aws.states.execution_time (.maximum, .minimum) aws.states.executions_aborted, failed, started, succeeded, timed_out aws.states.lambda_functions_scheduled, started, succeeded, timed_out aws.states.execution_throttled aws.states.lambda_function_time (.maximum, .minimum) aws.states.lambda_function_run_time (.maximum, .minimum) aws.states.lambda_function_schedule_time (.maximum, .minimum) aws.states.lambda_functions_started

Slide 73

Slide 73 text

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Summary Step Functions make AWS Lambda orchestration easy Lambda and Step Functions can be an interesting part of your infrastructure Monitor your Lambda functions to ensure peak performance Source can be found at: https://github.com/DataDog/mattw-reinvent2017-demo

Slide 74

Slide 74 text