Orchestrating your Lambda Workloads with AWS Step Functions

Transcript

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Best Practices for Orchestrating AWS Lambda Workloads Matthew Williams Evangelist @ Datadog
2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Passport pics
5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. http://sf.technovangelist.com
9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Matt Williams Evangelist at Datadog @technovangelist mattw@datadoghq.com Organizer of DevOps Days Boston 2017 Who am I
10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Who is Datadog SaaS-based Monitoring & Analytics Infrastructure, APM, Logs Open Source Agent Trillions of data points per day Booth 109 in Aria Booth 1021 in Venetian We are hiring!!
11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Last year I talked about AWS Lambda…
12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. @technovangelist AWS Lambda
13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. …but AWS Lambda != ec2 @technovangelist
14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. …but AWS Lambda != application @technovangelist
15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. …applications are complicated @technovangelist
16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. …lambdas should respect the single responsibility principle… @technovangelist
17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. @technovangelist
19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Other things learned over the last year Serverless is about Automation Serverless is about event-driven computing Scalability is different
20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. we can create many serverless functions @technovangelist
21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. we can create many serverless functions but how do we orchestrate them? @technovangelist
22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Event/Message/Database as Traffic Cop Amazon Kinesis Amazon SQS Amazon DynamoDB @technovangelist
23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. https://github.com/Nordstrom/hello-retail
24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Enter AWS Step Functions @technovangelist
26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. a step function is @technovangelist
27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. a step function is a state machine @technovangelist
28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. but what about SWF? (Amazon Simple Workflow Service) @technovangelist
29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • State Machine • Transitions • Executions • States • Task • Choice • Succeed/Fail • Pass • Wait • Parallel You define the State Machine with the Amazon States Language A diagram will be generated based on that structure AWS Step Function Concepts @technovangelist
30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Resource * ResultPath Retry Catch TimeoutSeconds HeartbeatSeconds A task can be a Lambda or an ‘activity’ Type * Next End Comment InputPath OutputPath AWS Step Function States: Task @technovangelist
31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. State Example: Task
32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Choices Default Type Next End Comment InputPath OutputPath AWS Step Function States: Choice @technovangelist
33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. State Example: Choice
34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cause Error Type Next End Comment InputPath OutputPath AWS Step Function States: Succeed / Fail @technovangelist
35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. State Example: Succeed / Fail
36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Result ResultPath Type Next End Comment InputPath OutputPath AWS Step Function States: Pass @technovangelist
37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. State Example: Pass
38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Seconds Timestamp SecondsPath TimestampPath Type Next End Comment InputPath OutputPath AWS Step Function States: Wait @technovangelist
39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. State Example: Wait
40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Branches ResultPath Retry Catch Type Next End Comment InputPath OutputPath AWS Step Function States: Parallel @technovangelist
41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. State Example: Parallel
42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Many ways to build Step Functions I used Serverless Framework
43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Benefits of Serverless Framework Define everything in a single file: • IAM Role Statements • Lambda Function Handlers • Lambda Function Triggers • Step Function States • CloudFormation for other things
44. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon ECR AWS Lambda Amazon S3 Amazon DynamoDB Amazon CloudFront AWS CodeBuild AWS CodeCommit AWS CloudFormation IAM Amazon API Gateway* AWS Step Functions Amazon SNS
45. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. serverless.yml service: name: mattw-reinvent2017-build-website plugins: - serverless-pseudo-parameters - serverless-step-functions - serverless-webpack custom: repoName: reinvent2017-website
46. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. serverless.yml cont’d provider: name: aws runtime: nodejs6.10 stage: ${opt:stage, 'dev'} region: ${opt:region, 'us-east-1'} iamRoleStatements: - Effect: "Allow" Action: - "states:StartExecution" Resource: - ${self:resources.Outputs.MyStateMachine.Value} - Effect: "Allow" Action: - "codecommit:GetCommit" - "codecommit:GetDifferences" Resource: - arn:aws:codecommit:${self:provider.region}:#{AWS::AccountId}:${…custom.repoName}
47. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. serverless.yml cont’d functions: buildSite: handler: handler.build whatChanged: handler: handler.whatChanged startStepFunction: handler: handler.startStepFunction events: - sns: arn:aws:sns:${…provider.region}:#{AWS::AccountId}:mattw-reinvent-websitebuild environment: statemachine_arn: ${self:resources.Outputs.MyStateMachine.Value}
48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. serverless.yml cont’d stepFunctions: stateMachines: FirstStateMachine: name: myStateMachine definition: StartAt: WhatChanged States: WhatChanged: Type: Task Resource: arn:aws:lambda:#{AWS::Region}:#{AWS::AccountId}\ :function:${self:service}-${self:provider.stage}-whatChanged Next: BuildDockerOrNot
49. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Demo @technovangelist
50. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS CodeCommit Amazon SNS AWS Lambda
51. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. startStepFunction: handler: handler.startStepFunction events: - sns: arn:aws:sns…mattw-reinvent-websitebuild environment: statemachine_arn: ${self:reso…puts.MyStateMachine.Value}
52. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. StartStepFunction Get Step Function ARN Get SNS details Start SF Execution
53. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 1. What changed in the repo? 2. If docker-related, build docker 1. Tell CodeBuild to rebuild docker image 2. Wait for it 3. Push to ECS repo 3. Use the docker image to build the website 4. Wait for it 5. Send Matt an SMS
54. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
55. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. WhatChanged: Type: Task Resource: arn:aws:lambda:... Next: BuildDockerOrNot Retry : - ErrorEquals: - Lambda.Unknown States.ALL IntervalSeconds: 3 MaxAttempts: 4
56. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. WhatChanged Get the commit that triggered the sf Get the previous commit Whats different If it was docker related return hasDocker = true else return hasDocker = false
57. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
58. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. BuildDockerOrNot: Type: Choice Choices: - Variable: "$.hasDocker" BooleanEquals: true Next: BuildDocker - Variable: "$.hasDocker" BooleanEquals: false Next: BuildSite
59. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. BuildDocker: Type: Task Resource: arn:aws:lambda…buildDocker Next: WaitForDockerBuild
60. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. BuildDocker Start the CodeBuild for docker image
61. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. phases: install: commands: - nohup /usr/local/bin/dockerd -G dockremap --host=unix:///… - timeout -t 15 sh -c "until docker info; do echo .; sleep 1; done" pre_build: commands: - echo Logging in to Amazon ECR... - $(aws ecr get-login --no-include-email --region us-east-1) build: commands: - echo Build started on `date` - echo Building the Docker image... - docker build -t mattw-stepfunction-demo:latest . - docker tag mattw-stepfunction-demo:latest ...dkr.ecr.us-east-1.amazona... post_build: commands: - echo Build completed on `date` - echo Pushing the Docker image... - docker push ...dkr.ecr.us-east-1.amazonaws.com/mattw-stepfunc...
62. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. WaitForDockerBuild: Type: Wait Seconds: 20 Next: IsTheContainerBuilt
63. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. IsTheContainerBuilt: Type: Task Resource: arn:aws:lambda... Next: ContainerBuiltOrNot
64. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
65. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. IsTheContainerBuilt Get build info from CodeBuild Check if its done
66. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
67. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ContainerBuiltOrNot: Type: Choice Choices: - Variable: "$.containerBuildDone" BooleanEquals: true Next: BuildSite - Variable: "$.containerBuildDone" BooleanEquals: false Next: WaitForDockerBuild
68. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. phases: build: commands: - echo "**************** In Build ******************" - aws s3 sync s3://mattw-reinvent2017-rawsitepages src/posts - export GATSBY_BUILDTIME=$(tail -1 .git/logs/HEAD | awk '{print $5}') - ln -s /backupmod/node_modules ./node_modules - gatsby build - echo "************* Build Complete ****************" post_build: commands: - echo "************** In Post Build *****************" - aws s3 sync public s3://mybucket --delete --acl public-read - echo "*********** Post Build Complete **************" # - command artifacts: files: - public/**/*
69. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Lambda Amazon S3 Amazon API Gateway* AWS Step Functions AWS Lambda Amazon DynamoDB AWS Lambda AWS CodeBuild
70. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How about monitoring? We monitor Lambda via Logs MONITORING|unix_epoch_timestamp|value|count| my.metric.name|#tag1:value,tag2 console.log(monitoringstring);
71. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What are the Lambda Metrics aws.lambda.duration, min, max, sum aws.lambda.errors aws.lambda.invocations aws.lambda.throttles aws.lambda.iterator_age
72. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What are the Step Function Metrics aws.states.execution_time (.maximum, .minimum) aws.states.executions_aborted, failed, started, succeeded, timed_out aws.states.lambda_functions_scheduled, started, succeeded, timed_out aws.states.execution_throttled aws.states.lambda_function_time (.maximum, .minimum) aws.states.lambda_function_run_time (.maximum, .minimum) aws.states.lambda_function_schedule_time (.maximum, .minimum) aws.states.lambda_functions_started
73. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Summary Step Functions make AWS Lambda orchestration easy Lambda and Step Functions can be an interesting part of your infrastructure Monitor your Lambda functions to ensure peak performance Source can be found at: https://github.com/DataDog/mattw-reinvent2017-demo
74. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. THANK YOU! M a t t h e w W i l l i a m s - @ t e c h n o v a n g e l i s t - D a t a d o g