Baltimore DevOps - Orchestrating Serverless with AWS Step Functions

OK, you have built your first Lambda function. It may even be useful (and not just a thumbnail generator). But an application is more than a single function; you need a lot of them. How do you get your functions working together? In this introduction Matt Williams will get you started with Step Functions and show you how easy it is to use them to orchestrate AWS Lambda.

Transcript

1. Orchestrating Serverless with AWS Step Functions
3. matt williams - @technovangelist Baltimore DevOps
7. matt williams - @technovangelist Baltimore DevOps
8. matt williams - @technovangelist Baltimore DevOps …but I didn’t bring a laptop
9. matt williams - @technovangelist Baltimore DevOps Who am I Matt Williams Evangelist at Datadog mattw@datadoghq.com TW: technovangelist YT: technovangelist IG/GH/XYZ: technovangelist Organizer of: DevOps Days Boston 2017 & 2018 Serverless Days Boston 2018
10. matt williams - @technovangelist Baltimore DevOps Who is Datadog SaaS-based Monitoring & Analytics Infrastructure, APM, Logs Open Source Agent Trillions of data points per day We are hiring!!
11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. I like to talk about Serverless…
12. matt williams - @technovangelist Baltimore DevOps What is AWS Lambda Its all about the function Serverless As far as you are concerned Don’t pay for idle Scale with ease FaaS Node 8.10/6.10, Golang, C# 1/2/2.1, Python 2.7/3.6, Java8 Trigger-based
13. matt williams - @technovangelist Baltimore DevOps Trigger Based? S3 DynamoDB Kinesis SNS SES Cognito CloudFormation Cloudwatch Logs & Events CodeCommit Config Alexa Lex API Gateway IoT CloudFront Lambda http://tvl.st/khvid
14. matt williams - @technovangelist Baltimore DevOps AWS Lambda
15. matt williams - @technovangelist Baltimore DevOps
16. matt williams - @technovangelist Baltimore DevOps That’s pretty cool right? But not perfect You still have to configure everything else S3 DynamoDB API Gateway etc SAM Cloudformation Serverless
17. matt williams - @technovangelist Baltimore DevOps AWS Lambda
18. matt williams - @technovangelist Baltimore DevOps …but serverless !== ec2
19. matt williams - @technovangelist Baltimore DevOps …but serverless !== applications
20. matt williams - @technovangelist Baltimore DevOps …applications are complicated
21. matt williams - @technovangelist Baltimore DevOps …serverless functions should respect the single responsibility principle…
22. matt williams - @technovangelist Baltimore DevOps
23. matt williams - @technovangelist Baltimore DevOps
24. matt williams - @technovangelist Baltimore DevOps we can create many serverless functions
25. matt williams - @technovangelist Baltimore DevOps we can create many serverless functions but how do we orchestrate them?
26. matt williams - @technovangelist Baltimore DevOps Event/Message/Database as Traffic Cop Amazon Kinesis Amazon SQS Amazon DynamoDB
27. matt williams - @technovangelist Baltimore DevOps https://github.com/Nordstrom/hello-retail
28. matt williams - @technovangelist Baltimore DevOps Enter Step Functions
29. matt williams - @technovangelist Baltimore DevOps Step Function Concepts State Machine States Task Choice Succeed/Fail Pass Wait Parallel Transitions Executions You define the State Machine with the Amazon States Language A diagram will be generated based on that structure
30. matt williams - @technovangelist Baltimore DevOps State Example: Task
31. matt williams - @technovangelist Baltimore DevOps State Example: Choice
32. matt williams - @technovangelist Baltimore DevOps State Example: Succeed / Fail
33. matt williams - @technovangelist Baltimore DevOps State Example: Pass
34. matt williams - @technovangelist Baltimore DevOps State Example: Wait
35. matt williams - @technovangelist Baltimore DevOps State Example: Parallel
36. matt williams - @technovangelist Baltimore DevOps Many ways to build AWS Step Functions I used Serverless Framework (http://tvl.st/sls)
37. matt williams - @technovangelist Baltimore DevOps Benefits of Serverless Framework Define everything in a single file: IAM Role Statements Function Handlers Function Triggers State Machine States AWS CloudFormation for other things
38. matt williams - @technovangelist Baltimore DevOps Amazon ECR AWS Lambda Amazon S3 Amazon DynamoDB Amazon CloudFront AWS CodeBuild AWS CodeCommit AWS CloudFormation IAM Amazon API Gateway* AWS Step Functions Amazon SNS
39. matt williams - @technovangelist Baltimore DevOps
41. matt williams - @technovangelist Baltimore DevOps
42. matt williams - @technovangelist Baltimore DevOps
43. matt williams - @technovangelist Baltimore DevOps
44. matt williams - @technovangelist Baltimore DevOps
45. matt williams - @technovangelist Baltimore DevOps 1.Use ML to figure out which slide 2.Is this slide different than the last time? 3.If so, pull slide data from Dynamo 4.If slide different then create document in s3 5.Build the site 6.Wait for it 1.Is it built 7.Done
46. matt williams - @technovangelist Baltimore DevOps Type: Task Resource: Function
47. matt williams - @technovangelist Baltimore DevOps
49. matt williams - @technovangelist Baltimore DevOps Type: Task Resource: Function
50. matt williams - @technovangelist Baltimore DevOps
51. matt williams - @technovangelist Baltimore DevOps
52. matt williams - @technovangelist Baltimore DevOps Type: Task Resource: Function
53. matt williams - @technovangelist Baltimore DevOps
55. matt williams - @technovangelist Baltimore DevOps How about monitoring? We monitor serverless via Logs MONITORING|unix_epoch_timestamp|value|count| my.metric.name|#tag1:value,tag2 console.log(monitoringstring);
56. matt williams - @technovangelist Baltimore DevOps What are the AWS Lambda Metrics aws.lambda.duration, min, max, sum aws.lambda.errors aws.lambda.invocations aws.lambda.throttles aws.lambda.iterator_age
57. matt williams - @technovangelist Baltimore DevOps What are the AWS Step Function Metrics aws.states.execution_time (.maximum, .minimum) aws.states.executions_aborted, failed, started, succeeded, timed_out aws.states.lambda_functions_scheduled, started, succeeded, timed_out aws.states.execution_throttled aws.states.lambda_function_time (.maximum, .minimum) aws.states.lambda_function_run_time (.maximum, .minimum) aws.states.lambda_function_schedule_time (.maximum, .minimum) aws.states.lambda_functions_started
58. matt williams - @technovangelist Baltimore DevOps
59. matt williams - @technovangelist Baltimore DevOps
60. matt williams - @technovangelist Baltimore DevOps
61. matt williams - @technovangelist Baltimore DevOps
62. matt williams - @technovangelist Baltimore DevOps
63. matt williams - @technovangelist Baltimore DevOps
64. matt williams - @technovangelist Baltimore DevOps AWS Lambda and AWS Step Function Challenges • Logging and debugging • Don’t trust any function when it claims a step is complete • Cold start depends on your use case • Every function should do one thing…but granularity depends
65. matt williams - @technovangelist Baltimore DevOps Matt Williams Evangelist at Datadog mattw@datadoghq.com all the socials: technovangelist Organizer of DevOps Days Boston 2017 & 2018