AWS Step Functions by example

I walk through several real world examples of how my team uses AWS Step Functions to simplify otherwise complicated, distributed workflows. Starting with some basic examples and working to more involved uses cases, you’ll learn how you can leverage Step Functions yourself.

Transcript

1. 8th Light Software is our craft. 8thlight.com AWS Step Functions by Example Myles Megyesi - github.com/mylesmegyesi
2. Overview Overview • What is AWS Step Functions? • Examples ◦ Ad-hoc Jobs ◦ Polling ◦ Retry ◦ Looping ◦ Nested State Machines
3. What is AWS Step Functions? • High-level workflow service • Serverless • Orchestrates other AWS Services What is AWS Step Functions?
4. Service Integrations What is AWS Step Functions? Lambda AWS Batch DynamoDB ECS Fargate SNS SQS Glue Glue DataBrew SageMaker EMR EMR on EKS CodeBuild Athena EKS API Gateway EventBridge Step Functions
5. Flows What is AWS Step Functions? Choice Parallel Map Pass Wait Success Fail
6. Amazon States Language What is AWS Step Functions? { "Comment": "Hello, World!", "StartAt": "Succeed", "States": { "Succeed": { "Type": "Success" } } }
7. Example 1 - Ad-hoc jobs Examples Use case • Need to run ECS tasks • aws-cli for invoking ECS tasks is a bit unwieldy
8. Example 1 - Ad-hoc jobs What is AWS Step Functions? { "StartAt": "Run Report and wait for it to complete", "States": { "Run Report and wait for it to complete": { "Type": "Task", "Resource": "arn:aws:states:::ecs:runTask.sync", "Parameters": { "Cluster": "<ecs-cluster-arn>", "TaskDefinition": "<ecs-task-definition-arn>", "NetworkConfiguration": { "AwsvpcConfiguration": { "AssignPublicIp": "DISABLED", "SecurityGroups": [ "<security-group-ids>" ], "Subnets": [ "<subnet-ids>" ] } } }, "End": true } } }
9. Example 2 - Polling Examples Use case • 3rd party asynchronous API a. Submit request b. Check status c. Get results • Post Process results • Send response to our client
10. Example 2 - Polling Examples
11. Example 2 - Polling Examples
12. Example 2 - Polling - Submit Request Examples "Submit Request": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": { "FunctionName": "<lambda-arn>" }, "Next": "Wait 15 seconds" }
13. Example 2 - Polling Examples
14. Example 2 - Polling - Wait for 15 seconds Examples "Wait 15 seconds": { "Type": "Wait", "Seconds": 15, "Next": "Get Status" }
15. Example 2 - Polling Examples
16. Example 2 - Polling - Get Status Examples "Get Status": { "Type": "Task" "Resource": "arn:aws:states:::lambda:invoke", "Parameters": { "FunctionName": "<lambda-arn>" }, "Next": "Evaluate Status" }
17. Example 2 - Polling Examples
18. Example 2 - Polling - Evaluate Status Examples "Evaluate Status": { "Type": "Choice" "Choices": [ { "Variable": "$.status", "StringEquals": "IN_PROGRESS", "Next": "Wait 1 Second" }, { "Variable": "$.status" "StringEquals": "SUCCEEDED", "Next": "Post Process Result", }, { "Variable": "$.status", "StringEquals": "FAILED", "Next": "Failed" } ] }
19. Example 2 - Polling Examples
20. Example 3 - Retry Examples Use case • Sending Response to end-client intermittently fails • Need to retry
21. Example 3 - Retry Examples
22. Example 3 - Retry Examples "Send Response to End Client": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": { "FunctionName": "<lambda-arn>" }, "Retry": [ { "ErrorEquals": [ "com.example.RequestFailureException" ], "IntervalSeconds": 2, "MaxAttempts": 6, "BackoffRate": 2 } ], "Next": "Succeeded" }
23. Example 4 - Looping Examples Use case • Result from 3rd party API now returns a list of PDF documents • Each document needs to be checked for viruses before sending to our end-client
24. Example 4 - Looping Examples
25. Example 4 - Looping Examples "Scan Documents for Viruses": { "Type": "Map", "ItemsPath": "$.documents", "MaxConcurrency": 10, "Iterator": { "StartAt": "Scan a Document for viruses", "States": { "Scan a Document for viruses": { "Type": "Task" "Resource": "arn:aws:states:::lambda:invoke", "Parameters": { "FunctionName": "<lambda-arn>" }, "End": true } } }, "Next": "Post Process Result" }
26. Example 5 - Nested State Machines Examples Use case • Our State Machine has grown quite large • Refactor into multiple State Machines
27. Example 5 - Nested State Machines Examples
28. Example 4 - Looping Examples "Get Response": { "Type": "Task", "Resource": "arn:aws:states:::states:startExecution", "Parameters": { "StateMachineArn": "<state-machine-arn>" }, "Next": "Post Process Result" }
29. What I didn’t cover Overview To name a few... • Local Development • Input-Output Processing • Error Handling • X-Ray tracing • Working around Step Functions service limits • Logging & monitoring • Human-in-the-loop
30. 8th Light Software is our craft. 8thlight.com Q + A
31. 8th Light Software is our craft. 8thlight.com Thank You! Myles Megyesi - github.com/mylesmegyesi