Orchestrating distributed business workflows with AWS Step Functions

As business processes grow in complexity, modern application developers look to distribute work across multiple independent services. While this can help with scale and enable more rapid development, it comes with challenges around increased failure handling requirements and state management. With AWS Step Functions you can ease some of these concerns and even reduce the burden on application developers looking to solve these complexity challenges. In this session, we cover what's new with Step Functions in 2020 and highlight the key patterns for building distributed business workflows to handle scale and workflow sophistication. We touch upon best practices for handling issues in your workflow and how you can instrument around your workflow for better observability.

Learning Objectives:

* Discover additional features/capabilities that have launched for AWS Step Functions in the past year
* Learn about patterns for handling failure and recovery in your workflows
* Learn about integrations with AWS services such as EventBridge, DynamoDB, SQS, SNS, Fargate, and more

To learn more about the services featured in this talk, please visit: https://aws.amazon.com/step-functions/

Transcript

1. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Rob Sutter, Sr. Developer Advocate 29 July 2020 AWS Step Functions Orchestrating distributed business workflows
2. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Agenda Orchestrating distributed business workflows • What are AWS Step Functions? • Nested workflows • Dynamic parallelism • Express Workflows • AWS Toolkit for Visual Studio Code support • AWS CodeBuild service integration • AWS SAM support • AWS CDK support • Error handling
3. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Who am I? Rob Sutter - robsut@amazon.com • Senior Developer Advocate - Serverless • Gopher and Scala type • Previously: • Co-founded WorkFone, a SaaS startup • Infrastructure at an e-commerce startup • Consulting, government, odd jobs here and there • The Florida State University, Management Information Systems ’05 • Twitch: /robsutter • Twitter: @rts_rob
4. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What are AWS Step Functions?
5. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What are AWS Step Functions? Serverless workflows that help you: • Build and update apps quickly • Improve resiliency • Write less code • Orchestrate long-running tasks • Modernize monoliths • Integrate with managed services • Handle errors and retries AWS Step Functions
6. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Serverless workflows Define Visualize Monitor
7. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What is serverless? No infrastructure provisioning, no management Automatic scaling Pay for value Highly available and secure
8. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Nested workflows
9. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Nested workflows • Build larger, more complex workflows out of smaller, simpler workflows • Create reusable building blocks and build faster • Swap workflow modules without customizing code • Return a string or a JSON object
10. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Dynamic parallelism
11. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Dynamic parallelism • “Map State” • Run identical tasks in parallel • Fanout pattern – dispatch a list of identical tasks to simplify workflows like order processing and instance management • Scatter-gather pattern – accelerate workflows such as file processing
12. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Express Workflows
13. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Express Workflows Express Workflows Standard Workflows Maximum duration 5 minutes 1 year Execution start rate Over 100,000/s Over 2,000/s State transition rate Nearly unlimited Over 4,000/s Pricing GB (memory) * s (time) Per state transition Execution history CloudWatch logs API, console, logs Execution semantics At-least-once Exactly-once Service integrations No Job-run or Callback All integrations and patterns
14. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Toolkit for Visual Studio Code
15. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Toolkit for Visual Studio Code • Create workflows with sample templates • Accelerate development with code snippets • Visualize workflows without leaving the IDE • Execute state machines in your AWS account
16. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS CodeBuild service integration
17. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS CodeBuild service integration • Start builds periodically or in response to events with Amazon EventBridge • Create webhooks to start builds using Amazon API Gateway • Parallel state for independent components • Map state for each branch
18. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS SAM support
19. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Serverless Application Model (AWS SAM) • AWS::Serverless::StateMachine component • Use DefinitionURI and DefinitionSubstitutions to create workflows from separate files • Apply AWS SAM policy templates to workflows • AWS SAM event sources
20. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS CDK support
21. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Cloud Development Kit (CDK) • Constructs for generating Step Functions • Customize, share, reuse • Use familiar programming languages and tools • Deploy infrastructure and runtime code together
22. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Error handling
23. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Retriers • Task, Parallel, and Map States • Switch/case statement on ErrorEquals value • IntervalSeconds is the number of seconds before first retry • MaxAttempts is the number of retries (may be 0) • BackoffRate is a multiplier
24. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Catchers • Task, Parallel, and Map States • Also known as Fallback States • Scanned in array order when there is an error and no Retrier or all retries have failed • Switch/case statement on ErrorEquals value • ResultPath for storing error • Next defines the next state
25. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Closing
26. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Review Orchestrating distributed business workflows • What are AWS Step Functions? • Nested workflows • Dynamic parallelism • Express Workflows • AWS Toolkit for Visual Studio Code support • AWS CodeBuild service integration • AWS SAM support • AWS CDK support • Error handling
27. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Additional resources AWS Step Functions resources page • aws.amazon.com/step-functions/resources/ AWS Step Functions video series • rbsttr.tv/sfn AWS Serverless YouTube channel • youtube.com/c/AWSServerless
28. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you! Twitter: @rts_rob Twitch: /robsutter