AWS Summit Amsterdam 2023 - SVS204

Large scale parallel data processing with AWS Step Functions Distributed Maps

Transcript

1. Large scale parallel data processing with Step Functions Distributed Map SVS204
2. About Me Pubudu Jayawardana @pubudusj From Amsterdam, the Netherlands Senior Backend Developer at starred.com AWS Community Builder (Serverless) AWS Certified - SA Pro https://medium.com/@pubudusj https://pubudu.dev https://dev.to/pubudusj
3. AWS Community Builders
4. Step Functions Distributed Map
5. ▪ To iterate over an array ▪ Limitations • 40 parallel iterations at a time • Max payload size - 256KB • Execution history - 25,000 events Map State
6. ▪ Totally separated child executions • 25,000 events each • 10,000 executions at a time • S3 as a source ▪ Result output to S3 ▪ Only applicable for Standard flows Distributed Map
7. Distributed Map
8. ▪ Source types: • S3 object list • JSON file in S3 • CSV file in S3 • S3 manifest file ▪ Limit no of items ▪ ItemSelector Source
9. ▪ Batching based on: • No of items • Size ▪ Modify input with Batch input Item Batching
10. ▪ Concurrency limit ▪ Child execution types: • Standard • Express ▪ Error threshold: • Percentage • No of items Runtime Settings
11. ▪ S3 location ▪ Logs • manifest.json • SUCCEEDED_n.json • FAILED_n.json • PENDING_n.json Export Result
12. Execution Details - Parent Event Log
13. Execution Details - Map Run
14. Execution Details - Single Child Execution
15. Process
16. ▪ SAAS application to measure candidate experience ▪ Send surveys ▪ Record the feedback ▪ Visualize in a dashboard (benchmark, filter, comparison) ▪ Transform / Enrich data Process
17. Source from MySQL Transform Save to S3 Load to Postgres Hourly ETL
18. ▪ Amazon Managed Workflows for Apache Airflow ▪ Amazon EMR Hourly ETL
20. Problem
21. ▪ Less visibility ▪ Cannot retry single table load ▪ Takes avg 20 minutes ▪ EC2 cost Data Load Step
22. Solution
25. Demo
26. ▪ Reduced time to avg 5 minutes ▪ Load data parallelly ▪ Better insights ▪ Retry individual table data load ▪ Cost effective Benefits
27. ▪ Use batching ▪ Set concurrency ▪ Set error threshold ▪ Use express child executions Tips / Lesson Learned
28. https://bit.ly/s3-to-postgres Read more about this
29. Thank You! @pubudusj