Building And Automating Serverless Auto-Scaling Data Pipelines In AWS (2024-10-08: AWS User Group Leeds)

The modern data professional navigates a dynamic data landscape, handling high-velocity raw data at ever-changing volumes. In this 30-minute intermediate session, I demonstrate a fully serverless auto-scaling data pipeline using AWS services.

This session includes:

- Getting and storing API data with AWS Lambda and Amazon S3.
- Transforming the API data with AWS Glue & Amazon Athena.
- Pipeline automation and orchestration with AWS Step Functions and Amazon EventBridge.

Ideal for Data, DevOps, and Architecture professionals, this session offers practical insights into building efficient serverless data pipelines. Join to enhance your skills and explore the latest in Data Engineering and AWS.

Resources at https://github.com/MrDamienJones/Community-Sessions/tree/main/BuildingAndAutomatingServerlessAutoScalingDataPipelinesInAWS

Transcript

1. Building And Automating Serverless Auto-Scaling Data Pipelines In AWS Damien Jones (he/him) AWS Consultant @ Steamhaus 2024-10-08 AWS User Group Leeds @amazonwebshark MrDamienJones amazonwebshark.com damien@amazonwebshark.com MrDamienJones https://www.flaticon.com/ @amazonwebshark
2. Agenda Problem Definition Solution Architecture Demos Summary & Questions github.com/MrDamienJones /Community-Sessions
3. Damien Jones Consultant @ Steamhaus Using AWS since 2019 Creator @ amazonwebshark.com Runner; Keen Gardener; Dog Dad He/Him Manchester UK Fin Fan
4. The 4 Vs Of Big Data Characteristics of Big Data… …and events… …and API requests… …metrics …traces …logs ...
5. Variety “The state of being diverse or varied.”
6. Variety “The state of being diverse or varied.” Structure Intent Sensitivity
7. Velocity “The speed at which something is moving in a given direction.”
8. Velocity “The speed at which something is moving in a given direction.” Streaming or Batch Synchronous or Asynchronous Scheduling
9. Veracity “The quality of being true or the habit of telling the truth.”
10. Veracity “The quality of being true or the habit of telling the truth.” External Security Validation & Health Internal Security
11. Volume “The amount of space occupied.”
12. Volume “The amount of space occupied.” Size or Amount Access Patterns Backups
13. Building And Automating Serverless Auto-Scaling Data Pipelines In AWS
14. Building And Automating Serverless Auto-Scaling Data Pipelines In AWS
15. Building And Automating Serverless Auto-Scaling Data Pipelines In AWS AWS Lambda Amazon S3
16. AWS Lambda Serverless compute service Supports multiple languages Auto-scales on demand Up to 1000 concurrent executions
17. Amazon S3 Serverless object storage Store anything for any reason 1000s of requests per second Object protection & integrity checks
18. Building And Automating Serverless Auto-Scaling Data Pipelines In AWS
19. Building And Automating Serverless Auto-Scaling Data Pipelines In AWS Amazon Athena AWS Glue
20. Amazon Athena Serverless interactive query service Query & access controls Create derived tables Read-Only & Open Table support
21. AWS Glue Fully managed serverless ETL service Crawlers discover data automatically Up to 2000 concurrent ETL job runs Data Catalog indexes data assets
22. Building And Automating Serverless Auto-Scaling Data Pipelines In AWS
23. Building And Automating Serverless Auto-Scaling Data Pipelines In AWS Amazon EventBridge Scheduler AWS Step Functions
24. AWS Step Functions Serverless task orchestration Invoke over 220 AWS services / 10k API calls Design workflows visually and as code Standard & Express workflows
29. AWS Step Functions Demo Lambda Function: API Call Glue Job: ETL Athena Query: MSCK REPAIR TABLE
31. Amazon EventBridge Scheduler Automate recurring & one-off tasks Invoke over 220 AWS services Set times or fixed-rate schedules Checks target response
32. Amazon EventBridge Scheduler Demo Set schedule Link Step Function workflow Set configuration
34. Summary Problem Definition Solution Architecture Demos Summary & Questions github.com/MrDamienJones /Community-Sessions
35. Thanks! github.com/MrDamienJones /Community-Sessions @amazonwebshark MrDamienJones amazonwebshark.com damien@amazonwebshark.com MrDamienJones @amazonwebshark