Building And Automating Serverless Auto-Scaling Data Pipelines In AWS (2024-11-20: AWS User Group Manchester)

The modern data professional navigates a dynamic data landscape, handling high-velocity raw data at ever-changing volumes. In this 30-minute intermediate session, I demonstrate a fully serverless auto-scaling data pipeline using AWS services.

This session includes:

- Getting and storing API data with AWS Lambda and Amazon S3.
- Transforming the API data with AWS Glue & Amazon Athena.
- Pipeline automation and orchestration with AWS Step Functions and Amazon EventBridge.

Ideal for Data, DevOps, and Architecture professionals, this session offers practical insights into building efficient serverless data pipelines. Join to enhance your skills and explore the latest in Data Engineering and AWS.

Resources at https://github.com/MrDamienJones/Community-Sessions/tree/main/BuildingAndAutomatingServerlessAutoScalingDataPipelinesInAWS

Transcript

1. Building And Automating Serverless Auto-Scaling Data Pipelines In AWS Damien Jones (he/him) AWS Consultant @ Steamhaus 2024-11-20 AWS User Group Manchester @amazonwebshark MrDamienJones amazonwebshark.com damien@amazonwebshark.com MrDamienJones https://www.flaticon.com/ @amazonwebshark
2. Agenda Problem Definition Solution Architecture Demos Summary & Questions github.com/MrDamienJones /Community-Sessions
4. Damien Jones Consultant @ Steamhaus Using AWS since 2019 Creator @ amazonwebshark.com Runner; Keen Gardener; Dog Dad He/Him Manchester UK Fin Fan
5. The 4 Vs Of Big Data Characteristics of Big Data… …and events… …and API requests… …metrics …traces …logs ...
6. Variety “The state of being diverse or varied.”
7. Variety “The state of being diverse or varied.” Structure Intent Sensitivity
8. Velocity “The speed at which something is moving in a given direction.”
9. Velocity “The speed at which something is moving in a given direction.” Streaming or Batch Synchronous or Asynchronous Scheduling
10. Veracity “The quality of being true or the habit of telling the truth.”
11. Veracity “The quality of being true or the habit of telling the truth.” External Security Validation & Health Internal Security
12. Volume “The amount of space occupied.”
13. Volume “The amount of space occupied.” Size or Amount Storage Options Backups
14. Volume Veracity Variety Velocity
15. Building And Automating Serverless Auto-Scaling Data Pipelines In AWS
16. Building And Automating Serverless Auto-Scaling Data Pipelines In AWS
17. Building And Automating Serverless Auto-Scaling Data Pipelines In AWS AWS Lambda Amazon S3
18. AWS Lambda Serverless compute service Supports multiple languages Auto-scales on demand Up to 1000 concurrent executions
22. Amazon S3 Serverless object storage Store anything for any reason 1000s of requests per second Object protection & integrity checks
24. S3  1 million buckets
25. Building And Automating Serverless Auto-Scaling Data Pipelines In AWS
26. Building And Automating Serverless Auto-Scaling Data Pipelines In AWS Amazon Athena AWS Glue
27. Amazon Athena Serverless interactive query service Query & access controls Create derived tables Read-Only & Open Table support
28. Wolfie Says Hi
29. AWS Glue Fully managed serverless ETL service Crawlers discover data automatically Up to 2000 concurrent ETL job runs Data Catalog indexes data assets
30. Havoc Says Blep
31. Building And Automating Serverless Auto-Scaling Data Pipelines In AWS
32. Building And Automating Serverless Auto-Scaling Data Pipelines In AWS Amazon EventBridge Scheduler AWS Step Functions
33. AWS Step Functions Serverless task orchestration Invoke over 220 AWS services / 10k API calls Standard & Express workflows Design workflows visually and as code
39. AWS Step Functions Demo Lambda Function: API Call Glue Job: ETL Athena Query: MSCK REPAIR TABLE
41. AWS Step Functions Alternatives Lambda function: + Cheaper - Less observability
42. AWS Step Functions Alternatives Glue Workflow: + Free! - Glue resources only
43. AWS Step Functions Alternatives Managed Airflow: + Customisable - Complexity
44. Amazon EventBridge Scheduler Automate recurring & one-off tasks Invoke over 220 AWS services Set times or fixed-rate schedules Checks target response
45. Amazon EventBridge Scheduler Demo Set schedule Link Step Function workflow Set configuration
47. Summary Problem Definition Solution Architecture Demos Summary & Questions github.com/MrDamienJones /Community-Sessions
48. Thanks! github.com/MrDamienJones /Community-Sessions @amazonwebshark MrDamienJones amazonwebshark.com damien@amazonwebshark.com MrDamienJones @amazonwebshark