Getting Started with Apache Airflow - Srinidhi

Transcript

1. Apache Airflow 2 Srinidhi Sridharan Independent Consultant - Data Engineer Pyspark, Airflow, AWS, Azure https://www.linkedin.com/in/srinidhi-s/
2. Data Warehouse Transformation Landing Transformation Transformation ….. Typical Data flow Scenario Scheduler - cron Error Handling Monitoring Handle Dependencies Scaling Increase in frequency , complexity and severity, scope to scale API
3. Airflow Python Workflow Management Tool / Orchestrator While creating Apache Airflow
4. How Airflow handles the expectations of an orchestrator ?
5. Airflow solves this using How do you handle long running tasks and dependencies? DAGs T4 T1 T3 T2 Create a DAG and define its properties Define the operators (tasks) of DAG The benefit of this is every node executes only once and every node has only one set of inputs and output and this makes the pipeline simpler to debug and define.
6. Airflow solves this using WebUI Can you monitor what's currently running? What's queued?
7. Airflow solves this using How do you know when the jobs fail and how do you debug? Alerts #alerts # alerts
8. Airflow solves this using How do you maintain multiple users, analyse previous run time, maintain connections? Metadata DB Postgres/ MySQL
9. Airflow solves this using Do you have reusable components you can use across workflows? Operators Postgres Operator Bash Operator Python Operator Python Classes Docker Operator Kubernetes Operator
10. Airflow solves this using How do you scale the jobs when it gets complex ? Executors Local Celery Kubernetes Sequential
11. Airflow ≠ • ! A data streaming solution. • It does not move huge data between tasks • ! Tied to a language. Workflows are defined in Python but Airflow can invoke frameworks built out of other languages • ! ETL Framework - Ideally
12. SLA monitoring The ability to look at the history and see how many time a dag ran/failed in the past? Easy access to a tasks log, checking the kind of information passed between tasks Easy integration with a secret manager system wide parameterisation of connections?
13. Creating DAG in Airflow DAG Instance ✓ DAG Tasks and dependency ✓ DAG Config ✓
15. What’s happening Under the hood ?
16. Master Node Scheduler Web Server Airflow Process Node Web UI / User Interface Executor Metadata DB (Postgres/MySQL) Airflow Services Queue Worker node1 Worker node 3 Celery Executor
17. Extensibility Airflow allows you to extend the functionality by: • Using Plugins (Ex: Modify UI to add extra menu items) • Adding custom Operators or building on top of the existing ones.
18. AGAIN
19. What we are about to see in demo • Orchestrating an ELT Job using Airflow ◦ Delegating heavy weight data manipulation to external frameworks ◦ Creating Dynamic Tasks ◦ Airflow API ◦ Custom Operators or Plugins
20. Managed Airflow Services • Astronomer Managed Airflow • AWS MWAA • Google Cloud compose
21. Shortcomings • Lack of data sharing among the tasks • Reparsing the DAG folder every few seconds • Setting up Airflow production is a bit of tedious process
22. For more queries: https://www.linkedin.com/in/srinidhi-s/