Introduction to Apache Airflow

An overview of Apache Airflow

Transcript

1. Introduction to Airflow Andrew Johnson 1
2. Background • Software Engineer on the Data Platform team at Stripe • Previously at Etsy and Explorys • Airflow runs all of our automated data processes • ~6k jobs • Mixture of MapReduce, Scalding, Redshift, and miscellaneous processes 2
3. Airflow is a Workflow Engine • Manage scheduling and running jobs and data pipelines • Ensures jobs are ordered correctly based on dependencies • Manage allocation of scarce resources • Provides mechanism for tracking the state of jobs and recovering from failure 3
4. Airflow Concepts • Task - defined unit of work • Task instance - Individual run of some task • DAG - Group of tasks connected into a graph by dependency edges, runs on some frequency • DAG run - Individual run of some DAG 4
5. Airflow Architecture 5 Database Scheduler Celery Workers Web UI
6. Scheduler • Responsible for running tasks at the appropriate time • Loads task and DAG definitions and loops to process tasks • Starts DAG runs at appropriate times • Task state is retrieved and updated from the database • Tasks are scheduled with Celery to run 6
7. Database • Stores all Airflow state • All processes read/write from here • DB access through sqlalchemy 7
8. Celery and Workers • Distributed task queue • Backed by Redis/RabbitMQ • Scheduler enqueues tasks with Celery • Some worker executes that task 8
9. Web UI • Allows viewing the state of DAGs/tasks 9
10. Web UI 10
11. Defining Tasks • Core abstraction of Operators • Run shell scripts, Python code • Custom operators for specialized tasks • Jinja templates t1 = BashOperator( task_id='print_date', bash_command='date', dag=dag) 11
12. Defining DAGs • Define upstream/downstream relationships between tasks • t2.set_upstream(t1) • Every task is assigned to a DAG • dag = DAG('tutorial', default_args=default_args) • Magic import • from airflow import DAG 12
13. Resource Management • Create pools in the UI • Fixed number of slots to limit parallelism • Assign priorities to tasks • Priorities propagate upstream 13
14. What works well? • Task definitions are very flexible and extensible • Simple primitives support building complex data pipelines • UI has many useful visualizations • Inspecting DAG structure is straightforward 14
15. What doesn’t work well? • Stability in master has traditionally been not great • Many users maintain forks • Visibility when things go wrong could be better • Large DAGs tend to break the UI • High-frequency (< hourly intervals) can be flaky 15
16. Resources • https://airflow.incubator.apache.org/ • https://github.com/apache/incubator-airflow 16
17. Questions? 17