Datasets: What it is, and how it was made

This is the slide deck I use to present during the Airflow Meetup Tokyo #3.

Conpass page (Japanese)
Meetup page (English)

Transcript

1. Datasets What it is, and how it was made Tzu-ping Chung 1
2. Call me TP (uraunsjr.com) Taipei, Taiwan Astronomer Inc. Python packaging (pip & others) PyCon Taiwan (2–3 Sep.) 2
3. The scenario 3
4. Existing solutions SubDagOperator Giant DAG + task groups TriggerDagRunOperator 4
5. # Upstream DAG. create = S3CreateObjectOperator( task_id="upload_data", s3_key=uri, data=transformed_data, ) trigger = TriggerDagRunOperator( task_id="trigger_downstream", trigger_dag_id="use_data", ) create >> trigger # Downstream DAG. dag = DAG( dag_id="use_data", schedule=None, ) @dag.task def use_s3_data(): hook = S3Hook() hook.download_file(uri, ...) 5
6. “DAG of DAGs” with DAG(dag_id="dag_of_dags", ...): trigger_create_dag = TriggerDagRunOperator( task_id="trigger_upstream", trigger_dag_id="create_data", wait_for_completion=True, ) trigger_use_dag = TriggerDagRunOperator( task_id="trigger_downstream", trigger_dag_id="use_data", wait_for_completion=True, ) trigger_create_dag >> trigger_use_dag 6
7. Separation of Concerns When adding downstream DAGs “Why was this DAG run?” Mixing DAG and task dependencies 7
8. Remove the “Lines” Publish-subscribe pattern (PubSub) Upstream task emits an event DAG scheduled when event detected 8
10. Background Design Rich scheduling Data lineage (task-level) AIP-48 10
11. Implementat Events Metadatabase stores published messages Dataset Manager subscribes DAGs on parse Scheduler loop calls Dataset Manager 11
12. # Upstream DAG. from common import dataset S3CreateObjectOperator( task_id="upload_data", s3_key=dataset.uri, data=transformed_data, outlets=[dataset], # <- ) # Downstream DAG. from common import dataset dag = DAG( dag_id="use_data", schedule=[dataset], # <- ) @dag.task def use_s3_data(): uri = dataset.uri h = S3Hook() h.download_file(uri, ...) 12
13. Future work External dataset inlets Dynamic lineage expansion Multi-paradigm DAG scheduling 13
14. Questions? 14