Airflow: Deferrable “Async” Operators How you can save tons of money!! Kaxil Naik Open Source Summit 2022

Who am I? ● Committer & PMC Member of Apache Airflow ● Director of Airflow Engineering @ Astronomer @kaxil

What is Apache Airflow?

A platform to programmatically author, schedule, and monitor workflows

Example DAG

Why deferrable operators? The problem around current operators & sensors What are they? And how do they work? Available async operators How to find & use the available async operators

Why deferrable operators?

Scheduler Time Scheduler Worker

Scheduler Time Scheduler Submit Spark Job Typical Operator Poll Spark cluster Job Completion Worker

Scheduler Time Scheduler Typical Sensor Wait for files to arrive in S3 Arrived! Worker

Scheduler Time Scheduler What a waste! Wait for files to arrive in S3 Arrived! Worker Submit Spark Job Poll Spark cluster Job Completion Wasted resources Wasted resources Operator Sensor

Scheduler Time Scheduler What a waste! Wait for files to arrive in S3 Arrived! Worker Submit Spark Job Poll Spark cluster Job Completion Wasted resources Wasted resources Operator Sensor

Time Polling for Spark Job completion Done! Multiple Sensors Wait for files to arrive in S3 Done! Polling for Bigquery Job completion Done! Polling for Spark Job completion Done! Wait for files to arrive in S3 Done! Wait for files to arrive in GCS Done! Imagine the Cost! Multiple worker slots Blocked !!

What are deferrable operators?

Scheduler Time Scheduler Worker Triggerer Worker

Scheduler Time Scheduler Submit Spark Job Async Operator Free slot Job Completion Worker Poll Spark cluster Triggerer

Scheduler Time Scheduler Task 1 Worker Slot (Sync vs Async) Task 2 Sync Operator Async Operator Task 1 Task 2 Task 3 Task 2 Task 1 Task 4

Scheduler Time Scheduler Submit Spark Job Async Operator Free slot Job Completion Worker Poll Spark cluster Triggerer Trigger - a new concept

Task runs on the Worker, then “defers” itself Example: Submit an API call and stores job_id in DB Runs on Triggerer Async polling until the criteria is met, stores response in DB Back to the Worker To show response in the logs and set Task state

Trigger is different than Operator

Must be asynchronous & quick So Triggerer can run thousands of them per CPU core Should not persistent state So we can shuffle them around between Triggerers as needed Must support running multiple copies of itself For reliability during network partitions

class DateTimeTrigger(BaseTrigger): def __init__(self, moment: datetime.datetime): super().__init__() self.moment = moment def serialize(self): return ("mymodule.DateTimeTrigger", {"moment": self.moment}) async def run(self): while self.moment > timezone.utcnow(): await asyncio.sleep(1) yield TriggerEvent(self.moment)

class WaitOneHourSensor(BaseSensorOperator): def execute(self, context): self.defer( trigger=TimeDeltaTrigger(timedelta(hours=1)), method_name="execute_complete", ) def execute_complete(self, context, event=None): # We have no more work to do here. Mark as complete. return

Available Deferrable Operators

Core Airflow & Providers https://github.com/apache/airflow/ Astronomer Providers https://github.com/astronomer/astronomer-providers 50+ async operators

Astronomer Providers https://github.com/astronomer/astronomer-providers 50+ async operators Built and maintained with ❤ by Astronomer Apache 2 licensed & Open-source Drop-in replacements of “sync” Operators Openlineage support

Astronomer Providers https://github.com/astronomer/astronomer-providers Async Operators available for: ● AWS, Google Cloud, Microsoft Azure ● Databricks, Snowflake ● Kubernetes, Apache Livy, Apache Hive ● HTTP, Filesystem from astronomer.providers.amazon.aws.sensors.s3 import S3KeySensorAsync as S3KeySensor

from astronomer.providers.amazon.aws.sensors.s3 import S3KeySensorAsync as S3KeySensor waiting_for_s3_key = S3KeySensor( task_id="waiting_for_s3_key" , bucket_key="sample_key.txt" , wildcard_match =False, bucket_name="sample-bucket" , )

Caveats when building one

More cost benefits when polling takes longer time We've seen over a 90% reduction in resources for a 10 min wait Not everything can be deferred It must be an external event/system with a unique identifier Triggerer logs are not visible in the Webserver/UI Only task logs from workers are displayed on UI

Thank You @kaxil