Fraud screening Revenue close, ledger risk_fraud_screening suspicious orders trigger storefront_daily_orders clean day finance_revenue_close trigger storefront_customer_invoices Each team ships its own Dag bundle in its own language. Hand-offs only through TriggerDagRun and a data contract on a shared lake.
Tags: []string{"finance"}, }, ) TypeScript const dag = new Dag( "storefront_daily_orders", { schedule: "@daily", queue: "typescript", }, ); Java @Builder.Dag( id = "risk_fraud_screening") public class FraudScreeningDag { @Builder.Task( id = "load_batch", queue = "java") public String loadBatch(...) { No Dag-level queue yet: set it on each task.
processor Task execution time Dag JSON Worker TI knows dag_id, task_id, queue Airflow Metadata DB process Answer a compatible serialized Dag JSON process process How do we spawn your task subprocess?
acme-risk-all.jar bundle.min.mjs finance JavaDagImporter NodeDagImporter ExecutableDagImporter BaseCoordinator parse_dag(path, ...) execute_task(what, ...) Dag processing task execution the caller never sees below this line JavaCoordinator NodeCoordinator spawns spawns JVM Node java -cp ... Main-Class node bundle.min.mjs ExecutableCoordinator spawns AbstractDagImporter can_handle(definition) list_dag_definitions(bundle) import_definition(definition, bundle) get_source_code(definition, dag_id) Go binary ./finance Each one spawns its process runtime. Callers only use parse_dag and execute_task. Dags Source code DagImportResult.dags DagSourceCode Whatever the artifact, you get Dags and their source.
Metadata DB 4 NodeDagImporter .can_handle() uses 1 NodeCoordinator serialized Dag JSON 3 2 spawns bundle.min.mjs 1 NodeDagImporter claims the bundle by extension or TS process runtime node bundle.min.mjs 2 NodeCoordinator spawns the TS process runtime. 4 The Dag processor process hands it to the manager. can_handle() . 3 The runtime returns the serialized Dag JSON to the DagImporter. 5 The manager persists it to the Metadata DB.
languages, one Airflow. Not to mention, once these come to every language: Human-in-the-loop Common AI features Providers, e.g. operators in TypeScript Asset-native, event-driven Every dialect talks to the others in the Airflow language.
build_prompts(rows: list[str]) -> list[str]: return [f"Summarize: {row}" for row in rows] One batch job, about half the price. OpenAI or Anthropic batch API · deferred, so no worker slot is held.
picks what happens next. 0.97 → rerun 0.62 → page_oncall Over the bar: it runs by itself Under its bar: a person decides Not a guess: a number you chose, per branch.
the top 5 customers by order count?", llm_conn_id="pydanticai_default", durable=True , toolsets=[SQLToolset(db_conn_id="postgres_default", ...)], ) Retries replay completed model and tool steps from cache.
the saved answer Every model call and tool call runs again, from step 1. A finished step hands back its stored result. The model is not called and the tool does not run. Only a step with no saved result runs live.
model call 2,773 tokens. The whole conversation, again. 1,163 tokens. 9 steps replayed from the cache. Same Dag, same failure, 1 run each, local gpt-oss:20b.
Every attempt starts with a fresh budget. 3 retries can spend it 4 times. 1 budget for the first run, every retry, and every HITL regeneration. 3.3+: the running total is kept in the task state store, so it survives retries.
# 1 · known failures ExceptionRetryPolicy(rules=[...]), # 2 · the model LLMRetryPolicy(llm_conn_id="pydanticai_default"), # 3 · the floor ExceptionRetryPolicy(rules=[...]), ]) Known failures first: no model call is spent on them.
# 1 · known failures ExceptionRetryPolicy(rules=[...]), # 2 · the model LLMRetryPolicy(llm_conn_id="pydanticai_default"), # 3 · the floor ExceptionRetryPolicy(rules=[...]), ]) Then the model decides what is left.
# 1 · known failures ExceptionRetryPolicy(rules=[...]), # 2 · the model LLMRetryPolicy(llm_conn_id="pydanticai_default"), # 3 · the floor ExceptionRetryPolicy(rules=[...]), ]) Then a floor, for when the model has no answer. The first RETRY or FAIL wins. All DEFAULT: the task's own retries. Airflow 3.4+ (3.3 via common.compat).
plain Dag: a schedule and a SQL task If anything looks off @task.llm_branch page #data-oncall HookToolset if it fails, try again, unless the password is wrong LLMRetryPolicy
plain Dag: a schedule and a SQL task If anything looks off @task.llm_branch page #data-oncall HookToolset if it fails, try again, unless the password is wrong LLMRetryPolicy Then just tell me it fails right away, with the model's reason on the task (Airflow 3.4+) Instead of a new SDK: every line works with the provider today.
AIP-112 Task Steps plan AIP-113 Dynamic Task Groups gate AIP-115 On-Demand Task Sections An agent could keep trying until it converges. A retry could skip the steps that already finished. The model could plan the graph at runtime. A person could run optional parts later. Accepted Draft Draft Draft
leew
elmetal
j341nono
armondando
junhat6
tanahiro2010
suneo3476
ivargrimstad
iwatsukayura
satoshi256kbyte
___akari11
minodriven
glassfiber
aleyda
chriscoyier
badams
samanthasiow
jeffersonlam
tapps
twada
lemiorhan
reverentgeek
greggifford
tessaabrams
![Deployment config [dag_processor] dag_bundle_config_list = [ {"name": "finance", "classpath": "airflow.dag_processing.bundles.local.LocalDagBundle",](https://files.speakerdeck.com/presentations/fa61c161e2a343df861ca93fb5d9175f/slide_15.jpg){kind=link}
![@task.llm class Entities(BaseModel): names: list[str] locations: list[str] @task.llm( llm_conn_id="pydanticai_default", system_prompt="Extract](https://files.speakerdeck.com/presentations/fa61c161e2a343df861ca93fb5d9175f/slide_34.jpg){kind=link}
