Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Apache Airflow Introduction

Lee Wei
November 16, 2026

Apache Airflow Introduction

Lee Wei

November 16, 2026

Transcript

61 slides
  1. A Dag is a Python file from airflow.sdk import dag,

    task @dag(schedule="0 7 * * *") def daily_report(): @task def extract(): return [3, 1, 4] @task def transform(rows): return sum(rows) @task def load(total): print(f"total={total}") load(transform(extract())) daily_report() 9
  2. A Dag is a Python file from airflow.sdk import dag,

    task @dag(schedule="0 7 * * *") def daily_report(): @task def extract(): return [3, 1, 4] @task def transform(rows): return sum(rows) @task def load(total): print(f"total={total}") load(transform(extract())) daily_report() 10
  3. Run it at 7 every morning Your cron script 0

    7 * * * python report.py Airflow @dag(schedule="0 7 * * *") 11
  4. Try again when it fails Your cron script for attempt

    in range(3): try: build_report() break except Exception: time.sleep(60) Airflow @task(retries=3) 12
  5. Tell someone when it fails Your cron script # I

    believe it won't fail Airflow @task(on_failure_callback=notify_team) 13
  6. See it in a browser Your cron script Airflow ssh

    report-server tail report.log 14
  7. Rerun the days you missed Your cron script for day

    in 01 02 03; do python report.py \ --date "2026-11-$day" done Airflow airflowctl backfill create \ --dag-id daily_report \ --from-date 2026-11-01 \ --to-date 2026-11-03 15
  8. Airflow is core plus providers Core apache-airflow-core · apache-airflow-task-sdk Providers

    · 80+ common.sql common.ai snowflake … common.io … amazon google 17
  9. Airflow is core plus providers Core apache-airflow-core · apache-airflow-task-sdk Providers

    · 80+ common.sql common.ai snowflake … common.io … amazon google 18
  10. Ask a model to write it @dag(schedule="0 7 * *

    *") def daily_report(): @task def extract(): return [3, 1, 4] @task.llm(llm_conn_id="pydanticai_default") def summarize(rows): return f"Summarize these numbers in two sentences: {rows}" @task def load(summary): print(summary) load(summarize(extract())) 20
  11. Swap the vendor, not the code llm_conn_id="pydanticai_default" ↓ type pydanticai

    password •••••• model openai:gpt-5 Switching vendors is a connection change, not a Dag change. 23
  12. When the model call fails @task.llm( llm_conn_id="pydanticai_default", retries=3 , output_retries=3

    , fallback_conn_ids=["anthropic_prod"] , ) # rerun the task # ask the model again # try another vendor Fallback: the next connection, inside the same attempt 30
  13. Or let a model decide if a retry is worth

    it @task( retries=3, retry_policy=LLMRetryPolicy(llm_conn_id="pydanticai_default") , ) def build_report(): ... Works on any task · Airflow 3.3+ The model reads the error: retry or not, and how long to wait 31
  14. Write, judge, repeat: Task Loops @task_group def refine(): draft =

    write_draft() judge(draft) # a model writes # a model scores it refinement = refine.loop( max_iterations=10 , until=good_enough) New in Airflow 3.4 · max_iterations is required 34
  15. A model call is just another task Retries & fallback

    A budget retries , output_retries , fallback_conn_ids usage_limits Human approval Loops require_approval .loop(until=…) 36
  16. If you already run Airflow: new in 3.4 Airflow 3.4.0

    · released 6 November 2026 Task Loops Graceful pause Beyond Python You just saw them: repeat until good enough Drain a Dag, then it pauses itself Dags in TypeScript, Go or Java 39
  17. Graceful pause: drain a Dag Running runs finish, then the

    Dag pauses itself. New state: draining · the UI offers Drain or Keep paused 40
  18. Write Java? Or TypeScript? Or Go? Now you can write

    Airflow Dags in Java, TypeScript and Go. Experimental language SDKs in 3.4 41
  19. 44

  20. 日本語 locale is about half done 695 of 1,319 keys

    translated, 52.7% Dashed line: 90%, the bar for a "complete" locale · breeze, 2026-11-16 45
  21. Contribute: start with Breeze ./scripts/tools/setup_breeze breeze start-airflow From a clone

    of apache/airflow · the dev environment contributors use 49
  22. Open source. Open governance. An Apache Software Foundation project ·

    Apache License 2.0 Releases voted on by the PMC 56
  23. One Dag One language One PR Write one for a

    script you run with cron Help translate the UI into yours Send it to apache/airflow まずは小さなPRから、ぜひ送ってみてください。 58
  24. Wei Lee / 李唯 リ ウェイ Apache Airflow PMC member

    · commitizen-tools maintainer wei-lee.me 60

More Decks by Lee Wei

Other Decks in Programming