Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
A Newcomer's Guide To Airflow's Architecture
Search
Andrew Godwin
July 12, 2021
Programming
0
360
A Newcomer's Guide To Airflow's Architecture
A talk I gave at Airflow Summit 2021.
Andrew Godwin
July 12, 2021
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
Reconciling Everything
andrewgodwin
1
320
Django Through The Years
andrewgodwin
0
210
Writing Maintainable Software At Scale
andrewgodwin
0
450
Async, Python, and the Future
andrewgodwin
2
680
How To Break Django: With Async
andrewgodwin
1
730
Taking Django's ORM Async
andrewgodwin
0
730
The Long Road To Asynchrony
andrewgodwin
0
660
The Scientist & The Engineer
andrewgodwin
1
770
Pioneering Real-Time
andrewgodwin
0
440
Other Decks in Programming
See All in Programming
Google I/O recap web編 大分Web祭り2025
kponda
0
2.9k
オホーツクでコミュニティを立ち上げた理由―地方出身プログラマの挑戦 / TechRAMEN 2025 Conference
lemonade_37
2
490
CEDEC2025 長期運営ゲームをあと10年続けるための0から始める自動テスト ~4000項目を50%自動化し、月1→毎日実行にした3年間~
akatsukigames_tech
0
150
AI時代のドメイン駆動設計-DDD実践におけるAI活用のあり方 / ddd-in-ai-era
minodriven
23
9k
書き捨てではなく継続開発可能なコードをAIコーディングエージェントで書くために意識していること
shuyakinjo
1
300
「リーダーは意思決定する人」って本当?~ 学びを現場で活かす、リーダー4ヶ月目の試行錯誤 ~
marina1017
0
240
TROCCO×dbtで実現する人にもAIにもやさしいデータ基盤
nealle
0
330
Understanding Ruby Grammar Through Conflicts
yui_knk
1
120
Kiroの仕様駆動開発から見えてきたAIコーディングとの正しい付き合い方
clshinji
1
140
GUI操作LLMの最新動向: UI-TARSと関連論文紹介
kfujikawa
0
1k
ワープロって実は計算機で
pepepper
2
1.4k
開発チーム・開発組織の設計改善スキルの向上
masuda220
PRO
13
7.5k
Featured
See All Featured
Music & Morning Musume
bryan
46
6.7k
RailsConf 2023
tenderlove
30
1.2k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
110
20k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Automating Front-end Workflow
addyosmani
1370
200k
Into the Great Unknown - MozCon
thekraken
40
2k
Being A Developer After 40
akosma
90
590k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
33
2.4k
Designing for humans not robots
tammielis
253
25k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
139
34k
How GitHub (no longer) Works
holman
315
140k
Navigating Team Friction
lara
189
15k
Transcript
A NEWCOMER'S GUIDE TO ANDREW GODWIN // @andrewgodwin AIRFLOW'S ARCHITECTURE
Hi, I’m Andrew Godwin • Principal Engineer at • Also
a Django core developer, ASGI author • Using Airflow since March 2021
None
High-Level Concepts What exactly is going on? The Good and
the Bad Or, How I Learned To Stop Worrying And Love The Scheduler Problems, Fixes & The Future Where we go from here
Differences from things I have worked on? (An eclectic variety
of web and backend systems)
"Real-time" versus batch The availability versus consistency tradeoff is different!
Simple concepts, hard to master In Django, it's the ORM. In Airflow, scheduling. It's all still distributed systems Which is fortunate, after fifteen years of doing them
Airflow grew organically It started off as an internal ETL
tool
None
DAG ➡ DagRun One per scheduled run, as the run
starts Operator ➡ Task When you call an operator in a DAG Task ➡ TaskInstance When a Task needs to run as part of a DagRun
Scheduler Works out what TaskInstances need to run Executor Runs
TaskInstances and records the results
Scheduler LocalExecutor Webserver Database DAG Files
Scheduler CeleryExecutor Webserver Database DAG Files Redis/Queue Workers
The Executor runs inside the Scheduler Its logic, at least,
and the tasks too for local ones
Everything talks to the database It's the single central point
of coordination
Scheduler, Workers, Webserver All can be run in a high-availability
pattern
Scheduler Works out what TaskInstances need to run Executor Runs
TaskInstances and records the results
Scheduler Works out what TaskInstances need to run Executor Runs
TaskInstances and records the results
Timing Dependencies Retries Concurrency Callbacks ...
Scheduler Works out what TaskInstances need to run Executor Runs
TaskInstances and records the results
Celery or Kubernetes Our two main options, currently
Scheduler CeleryExecutor Webserver Database DAG Files Redis/Queue Workers
Scheduler KubernetesExecutor Webserver Database DAG Files Kubernetes Task Pods
None
Tasks are the core part of the model DAGs are
more of a grouping/trigger mechanism
Very flexible runtime environments Airflow's strength, and its weakness
Airflow doesn't know what you're running This is both an
advantage and a disadvantage.
What can we improve? Let's talk about The Future
More Async & Eventing Anything that involves waiting!
Scheduler CeleryExecutor Webserver Database DAG Files Redis/Queue Workers Triggerer
Removing Database Connections APIs scale a lot better!
I do like the database, though There's a lot of
benefit in proven technology
Software Engineering is not just coding Any large-scale project needs
documentation, architecture, and coordination
Maintenance & compatibility is crucial Anyone can write a tool
- supporting it takes effort
Airflow is forged by people like you. Coding, documentation, triage,
QA, support - it all needs doing.
Thanks. Andrew Godwin @andrewgodwin
[email protected]