Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
A Newcomer's Guide To Airflow's Architecture
Search
Andrew Godwin
July 12, 2021
Programming
0
240
A Newcomer's Guide To Airflow's Architecture
A talk I gave at Airflow Summit 2021.
Andrew Godwin
July 12, 2021
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
Reconciling Everything
andrewgodwin
1
200
Django Through The Years
andrewgodwin
0
86
Writing Maintainable Software At Scale
andrewgodwin
0
330
Async, Python, and the Future
andrewgodwin
2
540
How To Break Django: With Async
andrewgodwin
1
570
Taking Django's ORM Async
andrewgodwin
0
580
The Long Road To Asynchrony
andrewgodwin
0
510
The Scientist & The Engineer
andrewgodwin
1
570
Pioneering Real-Time
andrewgodwin
0
270
Other Decks in Programming
See All in Programming
エンターテイメント業界で利用されるAWS
demuyan
0
210
二郎系ラーメンのコールで学ぶ AST 解析
memory1994
PRO
7
1.7k
Random\Randomizer クラスで日常のあれこれを解決しよう! / Random\Randomizer class solves familiar trouble
cocoeyes02
0
210
スクラムガイドのスプリントレトロスペクティブを改めて読みかえしてみた / Re-reading the Sprint Retrospective Section in the Scrum Guide
mackey0225
3
410
Snowflakeで眠ったデータを起こそう!
estie
0
110
大規模Reactアプリのリアーキテクチャ~8万行のTanStack Query移行の軌跡~
kj455
4
950
Hanami and htmx
bkuhlmann
0
210
Ruby Pattern Matching
bkuhlmann
0
920
FigmaとPHPで作る1ミリたりとも表示崩れしない最強の帳票印刷ソリューション
ttskch
43
19k
Node.js v22 で変わること
yosuke_furukawa
PRO
5
2.2k
Ruby Function Composition
bkuhlmann
1
330
データアナリストが行うDatabricksを活用したETLの自動化事例
shinoa
0
260
Featured
See All Featured
A designer walks into a library…
pauljervisheath
200
23k
A Modern Web Designer's Workflow
chriscoyier
689
190k
Build The Right Thing And Hit Your Dates
maggiecrowley
24
2k
Infographics Made Easy
chrislema
238
18k
Become a Pro
speakerdeck
PRO
11
4.5k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
125
32k
Code Reviewing Like a Champion
maltzj
514
39k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
30
6k
The Straight Up "How To Draw Better" Workshop
denniskardys
227
130k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
60
14k
Designing for Performance
lara
601
67k
YesSQL, Process and Tooling at Scale
rocio
164
13k
Transcript
A NEWCOMER'S GUIDE TO ANDREW GODWIN // @andrewgodwin AIRFLOW'S ARCHITECTURE
Hi, I’m Andrew Godwin • Principal Engineer at • Also
a Django core developer, ASGI author • Using Airflow since March 2021
None
High-Level Concepts What exactly is going on? The Good and
the Bad Or, How I Learned To Stop Worrying And Love The Scheduler Problems, Fixes & The Future Where we go from here
Differences from things I have worked on? (An eclectic variety
of web and backend systems)
"Real-time" versus batch The availability versus consistency tradeoff is different!
Simple concepts, hard to master In Django, it's the ORM. In Airflow, scheduling. It's all still distributed systems Which is fortunate, after fifteen years of doing them
Airflow grew organically It started off as an internal ETL
tool
None
DAG ➡ DagRun One per scheduled run, as the run
starts Operator ➡ Task When you call an operator in a DAG Task ➡ TaskInstance When a Task needs to run as part of a DagRun
Scheduler Works out what TaskInstances need to run Executor Runs
TaskInstances and records the results
Scheduler LocalExecutor Webserver Database DAG Files
Scheduler CeleryExecutor Webserver Database DAG Files Redis/Queue Workers
The Executor runs inside the Scheduler Its logic, at least,
and the tasks too for local ones
Everything talks to the database It's the single central point
of coordination
Scheduler, Workers, Webserver All can be run in a high-availability
pattern
Scheduler Works out what TaskInstances need to run Executor Runs
TaskInstances and records the results
Scheduler Works out what TaskInstances need to run Executor Runs
TaskInstances and records the results
Timing Dependencies Retries Concurrency Callbacks ...
Scheduler Works out what TaskInstances need to run Executor Runs
TaskInstances and records the results
Celery or Kubernetes Our two main options, currently
Scheduler CeleryExecutor Webserver Database DAG Files Redis/Queue Workers
Scheduler KubernetesExecutor Webserver Database DAG Files Kubernetes Task Pods
None
Tasks are the core part of the model DAGs are
more of a grouping/trigger mechanism
Very flexible runtime environments Airflow's strength, and its weakness
Airflow doesn't know what you're running This is both an
advantage and a disadvantage.
What can we improve? Let's talk about The Future
More Async & Eventing Anything that involves waiting!
Scheduler CeleryExecutor Webserver Database DAG Files Redis/Queue Workers Triggerer
Removing Database Connections APIs scale a lot better!
I do like the database, though There's a lot of
benefit in proven technology
Software Engineering is not just coding Any large-scale project needs
documentation, architecture, and coordination
Maintenance & compatibility is crucial Anyone can write a tool
- supporting it takes effort
Airflow is forged by people like you. Coding, documentation, triage,
QA, support - it all needs doing.
Thanks. Andrew Godwin @andrewgodwin
[email protected]