Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
A Newcomer's Guide To Airflow's Architecture
Search
Andrew Godwin
July 12, 2021
Programming
0
280
A Newcomer's Guide To Airflow's Architecture
A talk I gave at Airflow Summit 2021.
Andrew Godwin
July 12, 2021
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
Reconciling Everything
andrewgodwin
1
240
Django Through The Years
andrewgodwin
0
130
Writing Maintainable Software At Scale
andrewgodwin
0
370
Async, Python, and the Future
andrewgodwin
2
580
How To Break Django: With Async
andrewgodwin
1
630
Taking Django's ORM Async
andrewgodwin
0
640
The Long Road To Asynchrony
andrewgodwin
0
570
The Scientist & The Engineer
andrewgodwin
1
660
Pioneering Real-Time
andrewgodwin
0
320
Other Decks in Programming
See All in Programming
So you want to write a User-Defined Function for Flink? @ Current 2024 - Austin, TX
hpgrahsl
0
110
フロントエンドの標準仕様をどう追っているか / How I follow the frontend standards specs
petamoriken
3
170
Remix × Cloudflare Pages × Sentry 奮闘記 / remix-pages-sentry
nkzn
1
350
CDKを活用した 大規模コンテナ移行 プロジェクトの紹介
yoyoyopg
0
220
API Platform for Laravel
dunglas
1
1.2k
為醫療加裝Python的引擎
cclai999
0
270
XP2024 っていう国際会議に行ってきたよの記 / XP2024 Conference Report
bonotake
4
110
Applied NLP in the Age of Generative AI
inesmontani
PRO
3
1.1k
tsconfig.jsonの最近の新機能 ファイルパス編
uhyo
6
1k
データサイエンスのフルサイクル開発を実現する機械学習パイプライン
xcnkx
2
420
標準ライブラリの動向とイテレータのパフォーマンス
makki_d
3
180
ビット演算の話 / Let's play with bit operations
kaityo256
PRO
3
140
Featured
See All Featured
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
7
550
Designing for Performance
lara
604
68k
jQuery: Nuts, Bolts and Bling
dougneiner
61
7.4k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
5
170
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
26
4k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
248
21k
Automating Front-end Workflow
addyosmani
1365
200k
Learning to Love Humans: Emotional Interface Design
aarron
271
40k
It's Worth the Effort
3n
182
27k
Art, The Web, and Tiny UX
lynnandtonic
294
20k
How to train your dragon (web standard)
notwaldorf
87
5.6k
Building Better People: How to give real-time feedback that sticks.
wjessup
359
19k
Transcript
A NEWCOMER'S GUIDE TO ANDREW GODWIN // @andrewgodwin AIRFLOW'S ARCHITECTURE
Hi, I’m Andrew Godwin • Principal Engineer at • Also
a Django core developer, ASGI author • Using Airflow since March 2021
None
High-Level Concepts What exactly is going on? The Good and
the Bad Or, How I Learned To Stop Worrying And Love The Scheduler Problems, Fixes & The Future Where we go from here
Differences from things I have worked on? (An eclectic variety
of web and backend systems)
"Real-time" versus batch The availability versus consistency tradeoff is different!
Simple concepts, hard to master In Django, it's the ORM. In Airflow, scheduling. It's all still distributed systems Which is fortunate, after fifteen years of doing them
Airflow grew organically It started off as an internal ETL
tool
None
DAG ➡ DagRun One per scheduled run, as the run
starts Operator ➡ Task When you call an operator in a DAG Task ➡ TaskInstance When a Task needs to run as part of a DagRun
Scheduler Works out what TaskInstances need to run Executor Runs
TaskInstances and records the results
Scheduler LocalExecutor Webserver Database DAG Files
Scheduler CeleryExecutor Webserver Database DAG Files Redis/Queue Workers
The Executor runs inside the Scheduler Its logic, at least,
and the tasks too for local ones
Everything talks to the database It's the single central point
of coordination
Scheduler, Workers, Webserver All can be run in a high-availability
pattern
Scheduler Works out what TaskInstances need to run Executor Runs
TaskInstances and records the results
Scheduler Works out what TaskInstances need to run Executor Runs
TaskInstances and records the results
Timing Dependencies Retries Concurrency Callbacks ...
Scheduler Works out what TaskInstances need to run Executor Runs
TaskInstances and records the results
Celery or Kubernetes Our two main options, currently
Scheduler CeleryExecutor Webserver Database DAG Files Redis/Queue Workers
Scheduler KubernetesExecutor Webserver Database DAG Files Kubernetes Task Pods
None
Tasks are the core part of the model DAGs are
more of a grouping/trigger mechanism
Very flexible runtime environments Airflow's strength, and its weakness
Airflow doesn't know what you're running This is both an
advantage and a disadvantage.
What can we improve? Let's talk about The Future
More Async & Eventing Anything that involves waiting!
Scheduler CeleryExecutor Webserver Database DAG Files Redis/Queue Workers Triggerer
Removing Database Connections APIs scale a lot better!
I do like the database, though There's a lot of
benefit in proven technology
Software Engineering is not just coding Any large-scale project needs
documentation, architecture, and coordination
Maintenance & compatibility is crucial Anyone can write a tool
- supporting it takes effort
Airflow is forged by people like you. Coding, documentation, triage,
QA, support - it all needs doing.
Thanks. Andrew Godwin @andrewgodwin
[email protected]