Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
A Newcomer's Guide To Airflow's Architecture
Andrew Godwin
July 12, 2021
Programming
0
150
A Newcomer's Guide To Airflow's Architecture
A talk I gave at Airflow Summit 2021.
Andrew Godwin
July 12, 2021
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
Writing Maintainable Software At Scale
andrewgodwin
0
190
Async, Python, and the Future
andrewgodwin
2
440
How To Break Django: With Async
andrewgodwin
1
420
Taking Django's ORM Async
andrewgodwin
0
400
The Long Road To Asynchrony
andrewgodwin
0
420
The Scientist & The Engineer
andrewgodwin
1
440
Pioneering Real-Time
andrewgodwin
0
190
Just Add Await: Retrofitting Async Into Django
andrewgodwin
2
1.2k
Terrain, Art, Python and LiDAR
andrewgodwin
1
270
Other Decks in Programming
See All in Programming
SHOWROOMの分析目的を意識した伝え方・コミュニケーション
hatapu
0
230
Swift Observation
shiz
3
270
Hasura の Relationship と権限管理
karszawa
0
160
OIDC仕様に準拠した Makuake ID連携基盤構築の裏側
ymtdzzz
0
390
PHPDocにおける配列の型定義を少し知る
shimabox
1
130
ちょうぜつ改め21世紀ふつうのソフトウェア設計
tanakahisateru
7
6.3k
ポケモンで学ぶiOS 16弾丸ツアー 🚅
giginet
PRO
1
610
jq at the Shortcuts
cockscomb
1
400
AWSにおける標的型Bot対策
hacomono
0
400
tidy_rpart
bk_18
0
580
Gradle build: The time is now
nonews
1
440
Azure Functionsをサクッと開発、サクッとデプロイ/vscodeconf2023-baba
nina01
1
330
Featured
See All Featured
How GitHub Uses GitHub to Build GitHub
holman
465
280k
Clear Off the Table
cherdarchuk
79
290k
A Modern Web Designer's Workflow
chriscoyier
689
180k
Unsuck your backbone
ammeep
659
56k
How To Stay Up To Date on Web Technology
chriscoyier
779
250k
Streamline your AJAX requests with AmplifyJS and jQuery
dougneiner
128
8.8k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
236
1.1M
Scaling GitHub
holman
453
140k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
500
130k
Six Lessons from altMBA
skipperchong
15
2.3k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
44
14k
The Cult of Friendly URLs
andyhume
68
5.1k
Transcript
A NEWCOMER'S GUIDE TO ANDREW GODWIN // @andrewgodwin AIRFLOW'S ARCHITECTURE
Hi, I’m Andrew Godwin • Principal Engineer at • Also
a Django core developer, ASGI author • Using Airflow since March 2021
None
High-Level Concepts What exactly is going on? The Good and
the Bad Or, How I Learned To Stop Worrying And Love The Scheduler Problems, Fixes & The Future Where we go from here
Differences from things I have worked on? (An eclectic variety
of web and backend systems)
"Real-time" versus batch The availability versus consistency tradeoff is different!
Simple concepts, hard to master In Django, it's the ORM. In Airflow, scheduling. It's all still distributed systems Which is fortunate, after fifteen years of doing them
Airflow grew organically It started off as an internal ETL
tool
None
DAG ➡ DagRun One per scheduled run, as the run
starts Operator ➡ Task When you call an operator in a DAG Task ➡ TaskInstance When a Task needs to run as part of a DagRun
Scheduler Works out what TaskInstances need to run Executor Runs
TaskInstances and records the results
Scheduler LocalExecutor Webserver Database DAG Files
Scheduler CeleryExecutor Webserver Database DAG Files Redis/Queue Workers
The Executor runs inside the Scheduler Its logic, at least,
and the tasks too for local ones
Everything talks to the database It's the single central point
of coordination
Scheduler, Workers, Webserver All can be run in a high-availability
pattern
Scheduler Works out what TaskInstances need to run Executor Runs
TaskInstances and records the results
Scheduler Works out what TaskInstances need to run Executor Runs
TaskInstances and records the results
Timing Dependencies Retries Concurrency Callbacks ...
Scheduler Works out what TaskInstances need to run Executor Runs
TaskInstances and records the results
Celery or Kubernetes Our two main options, currently
Scheduler CeleryExecutor Webserver Database DAG Files Redis/Queue Workers
Scheduler KubernetesExecutor Webserver Database DAG Files Kubernetes Task Pods
None
Tasks are the core part of the model DAGs are
more of a grouping/trigger mechanism
Very flexible runtime environments Airflow's strength, and its weakness
Airflow doesn't know what you're running This is both an
advantage and a disadvantage.
What can we improve? Let's talk about The Future
More Async & Eventing Anything that involves waiting!
Scheduler CeleryExecutor Webserver Database DAG Files Redis/Queue Workers Triggerer
Removing Database Connections APIs scale a lot better!
I do like the database, though There's a lot of
benefit in proven technology
Software Engineering is not just coding Any large-scale project needs
documentation, architecture, and coordination
Maintenance & compatibility is crucial Anyone can write a tool
- supporting it takes effort
Airflow is forged by people like you. Coding, documentation, triage,
QA, support - it all needs doing.
Thanks. Andrew Godwin @andrewgodwin
[email protected]