Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
A Newcomer's Guide To Airflow's Architecture
Search
Andrew Godwin
July 12, 2021
Programming
0
270
A Newcomer's Guide To Airflow's Architecture
A talk I gave at Airflow Summit 2021.
Andrew Godwin
July 12, 2021
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
Reconciling Everything
andrewgodwin
1
230
Django Through The Years
andrewgodwin
0
110
Writing Maintainable Software At Scale
andrewgodwin
0
350
Async, Python, and the Future
andrewgodwin
2
560
How To Break Django: With Async
andrewgodwin
1
610
Taking Django's ORM Async
andrewgodwin
0
620
The Long Road To Asynchrony
andrewgodwin
0
540
The Scientist & The Engineer
andrewgodwin
1
630
Pioneering Real-Time
andrewgodwin
0
310
Other Decks in Programming
See All in Programming
Namespace on read
tagomoris
2
370
CSC307 Lecture 05
javiergs
PRO
0
210
最近追加した型の紹介とその振り返り
aki19035vc
0
170
CSC307 Lecture 13
javiergs
PRO
0
150
Product Management LT会_クアンド新家
shinshin
0
210
さきがけから振り返るアーキテクチャ刷新 / Reflecting on the Architectural Renewal from the Vanguard
nrslib
2
770
Microservices rules (July 2024) : what good looks like
cer
PRO
0
1.6k
日付と正規化
megmogmog1965
0
140
AWS CDKにおける「再利用性」を考える / aws-cdk-reusability
gotok365
6
1.3k
最古の関数型言語「Lisp」ことはじめ / lisp_in_kamiyama
uhooi
1
190
わかりやすい正解を捨てて、コトに向き合う - スクラムフェス金沢2024 スポンサーセッション
yusukekokubo
0
170
CSC307 Lecture 11
javiergs
PRO
0
240
Featured
See All Featured
Java REST API Framework Comparison - PWX 2021
mraible
PRO
20
7.2k
Git: the NoSQL Database
bkeepers
PRO
423
64k
10 Git Anti Patterns You Should be Aware of
lemiorhan
652
58k
Stop Working from a Prison Cell
hatefulcrawdad
266
20k
The Brand Is Dead. Long Live the Brand.
mthomps
52
36k
Building Your Own Lightsaber
phodgson
101
5.9k
Faster Mobile Websites
deanohume
303
30k
Six Lessons from altMBA
skipperchong
24
3.2k
VelocityConf: Rendering Performance Case Studies
addyosmani
321
23k
Practical Orchestrator
shlominoach
185
10k
Ruby is Unlike a Banana
tanoku
96
10k
Designing with Data
zakiwarfel
96
5k
Transcript
A NEWCOMER'S GUIDE TO ANDREW GODWIN // @andrewgodwin AIRFLOW'S ARCHITECTURE
Hi, I’m Andrew Godwin • Principal Engineer at • Also
a Django core developer, ASGI author • Using Airflow since March 2021
None
High-Level Concepts What exactly is going on? The Good and
the Bad Or, How I Learned To Stop Worrying And Love The Scheduler Problems, Fixes & The Future Where we go from here
Differences from things I have worked on? (An eclectic variety
of web and backend systems)
"Real-time" versus batch The availability versus consistency tradeoff is different!
Simple concepts, hard to master In Django, it's the ORM. In Airflow, scheduling. It's all still distributed systems Which is fortunate, after fifteen years of doing them
Airflow grew organically It started off as an internal ETL
tool
None
DAG ➡ DagRun One per scheduled run, as the run
starts Operator ➡ Task When you call an operator in a DAG Task ➡ TaskInstance When a Task needs to run as part of a DagRun
Scheduler Works out what TaskInstances need to run Executor Runs
TaskInstances and records the results
Scheduler LocalExecutor Webserver Database DAG Files
Scheduler CeleryExecutor Webserver Database DAG Files Redis/Queue Workers
The Executor runs inside the Scheduler Its logic, at least,
and the tasks too for local ones
Everything talks to the database It's the single central point
of coordination
Scheduler, Workers, Webserver All can be run in a high-availability
pattern
Scheduler Works out what TaskInstances need to run Executor Runs
TaskInstances and records the results
Scheduler Works out what TaskInstances need to run Executor Runs
TaskInstances and records the results
Timing Dependencies Retries Concurrency Callbacks ...
Scheduler Works out what TaskInstances need to run Executor Runs
TaskInstances and records the results
Celery or Kubernetes Our two main options, currently
Scheduler CeleryExecutor Webserver Database DAG Files Redis/Queue Workers
Scheduler KubernetesExecutor Webserver Database DAG Files Kubernetes Task Pods
None
Tasks are the core part of the model DAGs are
more of a grouping/trigger mechanism
Very flexible runtime environments Airflow's strength, and its weakness
Airflow doesn't know what you're running This is both an
advantage and a disadvantage.
What can we improve? Let's talk about The Future
More Async & Eventing Anything that involves waiting!
Scheduler CeleryExecutor Webserver Database DAG Files Redis/Queue Workers Triggerer
Removing Database Connections APIs scale a lot better!
I do like the database, though There's a lot of
benefit in proven technology
Software Engineering is not just coding Any large-scale project needs
documentation, architecture, and coordination
Maintenance & compatibility is crucial Anyone can write a tool
- supporting it takes effort
Airflow is forged by people like you. Coding, documentation, triage,
QA, support - it all needs doing.
Thanks. Andrew Godwin @andrewgodwin
[email protected]