Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Starts Airflow task execution directly from the...
Search
Lee Wei
May 08, 2024
Technology
0
140
Starts Airflow task execution directly from the triggerer
Lee Wei
May 08, 2024
Tweet
Share
More Decks by Lee Wei
See All by Lee Wei
住手啊,這樣用 Python 寫程式沒有人會獲得幸福的
leew
0
50
Unlocking the Future of Data Pipeline
leew
0
200
Hold on! You have a data team in PyCon Taiwan!
leew
0
19
朝聖之路
leew
0
82
20250328 Airflow 3.0 The First Glance
leew
0
240
踏入開源的第一步
leew
0
190
Unleash the Chaos - Developing a Linter for Un-Pythonic Code!
leew
0
87
Unlocking Python's Core Magic
leew
1
280
Unleash the Chaos - Developing a Linter for Un-Pythonic Code!
leew
1
180
Other Decks in Technology
See All in Technology
Azure SynapseからAzure Databricksへ 移行してわかった新時代のコスト問題!?
databricksjapan
0
140
M5製品で作るポン置きセルラー対応カメラ
sayacom
0
150
SoccerNet GSRの紹介と技術応用:選手視点映像を提供するサッカー作戦盤ツール
mixi_engineers
PRO
1
180
20201008_ファインディ_品質意識を育てる役目は人かAIか___2_.pdf
findy_eventslides
1
360
OpenAI gpt-oss ファインチューニング入門
kmotohas
2
990
ZOZOのAI活用実践〜社内基盤からサービス応用まで〜
zozotech
PRO
0
180
GopherCon Tour 概略
logica0419
2
190
o11yで育てる、強い内製開発組織
_awache
3
120
LLM時代にデータエンジニアの役割はどう変わるか?
ikkimiyazaki
0
260
実装で解き明かす並行処理の歴史
zozotech
PRO
1
350
多様な事業ドメインのクリエイターへ 価値を届けるための営みについて
massyuu
1
250
Large Vision Language Modelを用いた 文書画像データ化作業自動化の検証、運用 / shibuya_AI
sansan_randd
0
110
Featured
See All Featured
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
54
3k
Raft: Consensus for Rubyists
vanstee
139
7.1k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
3.1k
Visualization
eitanlees
148
16k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
48
9.7k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
46
7.6k
The Invisible Side of Design
smashingmag
301
51k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
Why You Should Never Use an ORM
jnunemaker
PRO
59
9.6k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
234
17k
Building Applications with DynamoDB
mza
96
6.6k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
34
6.1k
Transcript
Starts Air fl ow task execution directly from the triggerer
$ cat weilee.py __name__ = 李唯 / Wei Lee __what_i_am_doing__
= [ Software Engineer @ Astronomer, Committer @ Apache Airflow, Volunteers @ PyCon Taiwan, Maintainer of commitizen-tools, ] __github__ = G Lee-W __linkedin__ = l clleew __site__ = p http://wei-lee.me
File "weilee.py", line 1 __name__ = 李唯 / Wei Lee
^^^ SyntaxError: invalid syntax $ python weilee.py
None
How did deferrable operators work before this change?
What has been changed?
What has been changed?
What it could be in the future
But why?
"execute" methods are not always needed This is the only
logic when deferrable is set to true and does not need to be executed in the worker.
How does it affect DAG authors? More ef fi cient
operators to come
How does it affect operator authors? A new way to
implement operators in an async manner
The Original Way
The New Way
How does it work under the hook?
How does it work under the hook?
How does it work under the hook?
Limitations • It is necessary to go back to workers
to run the next_method (e.g., "execute_complete") • This feature does not yet fully support MappedOperator.