Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Starts Airflow task execution directly from the...
Search
Lee Wei
May 08, 2024
Technology
0
140
Starts Airflow task execution directly from the triggerer
Lee Wei
May 08, 2024
Tweet
Share
More Decks by Lee Wei
See All by Lee Wei
住手啊,這樣用 Python 寫程式沒有人會獲得幸福的
leew
0
55
Unlocking the Future of Data Pipeline
leew
0
230
Hold on! You have a data team in PyCon Taiwan!
leew
0
21
朝聖之路
leew
0
86
20250328 Airflow 3.0 The First Glance
leew
0
260
踏入開源的第一步
leew
0
220
Unleash the Chaos - Developing a Linter for Un-Pythonic Code!
leew
0
93
Unlocking Python's Core Magic
leew
1
290
Unleash the Chaos - Developing a Linter for Un-Pythonic Code!
leew
1
190
Other Decks in Technology
See All in Technology
書籍『実践 Apache Iceberg』の歩き方
ishikawa_satoru
0
230
DMMの検索システムをSolrからElasticCloudに移行した話
hmaa_ryo
0
230
AI時代の発信活動 ~技術者として認知してもらうための発信法~ / 20251028 Masaki Okuda
shift_evolve
PRO
1
120
入院医療費算定業務をAIで支援する:包括医療費支払い制度とDPCコーディング (公開版)
hagino3000
0
120
Amazon Athena で JSON・Parquet・Iceberg のデータを検索し、性能を比較してみた
shigeruoda
1
190
オブザーバビリティが育むシステム理解と好奇心
maruloop
3
1.6k
SRE × マネジメントレイヤーが挑戦した組織・会社のオブザーバビリティ改革 ― ビジネス価値と信頼性を両立するリアルな挑戦
coconala_engineer
0
290
Azure Well-Architected Framework入門
tomokusaba
1
140
re:Inventに行くまでにやっておきたいこと
nagisa53
0
720
RemoteFunctionを使ったコロケーション
mkazutaka
1
140
AIでデータ活用を加速させる取り組み / Leveraging AI to accelerate data utilization
okiyuki99
6
1.4k
【SORACOM UG Explorer 2025】さらなる10年へ ~ SORACOM MVC 発表
soracom
PRO
0
170
Featured
See All Featured
Why Our Code Smells
bkeepers
PRO
340
57k
Music & Morning Musume
bryan
46
6.9k
Building a Modern Day E-commerce SEO Strategy
aleyda
44
7.9k
Into the Great Unknown - MozCon
thekraken
40
2.1k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
46
2.5k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
285
14k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
34
2.3k
Rails Girls Zürich Keynote
gr2m
95
14k
Making the Leap to Tech Lead
cromwellryan
135
9.6k
RailsConf 2023
tenderlove
30
1.3k
Build your cross-platform service in a week with App Engine
jlugia
234
18k
Facilitating Awesome Meetings
lara
57
6.6k
Transcript
Starts Air fl ow task execution directly from the triggerer
$ cat weilee.py __name__ = 李唯 / Wei Lee __what_i_am_doing__
= [ Software Engineer @ Astronomer, Committer @ Apache Airflow, Volunteers @ PyCon Taiwan, Maintainer of commitizen-tools, ] __github__ = G Lee-W __linkedin__ = l clleew __site__ = p http://wei-lee.me
File "weilee.py", line 1 __name__ = 李唯 / Wei Lee
^^^ SyntaxError: invalid syntax $ python weilee.py
None
How did deferrable operators work before this change?
What has been changed?
What has been changed?
What it could be in the future
But why?
"execute" methods are not always needed This is the only
logic when deferrable is set to true and does not need to be executed in the worker.
How does it affect DAG authors? More ef fi cient
operators to come
How does it affect operator authors? A new way to
implement operators in an async manner
The Original Way
The New Way
How does it work under the hook?
How does it work under the hook?
How does it work under the hook?
Limitations • It is necessary to go back to workers
to run the next_method (e.g., "execute_complete") • This feature does not yet fully support MappedOperator.