Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Starts Airflow task execution directly from the...
Search
Lee Wei
May 08, 2024
Technology
0
85
Starts Airflow task execution directly from the triggerer
Lee Wei
May 08, 2024
Tweet
Share
More Decks by Lee Wei
See All by Lee Wei
Unleash the Chaos - Developing a Linter for Un-Pythonic Code!
leew
0
33
Unlocking Python's Core Magic
leew
1
220
Unleash the Chaos - Developing a Linter for Un-Pythonic Code!
leew
1
130
Intro to Airflow - From Zero to Hero
leew
0
870
Does Kobayashi-San write code vulnerable to SQL injection?
leew
0
96
Back to the last time we were here
leew
0
25
Atomic Commits - An Easy & Proven Way to Manage & Automate Release Process
leew
0
430
Python Table Manners @ Taichung.py
leew
0
340
Python Table Manners- Cut the Cookie Gracefully @ Hualien py
leew
0
200
Other Decks in Technology
See All in Technology
完璧を捨てろ! “攻め”のQAがもたらすスピードと革新/20250306 Hiroki Hachisuka
shift_evolve
0
150
AIエージェント開発のノウハウと課題
pharma_x_tech
9
5.2k
MLflowはどのようにLLMOpsの課題を解決するのか
taka_aki
0
150
AIエージェント入門
minorun365
PRO
35
20k
DeepSeekとは?何がいいの? - Databricksと学ぶDeepSeek! 〜これからのLLMに備えよ!〜
taka_aki
2
190
エンジニア主導の企画立案を可能にする組織とは?
recruitengineers
PRO
1
320
マルチアカウント環境における組織ポリシーについて まとめてみる
nrinetcom
PRO
2
110
データモデルYANGの処理系を再発明した話
tjmtrhs
0
380
自分のやることに価値を見出だせるようになり、挑戦する勇気をもらったベイトソンの考え / Scrum Fest Fukuoka 2025
bonbon0605
0
140
AIエージェント元年@日本生成AIユーザ会
shukob
1
270
Oracle Database Technology Night #87-1 : Exadata Database Service on Exascale Infrastructure(ExaDB-XS)サービス詳細
oracle4engineer
PRO
1
230
どちらかだけじゃもったいないかも? ECSとEKSを適材適所で併用するメリット、運用課題とそれらの対応について
tk3fftk
2
310
Featured
See All Featured
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
47
5.2k
Designing on Purpose - Digital PM Summit 2013
jponch
117
7.1k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
32
2.2k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
11
540
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
Documentation Writing (for coders)
carmenintech
69
4.6k
[RailsConf 2023] Rails as a piece of cake
palkan
53
5.3k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
Art, The Web, and Tiny UX
lynnandtonic
298
20k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
227
22k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
356
29k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
49
2.3k
Transcript
Starts Air fl ow task execution directly from the triggerer
$ cat weilee.py __name__ = 李唯 / Wei Lee __what_i_am_doing__
= [ Software Engineer @ Astronomer, Committer @ Apache Airflow, Volunteers @ PyCon Taiwan, Maintainer of commitizen-tools, ] __github__ = G Lee-W __linkedin__ = l clleew __site__ = p http://wei-lee.me
File "weilee.py", line 1 __name__ = 李唯 / Wei Lee
^^^ SyntaxError: invalid syntax $ python weilee.py
None
How did deferrable operators work before this change?
What has been changed?
What has been changed?
What it could be in the future
But why?
"execute" methods are not always needed This is the only
logic when deferrable is set to true and does not need to be executed in the worker.
How does it affect DAG authors? More ef fi cient
operators to come
How does it affect operator authors? A new way to
implement operators in an async manner
The Original Way
The New Way
How does it work under the hook?
How does it work under the hook?
How does it work under the hook?
Limitations • It is necessary to go back to workers
to run the next_method (e.g., "execute_complete") • This feature does not yet fully support MappedOperator.