20250328 Airflow 3.0 The First Glance

Transcript

1. wei-lee.me RC 1 is expected to happen here
2. wei-lee.me 1
3. wei-lee.me Airflow 3.0 The First Glance 🫣
4. wei-lee.me 2
5. wei-lee.me 3
6. wei-lee.me 4
7. wei-lee.me What is Airflow An open-source platform to programmatically author, schedule, monitor workflows
8. wei-lee.me What is a workflow
9. wei-lee.me What is a workflow
10. wei-lee.me What is a workflow
11. wei-lee.me 5
12. wei-lee.me 6
13. wei-lee.me Airflow is NOT a streaming solution 7
14. wei-lee.me But!
15. wei-lee.me
16. wei-lee.me 8
17. wei-lee.me Why Airflow A large user base
18. wei-lee.me Why Airflow Active Development
19. wei-lee.me Why Airflow Integrations with 3-party projects "JSCZUF "MJCBCB "NB[PO "QQSJTF "TBOB "SBOHP%# "QBDIF
    4QBSL "QBDIF
    1JOPU "QBDIF
    1JH "QBDIF
    -JWZ "QBDIF
    ,ZMJO "QBDIF
    ,BGLB "QBDIF
    )JWF "QBDIF
    )%'4 "QBDIF
    'MJOL "QBDIF
    %SVJE "QBDIF
    %SJMM "QBDIF
    $BTTBOESB "QBDIF
    #FBN %PDLFS %JTDPSE %JOHEJOH ECU %BUBEPH %BUBCSJDLT $PNNPO
    42- $PIFSF ,VCFSOFUFT $FMFSZ +JSB *#.
    $MPVEBOU )551 )BTIJDPSQ H31$ (PPHMF 'BDFCPPL '51 'BDFCPPL &YBTPM &MBTUJDTFBSDI 0QFO-JOFBHF 0QFO"* 0QFO'BB4 0%#$ /FPK .Z42- .POHP%# 8JO3. .442- 1431 .JDSPTPGU
    1PXFS4IFMM
    .JDSPTPGU
    "[VSF +FOLJOT +%#$ *."1 *OGMVY
    %# 1BQFSNJMM 1BHFSEVUZ 0SBDMF 0QTHFOJF 0QFO
    4FBSDI 4FHNFOU 4BNCB 4BMFTGPSDF 3FEJT 1SFTUP 1PTUHSF42- 1JOFDPOF 1H7FDUPS 5BCMFBV 5BCVMBS 44) 42-JUF 4OPXGMBLF 4.51 4MBDL 4JOHVMBSJUZ 4'51 4FOEHSJE 7FSUJDB 5SJOP 5FMFHSBN ;FOEFTL :BOEFY 8FBWJBUF 9
20. wei-lee.me
21. wei-lee.me Airflow 3.0 What's New? 🆕
22. wei-lee.me DAG versioning & Bundles
23. wei-lee.me Modern Web Application
24. wei-lee.me Task Execution Interface
25. wei-lee.me Data Assets & Asset-Centric Syntax
26. wei-lee.me External Event-Driven Scheduling
27. wei-lee.me Modern Web Application
28. wei-lee.me Airflow 1
29. wei-lee.me 10
30. wei-lee.me 11
31. wei-lee.me Airflow 2
32. wei-lee.me Airflow 3
33. wei-lee.me Airflow 3
34. How does Airflow work? A dag Example
35. wei-lee.me 12
36. wei-lee.me 13
37. wei-lee.me How to write a dag
38. wei-lee.me Trigger the dag
39. wei-lee.me The task succeeded and that's it
40. wei-lee.me 14
41. wei-lee.me Other Existing Features before 3.0 • Task retry • Sensors • Deferrable Operators • Dynamic Task Mapping • and many more
42. wei-lee.me 15
43. wei-lee.me Taskflow • Intuitive Python interface • Introduced in Airflow 2.0
44. wei-lee.me 曾有 人 說 Airflow 沒有...
45. wei-lee.me 16
46. wei-lee.me Event-Driven Scheduling Defining an Asset and its AssetWatcher
47. wei-lee.me Event-Driven Scheduling Asset
48. wei-lee.me Event-Driven Scheduling Schedule on Asset
49. wei-lee.me Event-Driven Scheduling Triggered by event updates
50. wei-lee.me Dag versioning
51. wei-lee.me Dag versioning
52. wei-lee.me Dag versioning
53. wei-lee.me Dag versioning
54. wei-lee.me 17 v1 v2 v3 v4 v5
55. wei-lee.me Data Assets
56. wei-lee.me Asset Define an asset and an asset alias
57. wei-lee.me Asset Emit an asset event
58. wei-lee.me Asset Schedule on multiple Assets
59. wei-lee.me 18
60. wei-lee.me Asset Schedule on an Asset
61. wei-lee.me Asset How the dependency looks like
62. wei-lee.me Asset Create an asset event manually
63. wei-lee.me Asset The second dag is still waiting for another asset
64. wei-lee.me 如果你從來不覺得升版開 心 過 19
65. wei-lee.me Migration tools
66. wei-lee.me Migration tools airflow config lint
67. wei-lee.me Migration tools airflow config update
68. wei-lee.me Migration tools ruff
69. wei-lee.me Migration tools ruff
70. wei-lee.me Migration tools ruff
71. wei-lee.me Migration tools ruff 20
72. wei-lee.me That's all for Airflow 3 the first glance 👀
73. wei-lee.me 讓我們把時間往前回朔 一 下 21
74. wei-lee.me 22 23
75. wei-lee.me
76. wei-lee.me
77. wei-lee.me
78. wei-lee.me 24
79. wei-lee.me 25
80. wei-lee.me
81. wei-lee.me Let's contribute to Airflow
82. wei-lee.me 26
83. wei-lee.me 27˙
84. wei-lee.me 我通常會出沒在這些社群
85. wei-lee.me 28 投稿 一 一
86. wei-lee.me $ cat weilee.py __name__ = 李唯 / Wei Lee __what_i_am_doing__ = [ Volunteer @ PyCon Taiwan, Member @ PAO (former PyCon APAC), Maintainer of commitizen-tools, Mentor (?) @ opensource4you, Committer @ Apache Airflow, Software Engineer @ Astronomer, ] __github__ = G Lee-W __linkedin__ = l clleew __site__ = p https://wei-lee.me
87. wei-lee.me File "weilee.py", line 1 __name__ = 李唯 / Wei Lee ^^^ SyntaxError: invalid syntax $ python weilee.py
88. wei-lee.me
89. wei-lee.me
90. wei-lee.me References • 進擊的巨 人 • BanG Dream! It's MyGO!!!!! • BanG Dream! Ave Mujica • 地。-關於地球的運動 • JOJO 的奇妙冒險 • Star Wars: The Force Awakens
91. wei-lee.me 28
92. wei-lee.me Q & A 30
93. wei-lee.me QR code links to this slide deck