Unlocking the Future of Data Pipeline

Transcript

1. wei-lee.me Unlocking the Future of Data Pipeline Apache Airflow 3
2. wei-lee.me
3. wei-lee.me 🛝 Slide deck
4. wei-lee.me What to expect? • Intro to Airflow • What's new in Airflow 3 • Migrate to Airflow 3
5. wei-lee.me Intro to Airflow
6. wei-lee.me What is Airflow An open-source platform to programmatically author, schedule, and monitor workflows
7. wei-lee.me What is a workflow
8. wei-lee.me What is a workflow
9. wei-lee.me What is a workflow the programmatically part
10. wei-lee.me What is a workflow the programmatically part
11. wei-lee.me What is a workflow the programmatically part
12. wei-lee.me What is a workflow the programmatically part
13. wei-lee.me What is a workflow the programmatically part
14. wei-lee.me What is a workflow the programmatically part
15. wei-lee.me 12
16. wei-lee.me What is a Dag Dag == Workflow in Airflow
17. wei-lee.me What is a Dag Dag == Workflow in Airflow.
18. wei-lee.me If you ever heard of a DAG or directed acyclic graph...
19. wei-lee.me
20. wei-lee.me Where do the tasks come from? Under the hook, it's a PythonOperator.
21. wei-lee.me What are Operators Examples • EmailOperator • HttpOperator • SQLExecuteQueryOperator • DockerOperator • HiveOperator • S3FileTransformOperator • PrestoToMySqlOperator • SlackAPIOperator • and etc.
22. wei-lee.me a template for a predefined task What are Operators Definition
23. wei-lee.me What are Operators Lots and lots of operators "JSCZUF "MJCBCB "NB[PO "QQSJTF "TBOB "SBOHP%# "QBDIF
    4QBSL "QBDIF
    1JOPU "QBDIF
    1JH "QBDIF
    -JWZ "QBDIF
    ,ZMJO "QBDIF
    ,BGLB "QBDIF
    )JWF "QBDIF
    )%'4 "QBDIF
    'MJOL "QBDIF
    %SVJE "QBDIF
    %SJMM "QBDIF
    $BTTBOESB "QBDIF
    #FBN %PDLFS %JTDPSE %JOHEJOH ECU %BUBEPH %BUBCSJDLT $PNNPO
    42- $PIFSF ,VCFSOFUFT $FMFSZ +JSB *#.
    $MPVEBOU )551 )BTIJDPSQ H31$ (PPHMF 'BDFCPPL '51 'BDFCPPL &YBTPM &MBTUJDTFBSDI 0QFO-JOFBHF 0QFO"* 0QFO'BB4 0%#$ /FPK .Z42- .POHP%# 8JO3. .442- 1431 .JDSPTPGU
    1PXFS4IFMM
    .JDSPTPGU
    "[VSF +FOLJOT +%#$ *."1 *OGMVY
    %# 1BQFSNJMM 1BHFSEVUZ 0SBDMF 0QTHFOJF 0QFO
    4FBSDI 4FHNFOU 4BNCB 4BMFTGPSDF 3FEJT 1SFTUP 1PTUHSF42- 1JOFDPOF 1H7FDUPS 5BCMFBV 5BCVMBS 44) 42-JUF 4OPXGMBLF 4.51 4MBDL 4JOHVMBSJUZ 4'51 4FOEHSJE 7FSUJDB 5SJOP 5FMFHSBN ;FOEFTL :BOEFY 8FBWJBUF 9
24. wei-lee.me Recap • Airflow → programatically orchestrating workflows • Dag == Airflow Workflows → A bunch of tasks • Operator → template for a task
25. wei-lee.me What's new in Airflow 3
26. wei-lee.me Airflow Improvment Proposals • UI (AIP-38, AIP-79, AIP-84) • Data Awareness (AIP-73, AIP-74, AIP-75) • External Event Driven (AIP-82) • Dag Versioning (AIP-63, AIP-64, AIP-65, AIP-66) • Task SDK (AIP-72) • Out of scope today (AIP-69, AIP-78, AIP-81, AIP-83)
27. wei-lee.me User Interface • AIP-38: Modern Web Application • AIP-79: Remove Flask AppBuilder as Core dependency • AIP-84: UI REST API
28. wei-lee.me Airflow 1
29. wei-lee.me 10
30. wei-lee.me Airflow 2
31. wei-lee.me Everyone really has no taste. 11
32. wei-lee.me Airflow 3
33. wei-lee.me Airflow 3
34. wei-lee.me Under the hook • Rewrite the frontend using React • Remove Flask AppBuilder from core • Redesign API endpoints using FastAPI (Get rid of the 6,000 lines views.py)
35. wei-lee.me Legacy views.py 😱
36. wei-lee.me Data Awareness • AIP-73 Expanded Data Awareness • AIP-74 Introducing Data Assets • AIP-75 New Asset-Centric Syntax • AIP-76 Asset Partitions (TODO)
37. wei-lee.me What if we have... • Team A: focus only on training the model • Team B: using the models to generate a report • These 2 teams don't share code with each other
38. wei-lee.me
39. wei-lee.me The uri here is just an identifier
40. wei-lee.me
41. wei-lee.me
42. wei-lee.me @asset
43. wei-lee.me What if the 2 teams don't have access to the same metacase (can't access the same asset)
44. wei-lee.me External Event Driven • AIP-82 External event driven scheduling in Airflow
45. wei-lee.me
46. wei-lee.me
47. wei-lee.me More details on how these Asset things can be used?
48. wei-lee.me Next sesion in the same room
49. wei-lee.me Dag Versioning • AIP-63: Dag Versioning • AIP-64: Keep TaskInstance try history • AIP-65: Improve Dag history in UI • AIP-66: Dag Bundles & Parsing
50. wei-lee.me Dag versioning
51. wei-lee.me Dag versioning
52. wei-lee.me Dag versioning
53. wei-lee.me Dag versioning
54. wei-lee.me 17 v1 v2 v3 v4 v5
55. wei-lee.me Task SDK • AIP-72 Task Execution Interface aka Task SDK
56. wei-lee.me
57. wei-lee.me
58. wei-lee.me
59. wei-lee.me
60. wei-lee.me
61. wei-lee.me Migrate to Airflow 3
62. wei-lee.me Never enjoyed being in a band. migrating Airflow 19
63. wei-lee.me Airflow Migration Tools
64. wei-lee.me But I still need to update lots of Dag code 🫠
65. wei-lee.me Ruff AIR rules
66. wei-lee.me Ruff AIR Rules Best practices • AIR001 Task Variable name should match task_id • AIR002 A Dag should have an explicit schedule argument
67. wei-lee.me Ruff AIR Rules Required changes for migrating to Airflow 3 • AIR301 Something removed in Airflow 3 • AIR302 Something moved from Ariflow core to Airflow provider in Airflow 3
68. wei-lee.me Ruff AIR Rules Suggested changes for migrating to Airflow 3 • AIR311 Similiar to AIR301, but it won't break your code for the time being (We added a backward-compatible layer) • AIR312 Similiar to AIR302, but it won't break your code for the time being
69. wei-lee.me How we migrate PyCon TW's airflow to airflow 3
70. wei-lee.me $ airflow db migrate Contributed back to Airflow
71. wei-lee.me $ airflow config lint
72. wei-lee.me $ uvx ruff check dags \ --preview \ --select=AIR3 \ --fix \ --unsafe-fixes
73. wei-lee.me Ruff AIR rocks 🚀
74. wei-lee.me It's almost like a miracle.
75. wei-lee.me More Ruff @ Airflow Summit 2025
76. wei-lee.me
77. wei-lee.me What's new in Airflow 3.1! • i18n (Taiwanese Mandarin support!) • Human-in-the-loop (AIP-90) • go-sdk (part of AIP-72) • UI-plugins (AIP-68) • etc.
78. wei-lee.me Internationalization (i18n) 1.English 2.Arabic 3.Catalan 4.German 5.Spanish 6.French 7.Hebrew 8.Hindi 9.Hungarian 10.Korean 11.Dutch 12.Polish 13.Taiwanese Mandarin 14.Turkish
79. wei-lee.me
80. wei-lee.me A friend from the Taiwan community initiated it!
81. wei-lee.me I like to be the first, No.1
82. wei-lee.me Human-in-the-loop • AIP-90 Human in the loop
83. wei-lee.me Tasks awaiting human input
84. wei-lee.me go-sdk
85. wei-lee.me go-sdk/example/main.go
86. wei-lee.me
87. wei-lee.me UI plugins
88. wei-lee.me AIP-68 Extended Plugin Interface for React Views
89. wei-lee.me That's it.
90. wei-lee.me I want to join
91. wei-lee.me Let's contribute to Airflow
92. wei-lee.me Python Taiwan Discord Channel
93. wei-lee.me __name__ = 李唯 / Wei Lee __what_i_am_doing__ = [ Everywhere @ PyCon Taiwan, Member @ Python Asia Organization, Meme Bot @ OpenSource4You, Mentor @ OpenSource4You, Maintainer @ commitizen-tools, Committer @ Apache Airflow, Software Engineer @ Astronomer, ] __github__ = Lee-W __linkedin__ = clleew __site__ = https://wei-lee.me $ cat weilee.py
94. wei-lee.me File "weilee.py", line 1 __name__ = 李唯 / Wei Lee ^^^ SyntaxError: invalid syntax $ 𝜋 thon weilee.py
95. wei-lee.me Stay Here for another great Airflow session!
96. wei-lee.me Join us for another Airflow discussion tomorrow @ 12:30!
97. wei-lee.me References • MIB • Fate/Zero • BanG Dream! It's MyGO!!!!! • JOJO 的奇妙冒險 • JOJO 的奇妙冒險 黃 金 之風 • 地。-關於地球的運動
98. wei-lee.me Thanks for your time!
99. wei-lee.me I’m sure you have some questions on your mind too.
100. wei-lee.me Q & A
101. wei-lee.me Q1: What's the cool editor you're using?
102. wei-lee.me
103. wei-lee.me Q & A