20250328 Airflow 3.0 The First Glance

by Lee Wei

Slide 1

Slide 1 text

wei-lee.me RC 1 is expected to happen here

Slide 2

Slide 2 text

wei-lee.me 1

Slide 3

Slide 3 text

wei-lee.me Airflow 3.0 The First Glance 🫣

Slide 4

Slide 4 text

wei-lee.me 2

Slide 5

Slide 5 text

wei-lee.me 3

Slide 6

Slide 6 text

wei-lee.me 4

Slide 7

Slide 7 text

wei-lee.me What is Airflow An open-source platform to programmatically author, schedule, monitor workflows

Slide 8

Slide 8 text

wei-lee.me What is a workflow

Slide 9

Slide 9 text

wei-lee.me What is a workflow

Slide 10

Slide 10 text

wei-lee.me What is a workflow

Slide 11

Slide 11 text

wei-lee.me 5

Slide 12

Slide 12 text

wei-lee.me 6

Slide 13

Slide 13 text

wei-lee.me Airflow is NOT a streaming solution 7

Slide 14

Slide 14 text

wei-lee.me But!

Slide 15

Slide 15 text

wei-lee.me

Slide 16

Slide 16 text

wei-lee.me 8

Slide 17

Slide 17 text

wei-lee.me Why Airflow A large user base

Slide 18

Slide 18 text

wei-lee.me Why Airflow Active Development

Slide 19

Slide 19 text

wei-lee.me Why Airflow Integrations with 3-party projects "JSCZUF "MJCBCB "NB[PO "QQSJTF "TBOB "SBOHP%# "QBDIF4QBSL "QBDIF1JOPU "QBDIF1JH "QBDIF-JWZ "QBDIF,ZMJO "QBDIF,BGLB "QBDIF)JWF "QBDIF)%'4 "QBDIF'MJOL "QBDIF%SVJE "QBDIF%SJMM "QBDIF$BTTBOESB "QBDIF#FBN %PDLFS %JTDPSE %JOHEJOH ECU %BUBEPH %BUBCSJDLT $PNNPO42- $PIFSF ,VCFSOFUFT $FMFSZ +JSB *#.$MPVEBOU )551 )BTIJDPSQ H31$ (PPHMF 'BDFCPPL '51 'BDFCPPL &YBTPM &MBTUJDTFBSDI 0QFO-JOFBHF 0QFO"* 0QFO'BB4 0%#$ /FPK .Z42- .POHP%# 8JO3. .442- 1431 .JDSPTPGU1PXFS4IFMM .JDSPTPGU"[VSF +FOLJOT +%#$ *."1 *OGMVY%# 1BQFSNJMM 1BHFSEVUZ 0SBDMF 0QTHFOJF 0QFO4FBSDI 4FHNFOU 4BNCB 4BMFTGPSDF 3FEJT 1SFTUP 1PTUHSF42- 1JOFDPOF 1H7FDUPS 5BCMFBV 5BCVMBS 44) 42-JUF 4OPXGMBLF 4.51 4MBDL 4JOHVMBSJUZ 4'51 4FOEHSJE 7FSUJDB 5SJOP 5FMFHSBN ;FOEFTL :BOEFY 8FBWJBUF 9

Slide 20

Slide 20 text

wei-lee.me

Slide 21

Slide 21 text

wei-lee.me Airflow 3.0 What's New? 🆕

Slide 22

Slide 22 text

wei-lee.me DAG versioning & Bundles

Slide 23

Slide 23 text

wei-lee.me Modern Web Application

Slide 24

Slide 24 text

wei-lee.me Task Execution Interface

Slide 25

Slide 25 text

wei-lee.me Data Assets & Asset-Centric Syntax

Slide 26

Slide 26 text

wei-lee.me External Event-Driven Scheduling

Slide 27

Slide 27 text

wei-lee.me Modern Web Application

Slide 28

Slide 28 text

wei-lee.me Airflow 1

Slide 29

Slide 29 text

wei-lee.me 10

Slide 30

Slide 30 text

wei-lee.me 11

Slide 31

Slide 31 text

wei-lee.me Airflow 2

Slide 32

Slide 32 text

wei-lee.me Airflow 3

Slide 33

Slide 33 text

wei-lee.me Airflow 3

Slide 34

Slide 34 text

How does Airflow work? A dag Example

Slide 35

Slide 35 text

wei-lee.me 12

Slide 36

Slide 36 text

wei-lee.me 13

Slide 37

Slide 37 text

wei-lee.me How to write a dag

Slide 38

Slide 38 text

wei-lee.me Trigger the dag

Slide 39

Slide 39 text

wei-lee.me The task succeeded and that's it

Slide 40

Slide 40 text

wei-lee.me 14

Slide 41

Slide 41 text

wei-lee.me Other Existing Features before 3.0 • Task retry • Sensors • Deferrable Operators • Dynamic Task Mapping • and many more

Slide 42

Slide 42 text

wei-lee.me 15

Slide 43

Slide 43 text

wei-lee.me Taskflow • Intuitive Python interface • Introduced in Airflow 2.0

Slide 44

Slide 44 text

wei-lee.me 曾有人說 Airflow 沒有...

Slide 45

Slide 45 text

wei-lee.me 16

Slide 46

Slide 46 text

wei-lee.me Event-Driven Scheduling Defining an Asset and its AssetWatcher

Slide 47

Slide 47 text

wei-lee.me Event-Driven Scheduling Asset

Slide 48

Slide 48 text

wei-lee.me Event-Driven Scheduling Schedule on Asset

Slide 49

Slide 49 text

wei-lee.me Event-Driven Scheduling Triggered by event updates

Slide 50

Slide 50 text

wei-lee.me Dag versioning

Slide 51

Slide 51 text

wei-lee.me Dag versioning

Slide 52

Slide 52 text

wei-lee.me Dag versioning

Slide 53

Slide 53 text

wei-lee.me Dag versioning

Slide 54

Slide 54 text

wei-lee.me 17 v1 v2 v3 v4 v5

Slide 55

Slide 55 text

wei-lee.me Data Assets

Slide 56

Slide 56 text

wei-lee.me Asset Define an asset and an asset alias

Slide 57

Slide 57 text

wei-lee.me Asset Emit an asset event

Slide 58

Slide 58 text

wei-lee.me Asset Schedule on multiple Assets

Slide 59

Slide 59 text

wei-lee.me 18

Slide 60

Slide 60 text

wei-lee.me Asset Schedule on an Asset

Slide 61

Slide 61 text

wei-lee.me Asset How the dependency looks like

Slide 62

Slide 62 text

wei-lee.me Asset Create an asset event manually

Slide 63

Slide 63 text

wei-lee.me Asset The second dag is still waiting for another asset

Slide 64

Slide 64 text

wei-lee.me 如果你從來不覺得升版開心過 19

Slide 65

Slide 65 text

wei-lee.me Migration tools

Slide 66

Slide 66 text

wei-lee.me Migration tools airflow config lint

Slide 67

Slide 67 text

wei-lee.me Migration tools airflow config update

Slide 68

Slide 68 text

wei-lee.me Migration tools ruff

Slide 69

Slide 69 text

wei-lee.me Migration tools ruff

Slide 70

Slide 70 text

wei-lee.me Migration tools ruff

Slide 71

Slide 71 text

wei-lee.me Migration tools ruff 20

Slide 72

Slide 72 text

wei-lee.me That's all for Airflow 3 the first glance 👀

Slide 73

Slide 73 text

wei-lee.me 讓我們把時間往前回朔一下 21

Slide 74

Slide 74 text

wei-lee.me 22 23

Slide 75

Slide 75 text

wei-lee.me

Slide 76

Slide 76 text

wei-lee.me

Slide 77

Slide 77 text

wei-lee.me

Slide 78

Slide 78 text

wei-lee.me 24

Slide 79

Slide 79 text

wei-lee.me 25

Slide 80

Slide 80 text

wei-lee.me

Slide 81

Slide 81 text

wei-lee.me Let's contribute to Airflow

Slide 82

Slide 82 text

wei-lee.me 26

Slide 83

Slide 83 text

wei-lee.me 27˙

Slide 84

Slide 84 text

wei-lee.me 我通常會出沒在這些社群

Slide 85

Slide 85 text

wei-lee.me 28 投稿一一

Slide 86

Slide 86 text

wei-lee.me $ cat weilee.py __name__ = 李唯 / Wei Lee __what_i_am_doing__ = [ Volunteer @ PyCon Taiwan, Member @ PAO (former PyCon APAC), Maintainer of commitizen-tools, Mentor (?) @ opensource4you, Committer @ Apache Airflow, Software Engineer @ Astronomer, ] __github__ = G Lee-W __linkedin__ = l clleew __site__ = p https://wei-lee.me

Slide 87

Slide 87 text

wei-lee.me File "weilee.py", line 1 __name__ = 李唯 / Wei Lee ^^^ SyntaxError: invalid syntax $ python weilee.py

Slide 88

Slide 88 text

wei-lee.me

Slide 89

Slide 89 text

wei-lee.me

Slide 90

Slide 90 text

wei-lee.me References • 進擊的巨人 • BanG Dream! It's MyGO!!!!! • BanG Dream! Ave Mujica • 地。-關於地球的運動 • JOJO 的奇妙冒險 • Star Wars: The Force Awakens