Intro to Airflow - From Zero to Hero

Transcript

1. Intro to Airflow From Zero to Hero
2. ❗Disclaimer❗
3. As this is an Airflow sharing, ...
5. $ cat weilee.py __name__ = 李唯 / Wei Lee __what_i_am_doing__ = [ Software Engineer @ Astronomer, Volunteers @ PyCon Taiwan, Maintainer of commitizen-tools, ] __github__ = G Lee-W __linkedin__ = l clleew __site__ = p http://wei-lee.me
6. File "weilee.py", line 1 __name__ = 李唯 / Wei Lee ^^^ SyntaxError: invalid syntax $ python weilee.py
7. Let's start with FAQ
8. Question 1 Are you an airflow committer?
9. No
10. Question 2 Are folks in your team all committers?
11. Well...You tell me. At least I'm not.
12. Question 3 You must have made significant contributions to Airflow before joining Astronomer, right?
13. This is the sole PR I made before joining.
14. Question 4 Does Astronomer have enough votes to push/block whatever feature?
15. The truth is I don't know. From what I heard, we could, but we don't do it this way
16. What is Airflow Apache Airflow™ is an open-source platform for developing, scheduling, and monitoring batch-oriented workflows. - from the Official Site
17. What Airflow is not? ❌ • The best framework to run event-based workflows infinitely. • The best streaming solution.
18. Why Airflow • dynamic • scalable • interactive through UI, CLI, REST API • extensible
19. Why Airflow A large user base
20. Why Airflow Active Development
21. Why Airflow Integrations with 3-party projects "JSCZUF "MJCBCB "NB[PO "QQSJTF "TBOB "SBOHP%# "QBDIF
    4QBSL "QBDIF
    1JOPU "QBDIF
    1JH "QBDIF
    -JWZ "QBDIF
    ,ZMJO "QBDIF
    ,BGLB "QBDIF
    )JWF "QBDIF
    )%'4 "QBDIF
    'MJOL "QBDIF
    %SVJE "QBDIF
    %SJMM "QBDIF
    $BTTBOESB "QBDIF
    #FBN %PDLFS %JTDPSE %JOHEJOH ECU %BUBEPH %BUBCSJDLT $PNNPO
    42- $PIFSF ,VCFSOFUFT $FMFSZ +JSB *#.
    $MPVEBOU )551 )BTIJDPSQ H31$ (PPHMF 'BDFCPPL '51 'BDFCPPL &YBTPM &MBTUJDTFBSDI 0QFO-JOFBHF 0QFO"* 0QFO'BB4 0%#$ /FPK .Z42- .POHP%# 8JO3. .442- 1431 .JDSPTPGU
    1PXFS4IFMM
    .JDSPTPGU
    "[VSF +FOLJOT +%#$ *."1 *OGMVY
    %# 1BQFSNJMM 1BHFSEVUZ 0SBDMF 0QTHFOJF 0QFO
    4FBSDI 4FHNFOU 4BNCB 4BMFTGPSDF 3FEJT 1SFTUP 1PTUHSF42- 1JOFDPOF 1H7FDUPS 5BCMFBV 5BCVMBS 44) 42-JUF 4OPXGMBLF 4.51 4MBDL 4JOHVMBSJUZ 4'51 4FOEHSJE 7FSUJDB 5SJOP 5FMFHSBN ;FOEFTL :BOEFY 8FBWJBUF
22. Why Airflow?
23. Why Airflow
24. How does Airflow work?
25. How does Airflow work? DAG Example
27. How Airflow works?
28. How does Airflow work? Trigger a DAG
29. How does Airflow work?
30. How does Airflow work? WebUI
31. How does Airflow work? XCom
32. How does Airflow work? Components Web Server Metadata Database Scheduler Triggerer Worker DAG Processor
33. How does Airflow work? DAG authors change DAGs code Web Server Metadata Database Scheduler Triggerer Worker DAG Processor
34. How does Airflow work? Users trigger DAGs from WebUI Web Server Metadata Database Scheduler Triggerer Worker DAG Processor
35. How does Airflow work? Users trigger DAGs from WebUI Web Server Metadata Database Scheduler Triggerer Worker DAG Processor
36. How does Airflow work? Users trigger DAGs from WebUI Web Server Metadata Database Scheduler Triggerer Worker DAG Processor
37. How does Airflow work? Users trigger DAGs from WebUI Web Server Metadata Database Scheduler Triggerer Worker DAG Processor
38. How does Airflow work? Deferrable Operators Traditional Deferrable Ref: https://docs.astronomer.io/learn/deferrable-operators
39. How does Airflow work? Deferrable Operators
40. How does Airflow work? Deferrable Operators Web Server Metadata Database Scheduler Triggerer Worker DAG Processor
41. How does Airflow work? Deferrable Operators Web Server Metadata Database Scheduler Triggerer Worker DAG Processor
42. How does Airflow work? Deferrable Operators Web Server Metadata Database Scheduler Triggerer Worker DAG Processor
43. How does Airflow work? Deferrable Operators Web Server Metadata Database Scheduler Triggerer Worker DAG Processor
44. How does Airflow work? Deferrable Operators Web Server Metadata Database Scheduler Triggerer Worker DAG Processor
45. Dataset Data-aware scheduling
46. Dataset Data-aware scheduling
47. Dataset Data-aware scheduling
49. Dataset Data-aware scheduling
50. Dataset Data-aware scheduling
51. Dataset Data-aware scheduling
52. Dataset Data-aware scheduling
53. Dataset Data-aware scheduling
55. Taskflow a cleaner way to write DAG
56. Upcoming things
57. Hybrid Execution
58. DAG Versioning (?)
59. Astronomer products
60. Deprecating
61. Deprecating (?)
62. Ask-Astro
63. Ask-Astro Slack bot in Airflow slack
64. astronomer-cosmos
65. astro-cli
66. astro-cli $ astro dev start
67. Astronomer Click "Get Started Free"!
68. Upcoming evens
69. Airflow Taipei Meetup March 9th @ Dcard
70. Remember I mentioned I'm a ...
71. Maintainer of commitizen-tools
72. Volunteer of PyCon Taiwan
73. BTW we also use Airflow
74. Guess who taught us Airflow
75. Link to this slide