Unlocking Python's Core Magic

Transcript

1. wei-lee.me 🔐 Unlocking Python's Core Magic
2. wei-lee.me QR Code links to the slide deck
3. wei-lee.me Do you know commitizen-tools?
4. wei-lee.me commitizen-tools Create committing rules for projects 🚀 auto bump versions ⬆ and auto changelog generation 📂
5. wei-lee.me Like, subscribe, and contribute
6. wei-lee.me Do you know Apache Airflow?
7. wei-lee.me Apache Airflow Apache Airflow® is an open-source platform for developing, scheduling, and monitoring batch-oriented workflows.
8. wei-lee.me Like, subscribe, and contribute
9. wei-lee.me
10. wei-lee.me Hey, I thought this was going to be a Python core-centric talk. Are we going to discuss libraries now?
11. wei-lee.me
12. wei-lee.me What is this talk about? * Python core magics and standard libraries * Real-world examples of how these techniques are used in those libraries
13. wei-lee.me
14. wei-lee.me This talk is highly inspired by
15. wei-lee.me And that's why I'll probably also talk too much. There are 135 pages.
16. wei-lee.me Dunder methods
17. wei-lee.me What dunder methods? • Methods that start and end with __ (e.g., __init__) • Dunder → Double Underscore • Allow us to customize classes in a more Pythonic way • Implicitly called by Python • Officially named as special methods
18. wei-lee.me List of dunder methods 1. x.__init__(a, b) 2. T.__new__(T, a, b) 3. x.__del__() 4. x.__eq__(y) 5. x.__ne__(y) 6. x.__lt__(y) 7. x.__rt__(y) 8. x.__le__(y) 9. x.__ge__(y) 10. x.__hash__() 11. x.__repr__() 12. x.__str__() 13. x.__bool__() 14. x.__int__() 15. x.__float__() 16. x.__bytes__() 17. x.__complex__() 18. x.__format__(s) 19. x.__enter__() 20. x.__exit__() 21. x.__len__() 22. x.__iter__() 23. x.__getitem__(a) 24. x.__setitem__(a, b) 25. x.__delitem__(a) 26. x.__contains__(a) 27. x.__reversed__() 28. x.__next__() 29. x.__missing__(a) 30. x.__length_hint__() 31. x.__add__(y) 32. y.__radd__(x) 33. x.__sub__(y) 34. y.__rsub__(x) 35. x.__mul__(y) 36. y.__rmul__(x) 37. x.__truediv__(y) 38. y.__rtruediv__(x) 39. x.__mod__(y) 40. y.__rmod__(x) 41. x.__floordiv__(y) 42. y.__rfloordiv__(x) 43. x.__pow__(y) 44. y.__rpow__(x) 45. x.__matmul__(y) 46. y.__rmatmul__(x) 47. x.__and__(y) 48. y.__rand__(x) 49. x.__or__(y) 50. y.__ror__(x) 51. x.__xor__(y) 52. y.__rxor__(x) 53. x.__rshift__(y) 54. y.__rrshift__(x) 55. x.__lshift__(y) 56. y.__rlshift__(x) 57. x.__neg__() 58. x.__pos__() 59. x.__invert__() 60. x.__divmod__(y) 61. x.__abs__() 62. x.__index__() 63. x.__round__() 64. x.__trunc__() 65. x.__floor__() 66. x.__ceil__() 67. x.__iadd__(y) 68. x.__isub__(y) 69. x.__imul__(y) 70. x.__itruediv__(y) 71. T.x.__set_name__(T, 'x') 72. T.x.__get__(t, T) 73. T.x.__set__(t, y) 74. T.x.__delete__(t) 75. T.__init_subclass__(U) 91. x.__ixor__(y) 92. x.__irshift__(y) 93. x.__ilshift__(y) 94. x.__getattribute__('y') 95. x.__getattr__('y') 96. x.__setattr__('y', z) 97. x.__delattr__('y') 98. x.__dir__() 99. x.__anext__() 100.x.__buffer__(flags) 101.x.__release_buffer__(m) 76. x.__mro_entries__([x]) 77. T.__class_getitem__(y) 78. type(base).__prepare__() 79. T.__instancecheck__(x) 80. T.__subclasscheck__(U) 81. x.__await__() 82. x.__aenter__() 83. x.__aexit__() 84. x.__aiter__() 85. x.__imod__(y) 86. x.__ifloordiv__(y) 87. x.__ipow__(y) 88. x.__imatmul__(y) 89. x.__iand__(y) 90. x.__ior__(y)
19. wei-lee.me Great Read by Trey Hunner
20. wei-lee.me
21. wei-lee.me __init__ Example in commitizen https://github.com/commitizen-tools/commitizen/pull/102/files
22. wei-lee.me __init__ 📕 How to use it?
23. wei-lee.me __init__ A method called whenever a class is initialed 🤔 What is it?
24. wei-lee.me __init__ Well known as the constructor of Python classes 🤔 What is it?
25. wei-lee.me
26. wei-lee.me __init__ Well known as the constructor of Python classes initializer 🤔 What is it?
27. wei-lee.me __new__ 🤔 What is it? The actual constructor of Python classes
28. wei-lee.me __new__ 📕 How to use it?
29. wei-lee.me __new__ 📕 How to use it?
30. wei-lee.me __new__ Example in Airflow https://github.com/apache/airflow/blob/35087d7d10714130cc3e9e9730e34b07fc56938d/ airflow/models/xcom_arg.py#L99-L102
31. wei-lee.me __new__ Example in Airflow - Why do we need it? https://github.com/apache/airflow/blob/35087d7d10714130cc3e9e9730e34b07fc56938d/ airflow/models/xcom_arg.py#L99-L102 XComArg should no longer be used. PlainXComArg should be used instead. But we want to keep XComArg backward compatibility.
32. wei-lee.me __new__ 🕑 When to use it? When you need to do operations before class creation
33. wei-lee.me __new__ 🕑 When to use it? - More specific Don't use it Use it only and only if you know what you're doing
34. wei-lee.me __new__ ⚠ Errors may occur and are harder to trace if not used carefully.
35. wei-lee.me __call__ 🤔 What is it? Called when the instance is called as a function
36. wei-lee.me __call__ 📕 How to use it?
37. wei-lee.me __call__ Example in commitizen https://github.com/commitizen-tools/commitizen/pull/102/files
38. wei-lee.me __call__ Example in commitizen - Why we designd it this way https://github.com/commitizen-tools/commitizen/blob/ ee41af27f88c9e718c2a3e18ed7c8c10ef290d25/commitizen/cli.py#L119
39. wei-lee.me __call__ Example in commitizen - Why we designd it this way https://github.com/commitizen-tools/commitizen/blob/ ee41af27f88c9e718c2a3e18ed7c8c10ef290d25/commitizen/cli.py#L119
40. wei-lee.me __call__ Example in commitizen - Why we designd it this way https://github.com/commitizen-tools/commitizen/blob/ ee41af27f88c9e718c2a3e18ed7c8c10ef290d25/commitizen/cli.py#L119
41. wei-lee.me __call__ 🕑 When to use it? When you want to call instances like a function
42. wei-lee.me __str__, __repr__ 🤔 What are they? String representation of objects
43. wei-lee.me __str__, __repr__ 🤔 What are they? __str__ __repr__ Called by str(obj), format(), print() repr() What it should be informal or nicely printable official look like a valid Python expression
44. wei-lee.me __str__, __repr__ 📕 How to use them?
45. wei-lee.me __str__, __repr__ 📕 How to use them?
46. wei-lee.me __str__ Example in Airflow https://github.com/apache/airflow/pull/39313/files
47. wei-lee.me __repr__ Example in Airflow https://github.com/apache/airflow/pull/41041/files
48. wei-lee.me __str__, __repr__ 🕑 When to use them? When you want to print your objects with useful info
49. wei-lee.me Operator Overloading * Dunder methods like __eq___, __and__, __or__, etc. * Called in operations like a == b, a and b, a or b, etc. 🤔 What are they?
50. wei-lee.me Operator Overloading 📕 How to use them?
51. wei-lee.me Operator Overloading 📕 How to use them?
52. wei-lee.me Operator Overloading Example in Airflow - __eq__ https://github.com/apache/airflow/pull/40478
53. wei-lee.me Operator Overloading Example in Airflow - __eq__ https://github.com/apache/airflow/pull/40478 It was later removed as "attr" already provides it
54. wei-lee.me Operator Overloading Example in Airflow - __and__ https://github.com/apache/airflow/pull/37101/files
55. wei-lee.me Operator Overloading Example in Airflow https://github.com/apache/airflow/pull/37101/files
56. wei-lee.me Operator Overloading 🕑 When to use it? When you want to leverage Python operators and make your object work more Pythonic
57. wei-lee.me PEP 562 Module __getattr__ and __dir__
58. wei-lee.me
59. wei-lee.me
60. wei-lee.me Before digging into PEP 562 let's start with what __getattr__ is
61. wei-lee.me __getattr__ 🤔 What is it? Called when unable to find an attribute
62. wei-lee.me __getattr__ 📕 How to use it? - without __getattr__
63. wei-lee.me __getattr__ 📕 How to use it? - with __getattr__
64. wei-lee.me __getattr__ 📕 How to use it? - with __getattr__
65. wei-lee.me Module __getattr__ 🤔 What is it? Called when a module is unable to find a specified attribute
66. wei-lee.me Module __getattr__ 📕 How to use it? - module __getattr__
67. wei-lee.me Module __getattr__ 📕 How to use it? - module __getattr__
68. wei-lee.me Module __getattr__ Example in Airflow - Deprecate global variable https://github.com/apache/airflow/pull/33189/files
69. wei-lee.me Module __getattr__ Example in Airflow - Lazy Import https://github.com/apache/airflow/blob/2.10.2rc1/airflow/__init__.py#L94
70. wei-lee.me Module __getattr__ 🕑 When to use it? When you want to handle attribute missing
71. wei-lee.me Module __getattr__ 🕑 When to use it? - More specific * Deprecate global variable * Lazy import
72. wei-lee.me try...except...
73. wei-lee.me try...except...else(?!)
74. wei-lee.me try...except...else... 🤔 What is it? If no exception occurs, execute the else block
75. wei-lee.me try...except...else... 📕 How to use it?
76. wei-lee.me try...except...else... Example in Airflow https://github.com/apache/airflow/pull/42408/files
77. wei-lee.me try...except...else... 🕑 When to use it? When you want to execute something when exceptions do not occur
78. wei-lee.me try...except...else... 🕑 When to use it? - More specific Move the logic that is not supposed to raise an exception out of the try block
79. wei-lee.me try...except...else... 🕑 When to use it? - More specific - Example (Before)
80. wei-lee.me try...except...else... 🕑 When to use it? - More specific - Example (After)
81. wei-lee.me And there's also a for...else...
82. wei-lee.me But we're not digging into that
83. wei-lee.me Decorators
84. wei-lee.me Decorator 🤔 What is it? A function that takes a function as an argument and wraps around that function
85. wei-lee.me Decorator 📕 How to use it?
86. wei-lee.me Decorator 📕 How to use it? - takes a function as an argument
87. wei-lee.me Decorator 📕 How to use it? - wraps around that function
88. wei-lee.me Decorator 📕 How to use it?
89. wei-lee.me Decorator 📕 How to use it? - Under the hook is just
90. wei-lee.me Decorator 📕 How to use it? - Just use functools.wraps
91. wei-lee.me Decorator Example in Airflow https://github.com/apache/airflow/pull/35367/files
92. wei-lee.me Decorator Example in Airflow - Duplicate Code Everywhere! https://github.com/apache/airflow/pull/35367/files
93. wei-lee.me Decorator Example in Airflow - Write Once https://github.com/apache/airflow/pull/35367/files
94. wei-lee.me Decorator Example in Airflow - Used Everywhere https://github.com/apache/airflow/pull/35367/files
95. wei-lee.me Decorator 🕑 When to use it? When you have common logic that needs to be applied to multiple functions
96. wei-lee.me Decorator All about decorators
97. wei-lee.me
98. wei-lee.me functools
99. wei-lee.me functools.partial 🤔 What is it? Used to freeze a portion of function arguments
100. wei-lee.me functools.partial 📕 How to use it?
101. wei-lee.me functools.partial 📕 How to use it?
102. wei-lee.me functools.partial Example in Airflow https://github.com/apache/airflow/pull/33189/files This should be true and has been fixed at https://github.com/apache/airflow/pull/35432.
103. wei-lee.me functools.partial Example in Airflow
104. wei-lee.me functools.partial 🕑 When to use it? When you have functions that looks almost the same except for a few arguments
105. wei-lee.me Iterator Protocol
106. wei-lee.me Iterator 🤔 What is it? https://docs.python.org/3/glossary.html#term-iterator
107. wei-lee.me Iterator 🤔 What is it? * Repeatedly calls to __next__() which returns successive items * Raise a StopIteration exception when there's no more data * Required __iter__() that returns the iterator object itself
108. wei-lee.me Iterator Protocol 🤔 What is it? https://docs.python.org/3/library/stdtypes.html#iterator-types
109. wei-lee.me Iterator Protocol 🤔 What is it? https://docs.python.org/3/library/stdtypes.html#iterator-types * __iter__() Return the iterator object itself * __next__() Return the next item from the iterator. Raise the StopIteration exception if there's no next item
110. wei-lee.me Iterator Protocol 📕 How to use it?
111. wei-lee.me Iterator Protocol 📕 How to use it?
112. wei-lee.me Iterator Protocol Example in Airflow https://github.com/apache/airflow/pull/36202/files
113. wei-lee.me Iterator Protocol Example in Airflow - Why do we need it? https://github.com/apache/airflow/pull/36202/files GCS blobs get new attributes when it's consumed
114. wei-lee.me Iterator Protocol 🕑 When to use it? When you want to make your class an iterator
115. wei-lee.me Takeaway Introducing some techniques you may or may not already know
116. wei-lee.me Takeaway Give minimum examples
117. wei-lee.me Takeaway Showing how they are used in real-world cases / open source
118. wei-lee.me Takeaway Hope to give you knowledge and confidence that
119. wei-lee.me
120. wei-lee.me
121. wei-lee.me $ cat weilee.py __name__ = 李唯 / Wei Lee __what_i_am_doing__ = [ First Time Speaker @ PyCon Japan, Volunteer @ PyCon Taiwan, Member @ PyCon APAC, Maintainer of commitizen-tools, Software Engineer @ Astronomer, Committer @ Apache Airflow, ] __github__ = Github Lee-W __linkedin__ = linkedin clleew __site__ = pen https://wei-lee.me
122. wei-lee.me File "weilee.py", line 1 __name__ = 李唯 / Wei Lee ^^^ SyntaxError: invalid syntax $ python weilee.py
123. PyCon Taiwan 2024 Kaohsiung, Sep 21-23
124. PyCon Taiwan Kaohsiung 21-23 SEP 2024
125. PyCon Taiwan Kaohsiung 21-23 SEP 2024 We had a wonderful time together 🎉
126. Foreign Participants Friends from Japan🇯🇵😉
127. PyCon Taiwan Kaohsiung 21-23 SEP 2024 NEXT YEAR……
128. PyCon Taiwan Kaohsiung 21-23 SEP 2024 台北 !!! Taipei !!!
129. PyCon Taiwan Kaohsiung 21-23 SEP 2024 SOCIAL MEDIA FACEBOOK INSTAGRAM X
130. wei-lee.me PyCon APAC 2024 Oct 25th - Oct 27th Indonesia
131. wei-lee.me PyCon APAC Booth
132. wei-lee.me PyCon JP 2024 Sprint! But I'll probably be a bit late. Hope I can arrive around 11:30. Sorry 😞
133. wei-lee.me It's the 133th page.
134. wei-lee.me I guess I also talk too much. Thanks James for the awesome keynote speech. and... this talk too much thing! 🙌
135. wei-lee.me