Tweet
Tweet

Slide 1

Slide 1 text

wei-lee.me ๐Ÿ” Unlocking Python's Core Magic

Slide 2

Slide 2 text

wei-lee.me QR Code links to the slide deck

Slide 3

Slide 3 text

wei-lee.me Do you know commitizen-tools?

Slide 4

Slide 4 text

wei-lee.me commitizen-tools Create committing rules for projects ๐Ÿš€ auto bump versions โฌ† and auto changelog generation ๐Ÿ“‚

Slide 5

Slide 5 text

wei-lee.me Like, subscribe, and contribute

Slide 6

Slide 6 text

wei-lee.me Do you know Apache Airflow?

Slide 7

Slide 7 text

wei-lee.me Apache Airflow Apache Airflowยฎ is an open-source platform for developing, scheduling, and monitoring batch-oriented workflows.

Slide 8

Slide 8 text

wei-lee.me Like, subscribe, and contribute

Slide 9

Slide 9 text

wei-lee.me

Slide 10

Slide 10 text

wei-lee.me Hey, I thought this was going to be a Python core-centric talk. Are we going to discuss libraries now?

Slide 11

Slide 11 text

wei-lee.me

Slide 12

Slide 12 text

wei-lee.me What is this talk about? * Python core magics and standard libraries * Real-world examples of how these techniques are used in those libraries

Slide 13

Slide 13 text

wei-lee.me

Slide 14

Slide 14 text

wei-lee.me This talk is highly inspired by

Slide 15

Slide 15 text

wei-lee.me And that's why I'll probably also talk too much. There are 135 pages.

Slide 16

Slide 16 text

wei-lee.me Dunder methods

Slide 17

Slide 17 text

wei-lee.me What dunder methods? โ€ข Methods that start and end with __ (e.g., __init__) โ€ข Dunder โ†’ Double Underscore โ€ข Allow us to customize classes in a more Pythonic way โ€ข Implicitly called by Python โ€ข Officially named as special methods

Slide 18

Slide 18 text

wei-lee.me List of dunder methods 1. x.__init__(a, b) 2. T.__new__(T, a, b) 3. x.__del__() 4. x.__eq__(y) 5. x.__ne__(y) 6. x.__lt__(y) 7. x.__rt__(y) 8. x.__le__(y) 9. x.__ge__(y) 10. x.__hash__() 11. x.__repr__() 12. x.__str__() 13. x.__bool__() 14. x.__int__() 15. x.__float__() 16. x.__bytes__() 17. x.__complex__() 18. x.__format__(s) 19. x.__enter__() 20. x.__exit__() 21. x.__len__() 22. x.__iter__() 23. x.__getitem__(a) 24. x.__setitem__(a, b) 25. x.__delitem__(a) 26. x.__contains__(a) 27. x.__reversed__() 28. x.__next__() 29. x.__missing__(a) 30. x.__length_hint__() 31. x.__add__(y) 32. y.__radd__(x) 33. x.__sub__(y) 34. y.__rsub__(x) 35. x.__mul__(y) 36. y.__rmul__(x) 37. x.__truediv__(y) 38. y.__rtruediv__(x) 39. x.__mod__(y) 40. y.__rmod__(x) 41. x.__floordiv__(y) 42. y.__rfloordiv__(x) 43. x.__pow__(y) 44. y.__rpow__(x) 45. x.__matmul__(y) 46. y.__rmatmul__(x) 47. x.__and__(y) 48. y.__rand__(x) 49. x.__or__(y) 50. y.__ror__(x) 51. x.__xor__(y) 52. y.__rxor__(x) 53. x.__rshift__(y) 54. y.__rrshift__(x) 55. x.__lshift__(y) 56. y.__rlshift__(x) 57. x.__neg__() 58. x.__pos__() 59. x.__invert__() 60. x.__divmod__(y) 61. x.__abs__() 62. x.__index__() 63. x.__round__() 64. x.__trunc__() 65. x.__floor__() 66. x.__ceil__() 67. x.__iadd__(y) 68. x.__isub__(y) 69. x.__imul__(y) 70. x.__itruediv__(y) 71. T.x.__set_name__(T, 'x') 72. T.x.__get__(t, T) 73. T.x.__set__(t, y) 74. T.x.__delete__(t) 75. T.__init_subclass__(U) 91. x.__ixor__(y) 92. x.__irshift__(y) 93. x.__ilshift__(y) 94. x.__getattribute__('y') 95. x.__getattr__('y') 96. x.__setattr__('y', z) 97. x.__delattr__('y') 98. x.__dir__() 99. x.__anext__() 100.x.__buffer__(flags) 101.x.__release_buffer__(m) 76. x.__mro_entries__([x]) 77. T.__class_getitem__(y) 78. type(base).__prepare__() 79. T.__instancecheck__(x) 80. T.__subclasscheck__(U) 81. x.__await__() 82. x.__aenter__() 83. x.__aexit__() 84. x.__aiter__() 85. x.__imod__(y) 86. x.__ifloordiv__(y) 87. x.__ipow__(y) 88. x.__imatmul__(y) 89. x.__iand__(y) 90. x.__ior__(y)

Slide 19

Slide 19 text

wei-lee.me Great Read by Trey Hunner

Slide 20

Slide 20 text

wei-lee.me

Slide 21

Slide 21 text

wei-lee.me __init__ Example in commitizen https://github.com/commitizen-tools/commitizen/pull/102/files

Slide 22

Slide 22 text

wei-lee.me __init__ ๐Ÿ“• How to use it?

Slide 23

Slide 23 text

wei-lee.me __init__ A method called whenever a class is initialed ๐Ÿค” What is it?

Slide 24

Slide 24 text

wei-lee.me __init__ Well known as the constructor of Python classes ๐Ÿค” What is it?

Slide 25

Slide 25 text

wei-lee.me

Slide 26

Slide 26 text

wei-lee.me __init__ Well known as the constructor of Python classes initializer ๐Ÿค” What is it?

Slide 27

Slide 27 text

wei-lee.me __new__ ๐Ÿค” What is it? The actual constructor of Python classes

Slide 28

Slide 28 text

wei-lee.me __new__ ๐Ÿ“• How to use it?

Slide 29

Slide 29 text

wei-lee.me __new__ ๐Ÿ“• How to use it?

Slide 30

Slide 30 text

wei-lee.me __new__ Example in Airflow https://github.com/apache/airflow/blob/35087d7d10714130cc3e9e9730e34b07fc56938d/ airflow/models/xcom_arg.py#L99-L102

Slide 31

Slide 31 text

wei-lee.me __new__ Example in Airflow - Why do we need it? https://github.com/apache/airflow/blob/35087d7d10714130cc3e9e9730e34b07fc56938d/ airflow/models/xcom_arg.py#L99-L102 XComArg should no longer be used. PlainXComArg should be used instead. But we want to keep XComArg backward compatibility.

Slide 32

Slide 32 text

wei-lee.me __new__ ๐Ÿ•‘ When to use it? When you need to do operations before class creation

Slide 33

Slide 33 text

wei-lee.me __new__ ๐Ÿ•‘ When to use it? - More specific Don't use it Use it only and only if you know what you're doing

Slide 34

Slide 34 text

wei-lee.me __new__ โš  Errors may occur and are harder to trace if not used carefully.

Slide 35

Slide 35 text

wei-lee.me __call__ ๐Ÿค” What is it? Called when the instance is called as a function

Slide 36

Slide 36 text

wei-lee.me __call__ ๐Ÿ“• How to use it?

Slide 37

Slide 37 text

wei-lee.me __call__ Example in commitizen https://github.com/commitizen-tools/commitizen/pull/102/files

Slide 38

Slide 38 text

wei-lee.me __call__ Example in commitizen - Why we designd it this way https://github.com/commitizen-tools/commitizen/blob/ ee41af27f88c9e718c2a3e18ed7c8c10ef290d25/commitizen/cli.py#L119

Slide 39

Slide 39 text

wei-lee.me __call__ Example in commitizen - Why we designd it this way https://github.com/commitizen-tools/commitizen/blob/ ee41af27f88c9e718c2a3e18ed7c8c10ef290d25/commitizen/cli.py#L119

Slide 40

Slide 40 text

wei-lee.me __call__ Example in commitizen - Why we designd it this way https://github.com/commitizen-tools/commitizen/blob/ ee41af27f88c9e718c2a3e18ed7c8c10ef290d25/commitizen/cli.py#L119

Slide 41

Slide 41 text

wei-lee.me __call__ ๐Ÿ•‘ When to use it? When you want to call instances like a function

Slide 42

Slide 42 text

wei-lee.me __str__, __repr__ ๐Ÿค” What are they? String representation of objects

Slide 43

Slide 43 text

wei-lee.me __str__, __repr__ ๐Ÿค” What are they? __str__ __repr__ Called by str(obj), format(), print() repr() What it should be informal or nicely printable official look like a valid Python expression

Slide 44

Slide 44 text

wei-lee.me __str__, __repr__ ๐Ÿ“• How to use them?

Slide 45

Slide 45 text

wei-lee.me __str__, __repr__ ๐Ÿ“• How to use them?

Slide 46

Slide 46 text

wei-lee.me __str__ Example in Airflow https://github.com/apache/airflow/pull/39313/files

Slide 47

Slide 47 text

wei-lee.me __repr__ Example in Airflow https://github.com/apache/airflow/pull/41041/files

Slide 48

Slide 48 text

wei-lee.me __str__, __repr__ ๐Ÿ•‘ When to use them? When you want to print your objects with useful info

Slide 49

Slide 49 text

wei-lee.me Operator Overloading * Dunder methods like __eq___, __and__, __or__, etc. * Called in operations like a == b, a and b, a or b, etc. ๐Ÿค” What are they?

Slide 50

Slide 50 text

wei-lee.me Operator Overloading ๐Ÿ“• How to use them?

Slide 51

Slide 51 text

wei-lee.me Operator Overloading ๐Ÿ“• How to use them?

Slide 52

Slide 52 text

wei-lee.me Operator Overloading Example in Airflow - __eq__ https://github.com/apache/airflow/pull/40478

Slide 53

Slide 53 text

wei-lee.me Operator Overloading Example in Airflow - __eq__ https://github.com/apache/airflow/pull/40478 It was later removed as "attr" already provides it

Slide 54

Slide 54 text

wei-lee.me Operator Overloading Example in Airflow - __and__ https://github.com/apache/airflow/pull/37101/files

Slide 55

Slide 55 text

wei-lee.me Operator Overloading Example in Airflow https://github.com/apache/airflow/pull/37101/files

Slide 56

Slide 56 text

wei-lee.me Operator Overloading ๐Ÿ•‘ When to use it? When you want to leverage Python operators and make your object work more Pythonic

Slide 57

Slide 57 text

wei-lee.me PEP 562 Module __getattr__ and __dir__

Slide 58

Slide 58 text

wei-lee.me

Slide 59

Slide 59 text

wei-lee.me

Slide 60

Slide 60 text

wei-lee.me Before digging into PEP 562 let's start with what __getattr__ is

Slide 61

Slide 61 text

wei-lee.me __getattr__ ๐Ÿค” What is it? Called when unable to find an attribute

Slide 62

Slide 62 text

wei-lee.me __getattr__ ๐Ÿ“• How to use it? - without __getattr__

Slide 63

Slide 63 text

wei-lee.me __getattr__ ๐Ÿ“• How to use it? - with __getattr__

Slide 64

Slide 64 text

wei-lee.me __getattr__ ๐Ÿ“• How to use it? - with __getattr__

Slide 65

Slide 65 text

wei-lee.me Module __getattr__ ๐Ÿค” What is it? Called when a module is unable to find a specified attribute

Slide 66

Slide 66 text

wei-lee.me Module __getattr__ ๐Ÿ“• How to use it? - module __getattr__

Slide 67

Slide 67 text

wei-lee.me Module __getattr__ ๐Ÿ“• How to use it? - module __getattr__

Slide 68

Slide 68 text

wei-lee.me Module __getattr__ Example in Airflow - Deprecate global variable https://github.com/apache/airflow/pull/33189/files

Slide 69

Slide 69 text

wei-lee.me Module __getattr__ Example in Airflow - Lazy Import https://github.com/apache/airflow/blob/2.10.2rc1/airflow/__init__.py#L94

Slide 70

Slide 70 text

wei-lee.me Module __getattr__ ๐Ÿ•‘ When to use it? When you want to handle attribute missing

Slide 71

Slide 71 text

wei-lee.me Module __getattr__ ๐Ÿ•‘ When to use it? - More specific * Deprecate global variable * Lazy import

Slide 72

Slide 72 text

wei-lee.me try...except...

Slide 73

Slide 73 text

wei-lee.me try...except...else(?!)

Slide 74

Slide 74 text

wei-lee.me try...except...else... ๐Ÿค” What is it? If no exception occurs, execute the else block

Slide 75

Slide 75 text

wei-lee.me try...except...else... ๐Ÿ“• How to use it?

Slide 76

Slide 76 text

wei-lee.me try...except...else... Example in Airflow https://github.com/apache/airflow/pull/42408/files

Slide 77

Slide 77 text

wei-lee.me try...except...else... ๐Ÿ•‘ When to use it? When you want to execute something when exceptions do not occur

Slide 78

Slide 78 text

wei-lee.me try...except...else... ๐Ÿ•‘ When to use it? - More specific Move the logic that is not supposed to raise an exception out of the try block

Slide 79

Slide 79 text

wei-lee.me try...except...else... ๐Ÿ•‘ When to use it? - More specific - Example (Before)

Slide 80

Slide 80 text

wei-lee.me try...except...else... ๐Ÿ•‘ When to use it? - More specific - Example (After)

Slide 81

Slide 81 text

wei-lee.me And there's also a for...else...

Slide 82

Slide 82 text

wei-lee.me But we're not digging into that

Slide 83

Slide 83 text

wei-lee.me Decorators

Slide 84

Slide 84 text

wei-lee.me Decorator ๐Ÿค” What is it? A function that takes a function as an argument and wraps around that function

Slide 85

Slide 85 text

wei-lee.me Decorator ๐Ÿ“• How to use it?

Slide 86

Slide 86 text

wei-lee.me Decorator ๐Ÿ“• How to use it? - takes a function as an argument

Slide 87

Slide 87 text

wei-lee.me Decorator ๐Ÿ“• How to use it? - wraps around that function

Slide 88

Slide 88 text

wei-lee.me Decorator ๐Ÿ“• How to use it?

Slide 89

Slide 89 text

wei-lee.me Decorator ๐Ÿ“• How to use it? - Under the hook is just

Slide 90

Slide 90 text

wei-lee.me Decorator ๐Ÿ“• How to use it? - Just use functools.wraps

Slide 91

Slide 91 text

wei-lee.me Decorator Example in Airflow https://github.com/apache/airflow/pull/35367/files

Slide 92

Slide 92 text

wei-lee.me Decorator Example in Airflow - Duplicate Code Everywhere! https://github.com/apache/airflow/pull/35367/files

Slide 93

Slide 93 text

wei-lee.me Decorator Example in Airflow - Write Once https://github.com/apache/airflow/pull/35367/files

Slide 94

Slide 94 text

wei-lee.me Decorator Example in Airflow - Used Everywhere https://github.com/apache/airflow/pull/35367/files

Slide 95

Slide 95 text

wei-lee.me Decorator ๐Ÿ•‘ When to use it? When you have common logic that needs to be applied to multiple functions

Slide 96

Slide 96 text

wei-lee.me Decorator All about decorators

Slide 97

Slide 97 text

wei-lee.me

Slide 98

Slide 98 text

wei-lee.me functools

Slide 99

Slide 99 text

wei-lee.me functools.partial ๐Ÿค” What is it? Used to freeze a portion of function arguments

Slide 100

Slide 100 text

wei-lee.me functools.partial ๐Ÿ“• How to use it?

Slide 101

Slide 101 text

wei-lee.me functools.partial ๐Ÿ“• How to use it?

Slide 102

Slide 102 text

wei-lee.me functools.partial Example in Airflow https://github.com/apache/airflow/pull/33189/files This should be true and has been fixed at https://github.com/apache/airflow/pull/35432.

Slide 103

Slide 103 text

wei-lee.me functools.partial Example in Airflow

Slide 104

Slide 104 text

wei-lee.me functools.partial ๐Ÿ•‘ When to use it? When you have functions that looks almost the same except for a few arguments

Slide 105

Slide 105 text

wei-lee.me Iterator Protocol

Slide 106

Slide 106 text

wei-lee.me Iterator ๐Ÿค” What is it? https://docs.python.org/3/glossary.html#term-iterator

Slide 107

Slide 107 text

wei-lee.me Iterator ๐Ÿค” What is it? * Repeatedly calls to __next__() which returns successive items * Raise a StopIteration exception when there's no more data * Required __iter__() that returns the iterator object itself

Slide 108

Slide 108 text

wei-lee.me Iterator Protocol ๐Ÿค” What is it? https://docs.python.org/3/library/stdtypes.html#iterator-types

Slide 109

Slide 109 text

wei-lee.me Iterator Protocol ๐Ÿค” What is it? https://docs.python.org/3/library/stdtypes.html#iterator-types * __iter__() Return the iterator object itself * __next__() Return the next item from the iterator. Raise the StopIteration exception if there's no next item

Slide 110

Slide 110 text

wei-lee.me Iterator Protocol ๐Ÿ“• How to use it?

Slide 111

Slide 111 text

wei-lee.me Iterator Protocol ๐Ÿ“• How to use it?

Slide 112

Slide 112 text

wei-lee.me Iterator Protocol Example in Airflow https://github.com/apache/airflow/pull/36202/files

Slide 113

Slide 113 text

wei-lee.me Iterator Protocol Example in Airflow - Why do we need it? https://github.com/apache/airflow/pull/36202/files GCS blobs get new attributes when it's consumed

Slide 114

Slide 114 text

wei-lee.me Iterator Protocol ๐Ÿ•‘ When to use it? When you want to make your class an iterator

Slide 115

Slide 115 text

wei-lee.me Takeaway Introducing some techniques you may or may not already know

Slide 116

Slide 116 text

wei-lee.me Takeaway Give minimum examples

Slide 117

Slide 117 text

wei-lee.me Takeaway Showing how they are used in real-world cases / open source

Slide 118

Slide 118 text

wei-lee.me Takeaway Hope to give you knowledge and confidence that

Slide 119

Slide 119 text

wei-lee.me

Slide 120

Slide 120 text

wei-lee.me

Slide 121

Slide 121 text

wei-lee.me $ cat weilee.py __name__ = ๆŽๅ”ฏ / Wei Lee __what_i_am_doing__ = [ First Time Speaker @ PyCon Japan, Volunteer @ PyCon Taiwan, Member @ PyCon APAC, Maintainer of commitizen-tools, Software Engineer @ Astronomer, Committer @ Apache Airflow, ] __github__ = Github Lee-W __linkedin__ = linkedin clleew __site__ = pen https://wei-lee.me

Slide 122

Slide 122 text

wei-lee.me File "weilee.py", line 1 __name__ = ๆŽๅ”ฏ / Wei Lee ^^^ SyntaxError: invalid syntax $ python weilee.py

Slide 123

Slide 123 text

PyCon Taiwan 2024 Kaohsiung, Sep 21-23

Slide 124

Slide 124 text

PyCon Taiwan Kaohsiung 21-23 SEP 2024

Slide 125

Slide 125 text

PyCon Taiwan Kaohsiung 21-23 SEP 2024 We had a wonderful time together ๐ŸŽ‰

Slide 126

Slide 126 text

Foreign Participants Friends from Japan๐Ÿ‡ฏ๐Ÿ‡ต๐Ÿ˜‰

Slide 127

Slide 127 text

PyCon Taiwan Kaohsiung 21-23 SEP 2024 NEXT YEARโ€ฆโ€ฆ

Slide 128

Slide 128 text

PyCon Taiwan Kaohsiung 21-23 SEP 2024 ๅฐๅŒ— !!! Taipei !!!

Slide 129

Slide 129 text

PyCon Taiwan Kaohsiung 21-23 SEP 2024 SOCIAL MEDIA FACEBOOK INSTAGRAM X

Slide 130

Slide 130 text

wei-lee.me PyCon APAC 2024 Oct 25th - Oct 27th Indonesia

Slide 131

Slide 131 text

wei-lee.me PyCon APAC Booth

Slide 132

Slide 132 text

wei-lee.me PyCon JP 2024 Sprint! But I'll probably be a bit late. Hope I can arrive around 11:30. Sorry ๐Ÿ˜ž

Slide 133

Slide 133 text

wei-lee.me It's the 133th page.

Slide 134

Slide 134 text

wei-lee.me I guess I also talk too much. Thanks James for the awesome keynote speech. and... this talk too much thing! ๐Ÿ™Œ

Slide 135

Slide 135 text

wei-lee.me