Unleash the Chaos - Developing a Linter for Un-Pythonic Code!

PyCon APAC 2025

Transcript

1. wei-lee.me Unleash the Chaos Developing a Linter for Un-Pythonic Code!
2. wei-lee.me QR Code to this slide deck
3. wei-lee.me CHAOS!!!!!!
4. wei-lee.me
5. wei-lee.me Well... It sounds more attracting than simply building a linter.
6. wei-lee.me The fact is... We're still going to start from building a standard linter.
7. wei-lee.me
8. wei-lee.me Sometimes, we require specific rules tailored to our needs.
9. wei-lee.me A real-world use case
10. wei-lee.me Back to the time...
11. wei-lee.me It required default values of argument deferrable in __init__ methods of any (operator) class to be conf.getboolean( "operators", "default_deferrable", fallback=False )
12. wei-lee.me ⛔
13. wei-lee.me ✅
14. wei-lee.me We're too lazy to do manual checks. So...
15. wei-lee.me Build our own linter 🔨
16. wei-lee.me We used the ast module
17. wei-lee.me What is ast anyway?
18. wei-lee.me A way to represent your code in a tree structure What is ast anyway?
19. wei-lee.me Minimum AST example
20. wei-lee.me Minimum AST example
21. wei-lee.me Minimum AST example
22. wei-lee.me Minimum AST example
23. wei-lee.me Minimum AST example
24. wei-lee.me Minimum AST example
25. wei-lee.me Minimum AST example
26. wei-lee.me Minimum AST example
27. wei-lee.me Minimum AST example
28. wei-lee.me Minimum AST example
29. wei-lee.me Minimum AST example
30. wei-lee.me Minimum AST example
31. wei-lee.me Go back to our example
32. wei-lee.me
33. wei-lee.me Well... It actually this huge even for such a simple case.
34. wei-lee.me What we want to check default values of argument deferrable in __init__ methods of any (operator) class to be conf.getboolean( "operators", "default_deferrable", fallback=False ) The requirement
35. wei-lee.me What we want to check 1. Search for class definitions 2. Search for __init__ methods 3. Search for arguments named as deferrable 4. Check the default value is conf.getboolean( "operators", "default_deferrable", fallback=False ) Formalize it into steps
36. wei-lee.me What we want to check 1. Search for ClassDef nodes 2. Search for FunctionDef nodes with name __init__ 3. Search for arguments nodes with name deferrable 4. Check the default values are conf.getboolean( "operators", "default_deferrable", fallback=False ) Formalize it into ast terms
37. wei-lee.me 1. Search for ClassDef node
38. wei-lee.me 1. Search for ClassDef node
39. wei-lee.me 1. Search for ClassDef node
40. wei-lee.me 2. Search for FunctionDef node with name __init__
41. wei-lee.me 2. Search for FunctionDef node with name __init__
42. wei-lee.me 3. Search for arguments named as deferrable
43. wei-lee.me 3. Search for arguments named as deferrable ast does not bind arguments and their defaults.
44. wei-lee.me 3. Search for arguments named as deferrable
45. wei-lee.me 3. Search for arguments named as deferrable ✅
46. wei-lee.me 3. Search for arguments named as deferrable ⛔
47. wei-lee.me 3. Search for arguments named as deferrable ❓
48. wei-lee.me 4. Check the default value
49. wei-lee.me 4. Check the default value
50. wei-lee.me 4. Check the default value All those comparisons in this step can be simplified as
51. wei-lee.me
52. wei-lee.me One step forward
53. wei-lee.me Auto-formatter!
54. wei-lee.me ast.NodeTransformer
55. wei-lee.me ast.NodeVisitor We can use it to simplify the linting logic.
56. wei-lee.me Build an auto-formatter Inherit from ast.NodeTransformer
57. wei-lee.me Build an auto-formatter visit_{Node type}
58. wei-lee.me Build an auto-formatter Logic after finding errors
59. wei-lee.me Build an auto-formatter Update "args.defaults" in the error node
60. wei-lee.me Build an auto-formatter Use DefaultDeferrableTransformer
61. wei-lee.me Build an auto-formatter Visit the ast tree
62. wei-lee.me Build an auto-formatter Fix line number due to node changes
63. wei-lee.me Build an auto-formatter Write the modified content back
64. wei-lee.me Let's see how this will be fixed!
65. wei-lee.me It fixed 🎉 But....
66. wei-lee.me All the formats are gone 😱
67. wei-lee.me Cheuk is right From PyCon APAC 2023
68. wei-lee.me
69. wei-lee.me
70. wei-lee.me Preserve More details
71. wei-lee.me Parsing the same module
72. wei-lee.me Syntax Tree from ast
73. wei-lee.me Syntax Tree from LibCST
74. wei-lee.me Syntax Tree from LibCST Visualized Version
75. wei-lee.me Syntax Tree from LibCST Let's zoom in a bit
76. wei-lee.me More features
77. wei-lee.me Metadata Providers
78. wei-lee.me Codemods
79. wei-lee.me Let's try this again CSTTransformer
80. wei-lee.me Let's try this again visit_{Node} / leave_{Node}
81. wei-lee.me Let's try this again a cleaner way to process parameters
82. wei-lee.me Let's try this again Compare syntax tree
83. wei-lee.me Let's try this again Update node
84. wei-lee.me Let's try this again It's now syntax tree visit transformer
85. wei-lee.me Everything looks good
86. wei-lee.me Except for it took 42.35 seconds to process 385 modules Btw, ast took 0.79. seconds (File loading took 0.49 seconds)
87. wei-lee.me What can we do? 1.Use ast to filter the modules that might contain an error 2.Use LibCST to fix those modules
88. wei-lee.me That's what I did in this PR
89. wei-lee.me How do other tools work?
90. wei-lee.me How black works?
91. wei-lee.me How black works? They're still building syntax trees. Just not using ast or LibCST
92. wei-lee.me How black works? 1. Sanitize lines 2. Parse the source code into a syntax tree (lib2to3_parse) 1. Get grammars in target versions 2. Tokenize 3. Return the syntax tree as Node object 3. Generate Line objects through LineGenerator 4. Transform line into formatted LineBlock In a super high level black == 24.8.0
93. wei-lee.me How black works? 1. Sanitize lines 2. Parse the source code into a syntax tree (lib2to3_parse) 1. Get grammars in target versions 2. Tokenize 3. Return the syntax tree as Node object 3. Generate Line objects through LineGenerator 4. Transform line into formatted LineBlock In a super high level black == 24.8.0
94. wei-lee.me How black works? 1. Sanitize lines 2. Parse the source code into a syntax tree (lib2to3_parse) 1. Get grammars in target versions 2. Tokenize 3. Return the syntax tree as Node object 3. Generate Line objects through LineGenerator 4. Transform line into formatted LineBlock In a super high level black == 24.8.0
95. wei-lee.me How black works? 1. Sanitize lines 2. Parse the source code into a syntax tree (lib2to3_parse) 1. Get grammars in target versions 2. Tokenize 3. Return the syntax tree as Node object 3. Generate Line objects through LineGenerator 4. Transform line into formatted LineBlock In a super high level black == 24.8.0
96. wei-lee.me How black works? 1. Sanitize lines 2. Parse the source code into a syntax tree (lib2to3_parse) 1. Get grammars in target versions 2. Tokenize 3. Return the syntax tree as Node object 3. Generate Line objects through LineGenerator 4. Transform line into formatted LineBlock In a super high level black == 24.8.0
97. wei-lee.me How black works? Syntax tree of "a = 1 + 2"
98. wei-lee.me How black works? Line object of "a = 1 + 2"
99. wei-lee.me How black is optimized?
100. wei-lee.me How about ruff?
101. wei-lee.me
102. wei-lee.me
103. wei-lee.me
104. wei-lee.me
105. wei-lee.me But if you take a deeper look
106. wei-lee.me
107. wei-lee.me It's still ast under the hook
108. wei-lee.me A minimum example
109. wei-lee.me Check if a deprecated name is used
110. wei-lee.me Get the qualified name (e.g., Dataset → airflow.datasets.Dataset)
111. wei-lee.me Check whether it's deprecated
112. wei-lee.me Check whether it's deprecated
113. wei-lee.me Push the error message
114. wei-lee.me That's it! (In a high level)
115. wei-lee.me Now you can also contribute to ruff with limited Rust knowledge 🙌
116. wei-lee.me
117. wei-lee.me H35?
118. wei-lee.me H1w a3t g5g s2e n8s?
119. wei-lee.me k8s i18n a11y l10n
120. wei-lee.me
121. wei-lee.me Setting up the rules 1.Skip builtins, dunder methods, and anything imported 2.Change all the top-level variable, class, and function names into numeronyms
122. wei-lee.me Nodes to exclude builtins, dunder methods
123. wei-lee.me Names to exclude import ... as ... / from ... import ... as ...
124. wei-lee.me Collect potential names ClassDef, FunctionDef, Name
125. wei-lee.me Collect potential names ClassDef, FunctionDef, Name
126. wei-lee.me Generate name mapping
127. wei-lee.me numeronymize
128. wei-lee.me NumeronymsTransformer
129. wei-lee.me 🙉
130. wei-lee.me Limitation (something I'm too lazy to implement) • Name collision • Scope • Inheritance
131. wei-lee.me It turns out that bringing chaos correctly is also tough.
132. wei-lee.me
133. wei-lee.me QR code links to my posts related to this talk
134. wei-lee.me Related PyCon Talks • Charlie Marsh - Ruff: An Extremely Fast Python Linter and Code Formatter, Written in Rust • Łukasz Langa - Life Is Better Painted Black, or: How to Stop Worrying and Embrace Auto-Formatting • Cheuk Ting Ho - Reformating your code without AI - let's see how a formatter works
135. wei-lee.me
136. wei-lee.me $ cat weilee.py __name__ = 李唯 / Wei Lee __what_i_am_doing__ = [ Volunteer @ PyCon Taiwan, First Time Speaker @ PyCon APAC, Member @ PyCon APAC, Maintainer of commitizen-tools, Software Engineer @ Astronomer, Committer @ Apache Airflow, ] __github__ = G Lee-W __linkedin__ = l clleew __site__ = p https://wei-lee.me
137. wei-lee.me File "weilee.py", line 1 __name__ = 李唯 / Wei Lee ^^^ SyntaxError: invalid syntax $ python weilee.py
138. wei-lee.me PyCon Taiwan 2025 6th-8th 2025 @ Taipei
139. wei-lee.me PyCon APAC 2025 Sprints 3rd March (Mon.)
140. wei-lee.me