Unleash the Chaos - Developing a Linter for Un-Pythonic Code!

Transcript

1. wei-lee.me Unleash the Chaos Developing a Linter for Un-Pythonic Code!
2. wei-lee.me QR Code to this slide deck
3. wei-lee.me CHAOS!!!!!!
4. wei-lee.me
5. wei-lee.me Well... It sounds more attracting than simply building a linter.
6. wei-lee.me The fact is... We're still going to build a normal linter
7. wei-lee.me The fact is... We're still going to build a normal linter at least for now.
8. wei-lee.me
9. wei-lee.me Well... Sometimes we require specific rules tailored to our needs.
10. wei-lee.me Such as?...
11. wei-lee.me A real-world use case
12. wei-lee.me Back to the time...
13. wei-lee.me It required default values of argument deferrable in __init__ methods of any (operator) class to be conf.getboolean( "operators", "default_deferrable", fallback=False )
14. wei-lee.me ⛔
15. wei-lee.me ✅
16. wei-lee.me What can we do?
17. wei-lee.me Kindly ask contributors to do the things right
18. wei-lee.me Build our own linter 🔨
19. wei-lee.me We used the ast module
20. wei-lee.me What is ast anyway?
21. wei-lee.me A way to represent your code in a tree structure What is ast anyway?
22. wei-lee.me Minimum AST example
23. wei-lee.me Minimum AST example
24. wei-lee.me Minimum AST example
25. wei-lee.me Minimum AST example
26. wei-lee.me Minimum AST example
27. wei-lee.me Minimum AST example
28. wei-lee.me Minimum AST example
29. wei-lee.me Minimum AST example
30. wei-lee.me Minimum AST example
31. wei-lee.me Minimum AST example
32. wei-lee.me Minimum AST example
33. wei-lee.me Minimum AST example
34. wei-lee.me Go back to our example
35. wei-lee.me wei-lee.me
36. wei-lee.me wei-lee.me Well... It actually this huge even for such a simple case.
37. wei-lee.me What we want to check default values of argument deferrable in __init__ methods of any (operator) class to be conf.getboolean( "operators", "default_deferrable", fallback=False )
38. wei-lee.me What we want to check 1. Search for class definitions 2. Search for __init__ methods 3. Search for arguments named as deferrable 4. Check the default value is conf.getboolean( "operators", "default_deferrable", fallback=False ) Formalize it into steps
39. wei-lee.me What we want to check 1. Search for ClassDef node 2. Search for FunctionDef node with name __init__ 3. Search for arguments node with name deferrable 4. Check the default value is conf.getboolean( "operators", "default_deferrable", fallback=False ) Formalize it into ast terms
40. wei-lee.me 1. Search for ClassDef node
41. wei-lee.me 1. Search for ClassDef node
42. wei-lee.me 1. Search for ClassDef node
43. wei-lee.me 2. Search for FunctionDef node with name __init__
44. wei-lee.me 2. Search for FunctionDef node with name __init__
45. wei-lee.me 3. Search for arguments named as deferrable
46. wei-lee.me 3. Search for arguments named as deferrable ast does not bind arguments and their defaults.
47. wei-lee.me 3. Search for arguments named as deferrable
48. wei-lee.me 3. Search for arguments named as deferrable ✅
49. wei-lee.me 3. Search for arguments named as deferrable ⛔
50. wei-lee.me 3. Search for arguments named as deferrable ❓
51. wei-lee.me 4. Check the default value
52. wei-lee.me 4. Check the default value
53. wei-lee.me 4. Check the default value All those comparisons in this step can be simplified as
54. wei-lee.me
55. wei-lee.me One step forward
56. wei-lee.me Auto-formatter!
57. wei-lee.me ast.NodeTransformer
58. wei-lee.me ast.NodeVisitor We can use it to simplify the linting logic.
59. wei-lee.me Build an auto-formatter Inherit from ast.NodeTransformer
60. wei-lee.me Build an auto-formatter visit_{Node type}
61. wei-lee.me Build an auto-formatter Logic after finding errors
62. wei-lee.me Build an auto-formatter Update "args.defaults" in the error node
63. wei-lee.me Build an auto-formatter Use DefaultDeferrableTransformer
64. wei-lee.me Build an auto-formatter Fix line number due to node changes
65. wei-lee.me Build an auto-formatter Write the modified content back
66. wei-lee.me Let's see how this will be fixed!
67. wei-lee.me It fixed 🎉 But....
68. wei-lee.me all the formats are gone 😱
69. wei-lee.me Cheuk is right From PyCon APAC 2023
70. wei-lee.me
71. wei-lee.me
72. wei-lee.me Preserve More details
73. wei-lee.me Parsing the same module
74. wei-lee.me Syntax Tree from ast
75. wei-lee.me Syntax Tree from LibCST
76. wei-lee.me Syntax Tree from LibCST Visualized Version
77. wei-lee.me Syntax Tree from LibCST Let's zoom in a bit
78. wei-lee.me More features
79. wei-lee.me Metadata Providers
80. wei-lee.me Codemods
81. wei-lee.me Let's try this again CSTTransformer
82. wei-lee.me Let's try this again visit_{Node} / leave_{Node}
83. wei-lee.me Let's try this again a cleaner way to process parameters
84. wei-lee.me Let's try this again compare syntax tree
85. wei-lee.me Let's try this again Update node
86. wei-lee.me Let's try this again It's now syntax tree visit transformer
87. wei-lee.me Everything looks good
88. wei-lee.me Except for it took 9 seconds to process 386 modules Btw, ast took 1.5 seconds
89. wei-lee.me What can we do? 1. Use ast to filter the modules that might contain an error 2. Use LibCST to fix those modules
90. wei-lee.me That's what I did in this PR
91. wei-lee.me How do other tools work?
92. wei-lee.me How black works?
93. wei-lee.me How black works? They're still building syntax trees. Just not using ast or LibCST
94. wei-lee.me How black works? 1. Sanitize lines 2. Parse the source code into a syntax tree (lib2to3_parse) 1. Get grammars in target versions 2. Tokenize 3. Return the syntax tree as Node object 3. Generate Line objects through LineGenerator 4. Transform line into formatted LineBlock In a super high level
95. wei-lee.me How black works? 1. Sanitize lines 2. Parse the source code into a syntax tree (lib2to3_parse) 1. Get grammars in target versions 2. Tokenize 3. Return the syntax tree as Node object 3. Generate Line objects through LineGenerator 4. Transform line into formatted LineBlock In a super high level
96. wei-lee.me How black works? 1. Sanitize lines 2. Parse the source code into a syntax tree (lib2to3_parse) 1. Get grammars in target versions 2. Tokenize 3. Return the syntax tree as Node object 3. Generate Line objects through LineGenerator 4. Transform line into formatted LineBlock In a super high level
97. wei-lee.me How black works? 1. Sanitize lines 2. Parse the source code into a syntax tree (lib2to3_parse) 1. Get grammars in target versions 2. Tokenize 3. Return the syntax tree as Node object 3. Generate Line objects through LineGenerator 4. Transform line into formatted LineBlock In a super high level
98. wei-lee.me How black works? 1. Sanitize lines 2. Parse the source code into a syntax tree (lib2to3_parse) 1. Get grammars in target versions 2. Tokenize 3. Return the syntax tree as Node object 3. Generate Line objects through LineGenerator 4. Transform line into formatted LineBlock In a super high level
99. wei-lee.me How black works? • \ Syntax tree of "a = 1 + 2"
100. wei-lee.me How black works? Line object of "a = 1 + 2"
101. wei-lee.me How black is optimized?
102. wei-lee.me How about ruff?
103. wei-lee.me Don't know much about Rust... Sorry.. But looks like they're doing something similar How about ruff?
104. wei-lee.me
105. wei-lee.me
106. wei-lee.me
107. wei-lee.me
108. wei-lee.me H35?
109. wei-lee.me H1w a3t g5g s2e n8s?
110. wei-lee.me k8s i18n a11y l10n
111. wei-lee.me
112. wei-lee.me Setting up the rules 1. Skip builtins, dunder methods, and anything imported 2. Change all the top-level variable, class, and function names into numeronyms
113. wei-lee.me Nodes to exclude builtins, dunder methods
114. wei-lee.me Names to exclude import ... as ... / from ... import ... as ...
115. wei-lee.me Collect potential names ClassDef, FunctionDef, Name
116. wei-lee.me Collect potential names ClassDef, FunctionDef, Name
117. wei-lee.me Generate name mapping
118. wei-lee.me numeronymize wei-lee.me
119. wei-lee.me NumeronymsTransformer
120. wei-lee.me 🙉
121. wei-lee.me Limitation • Name collision • Scope • Inheritance
122. wei-lee.me It turns out that bringing chaos correctly is also tough.
123. wei-lee.me
124. wei-lee.me QR code links to my posts related to this talk
125. wei-lee.me Related PyCon Talks • Charlie Marsh - Ruff: An Extremely Fast Python Linter and Code Formatter, Written in Rust • Łukasz Langa - Life Is Better Painted Black, or: How to Stop Worrying and Embrace Auto- Formatting • Cheuk Ting Ho - Reformating your code without AI - let's see how a formatter works
126. wei-lee.me
127. wei-lee.me $ cat weilee.py __name__ = 李唯 / Wei Lee __what_i_am_doing__ = [ Volunteer @ PyCon Taiwan, First Time Speaker @ PyCon Taiwan, Member @ PyCon APAC, Maintainer of commitizen-tools, Software Engineer @ Astronomer, Committer @ Apache Airflow, ] __github__ = G Lee-W __linkedin__ = l clleew __site__ = p http://wei-lee.me
128. wei-lee.me File "weilee.py", line 1 __name__ = 李唯 / Wei Lee ^^^ SyntaxError: invalid syntax $ python weilee.py
129. wei-lee.me PyCon TW 2024 Sprints 23rd September (Mon.)
130. wei-lee.me