Hacking the Python AST

Talk at Pycon India 2016
CC BY-SA 4.0

Transcript

1. @jargnar github.com/jargnar Hacking the Python AST
2. 01010101010101010101010101...
3. A primer on compiling computer languages Front end Middle end Back end C JAVA ARM x86 IR IR
4. Intel’s developer manual
5. A primer on compiling computer languages Front end Middle end Back end C JAVA ARM x86 IR IR
6. @jargnar github.com/jargnar The Dragon Book
7. Grammars S –> AB A –> aA | ε B –> b | bB S AB S –> AB aAB A –> aA aaAB A –> aA aaaAB A –> aA aaaεB A –> ε aaab B –> b A parse of aaab Does aaab belong to this Grammar?
8. @jargnar github.com/jargnar LL(1) Grammar Left-to-right scan Leftmost derivation
9. @jargnar github.com/jargnar Let’s take a quick look at Python’s Grammar (Grammar/Grammar)
10. Design of the CPython Compiler Parser/ pgen.c Python/ ast.c Python/ compile.c Python/ compile.c Parse Tree .py file Byte code AST CFG
11. @jargnar github.com/jargnar dis
12. Design of the CPython Compiler Parser/ pgen.c Python/ ast.c Python/ compile.c Python/ compile.c Parse Tree .py file Byte code AST CFG
13. Python to x86 and then the 01010101s Parser/ pgen.c Python/ ast.c Python/ compile.c Python/ compile.c Parse Tree .py file Byte code AST CFG Python/ ceval.c
14. @jargnar github.com/jargnar
15. Accessing the AST a = 26 b = 0 print(a + b) example.py In [1]: import ast In [2]: source = open('example.py').read() In [3]: tree = ast.parse(source)
16. Accessing the AST a = 26 b = 0 print(a + b) example.py Module(body=[ Expr(value=Str(s='Docstring')), Assign( targets=[Name( id='a', ctx=Store())], value=Num(n=26)), Assign( targets=[Name( id='b', ctx=Store())], value=Num(n=0)), Expr(value=Call( func=Name( id='print', ctx=Load()), args=[BinOp( left=Name( id='a', ctx=Load()), op=Add(), right=Name( id='b', ctx=Load()))], keywords=[]))])
17. Having a little fun, now In [3]: tree = ast.parse('x = 5') In [4]: print(astunparse.dump(tree)) Module(body=[Assign( targets=[Name( id='x', ctx=Store())], value=Num(n=5))]) In [5]: tree.body[0].targets[0].id Out[5]: 'x' In [6]: astunparse.unparse(tree) x = 5 In [7]: tree.body[0].targets[0].id = 'y' In [8]: print(astunparse.unparse(tree)) y = 5
18. @jargnar github.com/jargnar Let’s do absolutely silly things with ast.NodeVisitor ast.NodeTransformer
19. @jargnar github.com/jargnar Even more silly things with Flake8
20. @jargnar github.com/jargnar Hacking is a mindset, not a skill. suhas.org