Tweet
Tweet

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

2 © 2019 Pinterest. All rights reserved. Joe Gordon, Site Reliability Engineer Syntax Trees and Python - Automated Code Transformations

Slide 3

Slide 3 text

3 © 2019 Pinterest. All rights reserved. Introduction Introduction 1 2 3 4

Slide 4

Slide 4 text

4 © 2019 Pinterest. All rights reserved. Our mission to create a life the inspiration everyone they love. To bring

Slide 5

Slide 5 text

5 © 2019 Pinterest. All rights reserved.

Slide 6

Slide 6 text

6 © 2019 Pinterest. All rights reserved. Python at Pinterest ● 250 Million monthly active users ● Used for every request ● Over 2.6 million lines along with 600,000 lines of comments

Slide 7

Slide 7 text

7 © 2019 Pinterest. All rights reserved. Python 3 Manually porting 2.6 million lines of Python 2 to support Python 3 is tedious and would take too long. Safe Transformation shouldn’t introduce new issues Quick Minimal developer time required to apply. Generalization Refactoring anything at scale has these requirements Problem Statement Refactoring at scale

Slide 8

Slide 8 text

8 © 2019 Pinterest. All rights reserved. Theory Theory 1 2 3 4

Slide 9

Slide 9 text

9 © 2019 Pinterest. All rights reserved. Automated Code Transformations ● Source → Syntax Tree → transform (on tree) → Source ● Apply a series of fixers to transform source code ● Safely automate tedious tasks

Slide 10

Slide 10 text

10 © 2019 Pinterest. All rights reserved. Automated Code Transformations Applications ● Applying style guides ● Porting code to Python 3 ● Refactoring code ○ Removing a dependency ○ Moving to a new API At Pinterest Porting code to Python 3

Slide 11

Slide 11 text

Automated Code Transformations ● Go ○ gofmt ○ go tool fix ● Javascript ○ https://babeljs.io ● C/C++/C# ○ clang-format ● ...

Slide 12

Slide 12 text

12 © 2019 Pinterest. All rights reserved. Work How They Automated Code Transformations

Slide 13

Slide 13 text

13 © 2019 Pinterest. All rights reserved. Regular Expressions

Slide 14

Slide 14 text

14 © 2019 Pinterest. All rights reserved. Regular Expressions ● Quick - for simple cases ● Unsafe ○ `__author__ = ‘bob’` ○ Comments ○ docstrings

Slide 15

Slide 15 text

15 © 2019 Pinterest. All rights reserved. Abstract Syntax Trees Rendered with show_ast

Slide 16

Slide 16 text

16 © 2019 Pinterest. All rights reserved. Abstract Syntax Trees Rendered with show_ast

Slide 17

Slide 17 text

17 © 2019 Pinterest. All rights reserved. Abstract Syntax Trees Linting

Slide 18

Slide 18 text

18 © 2019 Pinterest. All rights reserved. Abstract Syntax Trees Linting https://github.com/jparise/flake8-author/blob/master/flake8_author.py#L71

Slide 19

Slide 19 text

19 © 2019 Pinterest. All rights reserved. Syntax Trees ● Safer ● Multi line transformations ● Can get complex quickly Automated Code Transformation

Slide 20

Slide 20 text

20 © 2019 Pinterest. All rights reserved. Syntax Trees “This is a very concrete parse tree; we need to keep every token and even the comments and whitespace between tokens.” https://github.com/python/cpython/blob/c57e6e2e52d5d8b4005753bed789d99ebe407fb6/Lib/lib2to3/pytree.py Solves recreating original code from syntax tree Concrete vs Abstract

Slide 21

Slide 21 text

21 © 2019 Pinterest. All rights reserved. lib2to3 ● Concrete syntax tree ● Added in Python 2.6 ● Bundled with fixers for porting code to Python 3 ○ Example: Except X,T to except X as T ● Preserves formatting information ○ node.prefix ○ node.get_suffix() ● Track if node was changed >>> node Leaf(22, '=') >>> node.get_suffix() ' ' Syntax Trees ast ● Abstract syntax tree ● Added in Python 2.6 ● Good for static code analysis Python libraries

Slide 22

Slide 22 text

22 © 2019 Pinterest. All rights reserved. Tooling Tooling 1 2 3 4

Slide 23

Slide 23 text

23 © 2019 Pinterest. All rights reserved. Using lib2to3 ● Automated Python 2 to 3 code translation ● Concrete Syntax Tree ● Complex interface ● Powerful and safe ● Useful framework around fixers Reference: http://python3porting.com/fixers.html https://docs.python.org/2/library/2to3.html

Slide 24

Slide 24 text

24 © 2019 Pinterest. All rights reserved. Using lib2to3 https://github.com/python/cpython/blob/c57e6e2e52d5d8b4005753bed789d99ebe407fb6/Lib/lib2to3/fixes/fix_numliterals.py

Slide 25

Slide 25 text

25 © 2019 Pinterest. All rights reserved. Using lib2to3 https://github.com/python/cpython/blob/c57e6e2e52d5d8b4005753bed789d99ebe407fb6/Lib/lib2to3/fixes/fix_asserts.py

Slide 26

Slide 26 text

26 © 2019 Pinterest. All rights reserved. Using lib2to3 https://github.com/python/cpython/blob/c57e6e2e52d5d8b4005753bed789d99ebe407fb6/Lib/lib2to3/fixes/fix_throw.py

Slide 27

Slide 27 text

27 © 2019 Pinterest. All rights reserved. Using lib2to3 Input Output Fixer Runner

Slide 28

Slide 28 text

28 © 2019 Pinterest. All rights reserved. Lib2to3 based tools ● Python-future ● python-modernize ● Black ● Bowler

Slide 29

Slide 29 text

29 © 2019 Pinterest. All rights reserved. python-future ● Compatibility layer to concurrently support Py2 and Py3 ● Py3 idioms ● Uses lib2to3 python-future.org

Slide 30

Slide 30 text

30 © 2019 Pinterest. All rights reserved. python-modernize ● Converts Py2 code into a common subset of Py2 and Py3 ● Uses six and lib2to3 ● Futurize converts Py2 into (almost) standard Py3 code python-modernize.readthedocs.io

Slide 31

Slide 31 text

31 © 2019 Pinterest. All rights reserved. Modernize vs futurize modernize futurize

Slide 32

Slide 32 text

32 © 2019 Pinterest. All rights reserved. bowler ● Requires Py3.6 can be run against Py2 ● Lib2to3 based ● Simple to execute: bowler run ... pybowler.io

Slide 33

Slide 33 text

33 © 2019 Pinterest. All rights reserved. bowler pybowler.io

Slide 34

Slide 34 text

34 © 2019 Pinterest. All rights reserved. bowler

Slide 35

Slide 35 text

35 © 2019 Pinterest. All rights reserved. Bowler vs lib2to3

Slide 36

Slide 36 text

36 © 2019 Pinterest. All rights reserved. black ● Requires Py3.6 can be run against Py2 ● Lib2to3 based ● Validates CST transformation with AST black.readthedocs.io

Slide 37

Slide 37 text

37 © 2019 Pinterest. All rights reserved. Conclusion Conclusion 1 2 3 4

Slide 38

Slide 38 text

38 © 2019 Pinterest. All rights reserved. Conclusion ● Syntax trees make code transformations quick and safe ● Saved countless hours of tedious labor ● Complex edge cases are still complex

Slide 39

Slide 39 text

39 © 2019 Pinterest. All rights reserved.