AST in Wonderland

Presented at Full Stack Antwerp and PHP development Eindhoven.
---
We read and write code every day to build maintainable software. To help us in our effort, we use and rely on numerous tools. As we write software for increasingly complex problems, our tooling should also evolve to support this. Like ancient craftsmen, we should not only improve our skills but also the tools we use.

In his presentation 'AST in Wonderland', Jeroen will take you on an adventure, that started for him, with Abstract Syntax Trees. In his journey, he found new tools and techniques to help us in our code writing endeavor. He encountered mutants, lakes and even artificial life.

Join him and see how deep the rabbit hole goes...

Speaker: Jeroen Herczeg is a freelance developer with 13 years of experience. He is a fan of cyberpunk and loves to travel with his girlfriend and cats.

Transcript

1. Wonderland AST in
2. Hello AST Chapter One Chapter Two Deep Learning Chapter Three ML on Code
3. Hello AST Chapter One
6. https://github.com/nikic
7. https://github.com/nikic/PHP-Parser
9. <?php namespace Meetup\PHPLimburg; class Presentation { public function hello() { return 'Hello!'; } }
10. Lexical analysis
11. <?php namespace Meetup\PHPLimburg; class Presentation { public function hello() { return 'Hello!'; } } T_OPEN_TAG T_WHITESPACE T_NAMESPACE T_WHITESPACE T_CLASS T_PUBLIC T_RETURN T_STRING T_STRING T_STRING T_FUNCTION T_STRING T_CONSTANT_ENCAPSED_STRING
12. <?php namespace Meetup\function; class Namespace { public function **() { return 'return'; } } T_OPEN_TAG T_WHITESPACE T_NAMESPACE T_WHITESPACE T_CLASS T_PUBLIC T_RETURN T_STRING T_FUNCTION T_NAMESPACE T_FUNCTION T_POW T_CONSTANT_ENCAPSED_STRING
13. ext-tokenizer
14. <?php namespace Meetup\PHPLimburg; class Presentation { public function hello() { return 'Hello!'; } } T_OPEN_TAG T_WHITESPACE T_NAMESPACE T_WHITESPACE T_CLASS T_PUBLIC T_RETURN T_STRING T_STRING T_STRING T_FUNCTION T_STRING T_CONSTANT_ENCAPSED_STRING
15. Syntax analysis
16. PHP Parse error: syntax error, unexpected ';' in /Users/jeroenherczeg/Code/ast/Presentation.php on line 7
17. <?php namespace Meetup\PHPLimburg; class Presentation { public function hello() { return 'Hello!'; } } T_OPEN_TAG T_WHITESPACE T_NAMESPACE T_WHITESPACE T_CLASS T_PUBLIC T_RETURN T_STRING T_STRING T_STRING T_FUNCTION T_STRING T_CONSTANT_ENCAPSED_STRING
19. Lexer Parser Compiler PHP Code Token stream Opcodes
20. Lexer Parser Compiler PHP Code Token stream Opcodes
22. <?php namespace Meetup\PHPLimburg; class Presentation { public function hello() { return 'Hello!'; } } T_OPEN_TAG T_WHITESPACE T_NAMESPACE T_WHITESPACE T_CLASS T_PUBLIC T_RETURN T_STRING T_STRING T_STRING T_FUNCTION T_STRING T_CONSTANT_ENCAPSED_STRING
24. AST_NAMESPACE AST_CLASS AST_METHOD AST_RETURN
26. AST_VAR AST_CLASS AST_METHOD AST_ASSIGN AST_CALL AST_IF AST_RETURN AST_RETURN
27. Lexer PHP Code Token stream Opcodes Parser Compiler AST
28. ext-ast https://github.com/nikic/php-ast
29. https://github.com/nikic/PHP-Parser
30. Parsing PHP code into an AST
31. Converting an AST back to PHP code
32. Traverse an AST
33. Modify an AST
34. AST construction using builders
35. Coding Standard Tool
36. AST_BINARY_OP AST_BINARY_OP 3 1 2 AST_ECHO
37. AST_BINARY_OP AST_BINARY_OP 3 1 2 AST_ECHO
38. Static code analyser
39. • PHPStan https://github.com/phpstan/phpstan • Psalm https://github.com/vimeo/psalm • Phan https://github.com/phan/phan • Exakat https://github.com/exakat/exakat
40. • Finding probable bugs • Locating “dead” code • Detecting performance issues • Improving code structure and maintainability
41. Code Analysis Rules Report
42. Exakat - 9.572. Useless Return A return value inside a constructor is inaccessible. Codebase Codebase Codebase AST Functions/UselessReturn src/Presentation.php:7 Functions/UselessReturn src/Presentation.php:7
43. Call graphs
44. Control flow graphs
45. Control flow graphs
46. Mutation testing
50. https://infection.github.io/guide/
57. Refactoring
58. PHP Supported Versions
59. Rector is a reconstructor tool. It does instant upgrades and instant refactoring of your code. Rector
61. https://laravelshift.com
63. Hello AST Chapter One • AST is a tree representation of source code • Why it was introduced in PHP7 in the internals • How we can use AST in userland (PHP-Parser) • Example of cool dev tools using AST
64. Chapter Two Deep Learning
65. Three waves of A.I. Handcrafted knowledge Statistical learning Contextual adaptation
67. How long does an order take?
68. Chef #1 Chef #2 Salade 1 minute 5 minutes Rice bowl 5 minutes 10 minutes Sushi 5 minutes 20 minutes
72. Chef #1 Chef #2 Salade Rice bowl Sushi 1 minute 5 minutes 10 minutes 15 minutes 20 minutes How long does an order take?
73. Chef #1 Chef #2 Salade Rice bowl Sushi 1 minute 5 minutes 10 minutes 15 minutes 20 minutes
74. INPUTS OUTPUTS HIDDEN LAYER
75. INPUTS OUTPUTS HIDDEN LAYER Weight Weight Weight Weight Bias Bias Weight Weight Weight Weight Bias Bias
76. INPUTS OUTPUTS HIDDEN LAYER 0.4 0.32 0.2 0.5 3 1 0.66 0.3 0.3 0.6 1 2
77. INPUTS OUTPUTS HIDDEN LAYER 0.4 0.32 0.2 0.5 3 1 0.66 0.3 0.3 0.6 1 1 0 (1 x 0.4) + (0 x 0.5) + 3 = 3.4 2
78. INPUTS OUTPUTS HIDDEN LAYER 0.4 0.32 0.2 0.5 3 1 0.66 0.3 0.3 0.6 1 1 0 3.4 0.9 Activation function 0.9 2
79. Chef #1 Chef #2 Salade Rice bowl Sushi 1 minute 5 minutes 10 minutes 15 minutes 20 minutes 1 1 0 0 0
80. Chef #1 Chef #2 Salade Rice bowl Sushi 1 minute 5 minutes 10 minutes 15 minutes 20 minutes 1 1 0 0 0
81. Chef #1 Chef #2 Salade Rice bowl Sushi 1 minute 5 minutes 10 minutes 15 minutes 20 minutes 1 1 0 0 0 0.05 0.60 0.10 0.05 0.20
82. Chef #1 Chef #2 Salade Rice bowl Sushi 1 minute 5 minutes 10 minutes 15 minutes 20 minutes 1 1 0 0 0 0.05 0.60 0.10 0.05 0.20
83. Chef #1 Chef #2 Salade Rice bowl Sushi 1 minute 5 minutes 10 minutes 15 minutes 20 minutes 0 0 1 1 0
84. Chef #1 Chef #2 Salade Rice bowl Sushi 1 minute 5 minutes 10 minutes 15 minutes 20 minutes 0 0 1 1 0
85. Chef #1 Chef #2 Salade Rice bowl Sushi 1 minute 5 minutes 10 minutes 15 minutes 20 minutes 0 0 1 1 0 0.34 0.20 0.16 0.03 0.27
86. Chef #1 Chef #2 Salade Rice bowl Sushi 1 minute 5 minutes 10 minutes 15 minutes 20 minutes 0.34 0.20 0.16 0.03 0.27 0 0 1 1 0
88. Vision Natural language processing (NLP) • Translation • Summarisation • Sentiment analysis • Dialogue • Classification • Object detection • Facial recognition • Generation Speech Robotics Planning
89. Deep Learning Chapter Two • Where we are with A.I. • Deep learning isn’t magic, it’s statistics! • What problems we can solve with ML. • Very exciting times!
90. Chapter Three ML on Code
91. In this project, you'll generate your own Simpsons TV scripts using RNNs. You'll be using part of the Simpsons dataset of scripts from 27 seasons. The neural network you'll build will generate a new TV script.
96. 1. Request list of package names 2. Loop through list and request package information 3. Clone package using repository url
97. 3 hours later …
100. 207k packages
101. 590GB Size
102. 7.4M PHP files
103. 21M commits
104. 103 Hours
110. https://ml4code.github.io
111. Assisted Naming https://arxiv.org/pdf/1602.03001.pdf
112. Code Recommendation https://arxiv.org/pdf/1812.01158.pdf
113. Assisted Code Review
114. ML on Code Chapter Three • We can use NLP also for programming languages • Programming languages have great features • They can help to improve the developer tools
115. Thank you 
 Jeroen Herczeg @jeroenherczeg