AST in Wonderland

We read and write code every day to build maintainable software. To help us in our effort, we use and rely on numerous tools. As we write software for increasingly complex problems, our tooling should also evolve to support this. Like ancient craftsmen, we should not only improve our skills but also the tools we use.

In his presentation "AST in Wonderland", Jeroen will take you on an adventure, that started for him, with Abstract Syntax Trees. In his journey, he found new tools and techniques to help us in our code writing endeavor. He encountered mutants, lakes and even artificial life.
Join him and see how deep the rabbit hole goes...

Transcript

1. Wonderland AST in
2. Jeroen Herczeg @jeroenherczeg jeroen@herczeg.be
3. Generalist A person who knows a lot about everything, but knows nothing about something.
4. Wonderland AST in
5. Nikita Chapter One
6. What’s new in PHP7 • Scalar type hints • Return type declarations • Anonymous classes • Generator delegation • Generator return expressions • The null coalesce operator • The space ship operator • Throwables • Abstract syntax tree • The Integer division function • Uniform variable syntax
7. What’s new in PHP7 • Scalar type hints • Return type declarations • Anonymous classes • Generator delegation • Generator return expressions • The null coalesce operator • The space ship operator • Throwables • Abstract syntax tree • The Integer division function • Uniform variable syntax
8. https://wiki.php.net/rfc/abstract_syntax_tree
9. https://wiki.php.net/rfc/abstract_syntax_tree
10. https://wiki.php.net/rfc/abstract_syntax_tree
11. https://wiki.php.net/rfc/abstract_syntax_tree
12. https://github.com/nikic
13. https://github.com/nikic/FastRoute
14. https://github.com/nikic
15. https://github.com/nikic/PHP-Parser
16. Hello AST Chapter Two
17. https://wiki.php.net/rfc/abstract_syntax_tree
18. <?php namespace Meetup\PHPLimburg; class Presentation { public function hello() { return 'Hello!'; } }
19. Lexical analysis
20. Lexical analysis, lexing or tokenization is the process of converting a sequence of characters into a sequence of tokens. A program that performs lexical analysis is called a lexer, tokenizer, or scanner.
21. <?php namespace Meetup\PHPLimburg; class Presentation { public function hello() { return 'Hello!'; } } T_OPEN_TAG T_WHITESPACE T_NAMESPACE T_WHITESPACE T_CLASS T_PUBLIC T_RETURN T_STRING T_STRING T_STRING T_FUNCTION T_STRING T_CONSTANT_ENCAPSED_STRING
22. <?php namespace Meetup\function; class Namespace { public function **() { return 'return'; } } T_OPEN_TAG T_WHITESPACE T_NAMESPACE T_WHITESPACE T_CLASS T_PUBLIC T_RETURN T_STRING T_FUNCTION T_NAMESPACE T_FUNCTION T_POW T_CONSTANT_ENCAPSED_STRING
23. https://github.com/php/php-src/blob/PHP-5.6.40/Zend/zend_language_scanner.l https://github.com/php/php-src/blob/PHP-5.6.40/Zend/zend_language_scanner.c Lexer
24. ext-tokenizer
27. <?php namespace Meetup\PHPLimburg; class Presentation { public function hello() { return 'Hello!'; } } T_OPEN_TAG T_WHITESPACE T_NAMESPACE T_WHITESPACE T_CLASS T_PUBLIC T_RETURN T_STRING T_STRING T_STRING T_FUNCTION T_STRING T_CONSTANT_ENCAPSED_STRING
28. Syntax analysis
29. Parsing, syntax analysis, or syntactic analysis is the process of analysing tokens conforming to the rules of the grammar.
30. https://github.com/php/php-src/blob/PHP-5.6.40/Zend/zend_language_parser.y Parser
31. https://www.youtube.com/watch?v=eF9qWbuQLuw - Bisqwit: Parser and Lexer PHP Parse error: syntax error, unexpected ';' in /Users/jeroenherczeg/Code/ast/Presentation.php on line 7
32. <?php namespace Meetup\PHPLimburg; class Presentation { public function hello() { return 'Hello!'; } } T_OPEN_TAG T_WHITESPACE T_NAMESPACE T_WHITESPACE T_CLASS T_PUBLIC T_RETURN T_STRING T_STRING T_STRING T_FUNCTION T_STRING T_CONSTANT_ENCAPSED_STRING
34. Lexer Parser Compiler PHP Code Token stream Opcodes
35. Lexer Parser Compiler PHP Code Token stream Opcodes
37. <?php namespace Meetup\PHPLimburg; class Presentation { public function hello() { return 'Hello!'; } } T_OPEN_TAG T_WHITESPACE T_NAMESPACE T_WHITESPACE T_CLASS T_PUBLIC T_RETURN T_STRING T_STRING T_STRING T_FUNCTION T_STRING T_CONSTANT_ENCAPSED_STRING
39. AST_NAMESPACE AST_CLASS AST_METHOD AST_RETURN
41. AST_VAR AST_CLASS AST_METHOD AST_ASSIGN AST_CALL AST_IF AST_RETURN AST_RETURN
42. Lexer PHP Code Token stream Opcodes Parser Compiler AST
43. ext-ast https://github.com/nikic/php-ast
44. ext-ast PHP-Parser Tolerant PHP Parser Extension Composer Composer Only version of PHP installed PHP >= 7.0 PHP5 - PHP7 PHP5 - PHP7 Fastest (C) Slower (PHP) Slower (PHP) Syntactically valid Partial AST Tolerant (IDE) https://github.com/nikic/php-ast https://github.com/nikic/PHP-Parser https://github.com/Microsoft/tolerant- php-parser
45. Practice in userland Chapter Three
46. https://github.com/nikic/PHP-Parser
47. Parsing PHP code into an AST
48. Parsing invalid code into an AST
49. Converting an AST back to PHP code
50. Traverse an AST
51. Modify an AST
52. AST construction using builders
53. AST in userland Chapter Four
54. Coding Standard Tool
55. Coding standard tools use tokens.
56. AST_BINARY_OP AST_BINARY_OP 3 1 2 AST_ECHO
57. AST_BINARY_OP AST_BINARY_OP 3 1 2 AST_ECHO
58. Static code analyser
59. Static code analysis is the analysis of source code that is performed without actually executing it.
60. • PHPStan https://github.com/phpstan/phpstan • Psalm https://github.com/vimeo/psalm • Phan https://github.com/phan/phan • Exakat https://github.com/exakat/exakat
61. • Finding probable bugs • Locating “dead” code • Detecting performance issues • Improving code structure and maintainability
62. Code Analysis Rules Report
63. Exakat - 9.572. Useless Return A return value inside a constructor is inaccessible. Codebase Codebase Codebase AST Functions/UselessReturn src/Presentation.php:7 Functions/UselessReturn src/Presentation.php:7
64. Exakat - 9.518. Unused Functions A function is never called. Codebase Codebase Codebase AST ??
65. Call graph
66. Control flow graph
67. Code intelligence
69. Language Server Protocol The Language Server Protocol (LSP) defines the protocol
 used between an editor or IDE and a language server
 that provides language features like auto complete, go to definition, find all references etc. https://microsoft.github.io/language-server-protocol
70. Mutation testing
71. Mutation testing is used to measure the effectiveness of a test set in terms of its ability to detect faults. It involves modifying a program in small ways. Each mutated version is called a Mutant. If mutated program produces failing tests, this is called a killed mutant. If tests are green with mutated code, then we have an escaped mutant.
75. https://infection.github.io/guide/
82. Transpiler
83. A transpiler takes the source code of a program written in a programming language as its input and produces the equivalent source code in the same or a different programming language.
85. Refactoring
86. Rector is a reconstructor tool. It does instant upgrades and instant refactoring of your code. Rector
88. https://laravelshift.com
90. Documentation Generation
91. ApiGen/ApiGen
92. zircote/swagger-php
94. Code Metrics
95. sebastianbergmann/phploc
96. phpmetrics/PhpMetrics
97. exakat inventories
98. bmitch/churn-php
100. Study the past Chapter Five
102. knowledge loss
104. Language Chapter Six
106. Vision Natural language processing (NLP) • Translation • Summarisation • Sentiment analysis • Dialogue • … • Classification • Object detection • Facial recognition • Generation • …
107. In this project, you'll generate your own Simpsons TV scripts using RNNs. You'll be using part of the Simpsons dataset of scripts from 27 seasons. The neural network you'll build will generate a new TV script.
108. https://github.com/jeroenherczeg/tv-script-generation
110. Like writing and speaking, software development is an act of human communication.
111. Machine Learning on Source Code This new line of research is inherently interdisciplinary, uniting the machine learning and natural language processing communities with software engineering and programming language communities.
112. https://ml4code.github.io
113. Code Summarisation https://arxiv.org/pdf/1602.03001.pdf
114. Code Recommendation https://arxiv.org/pdf/1812.01158.pdf
115. Type Inference http://vhellendoorn.github.io/PDF/fse2018-j2t.pdf
116. Vulnerability Detection https://arxiv.org/pdf/1807.04320.pdf
118. Data lake Chapter Seven
122. 1. Request list of package names 2. Loop through list and request package information 3. Clone package using repository url
123. 3 hours later …
126. 207k packages
127. 590GB Size
128. 7.4M PHP files
129. 21M commits
130. 103 Hours
131. Start exploring Chapter Eight
132. https://php-ml.org
133. Nikita Popov @nikita_ppv Tomáš Votruba @VotrubaT Arkadiusz Kondas @ArkadiuszKondas
134. Thank you