SinnerSchrader Tech Session '18 - What the AST?

Two fundamental things that make the life of every developer easier are tokenizers and Abstract Syntax Trees. Whether it's your favorite syntax highlighting, linter, code formatting tool, or tools like the TypeScript compiler or Babel. All rely on these fundamental concepts. But how do they work?

In this session we'll look at what Tokenizer and Syntax parsers do, how an Abstract Syntax Tree looks like, why they are useful and other ways developers can benefit from them.

Transcript

1. AST? Dominik Kundel | @dkundel | #s2techsession | #whatTheAST What the Photo by Paweł Czerwiński on Unsplash
2. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST
3. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Have you used these tools?
4. AST? Dominik Kundel | @dkundel | #s2techsession | #whatTheAST What the Photo by Paweł Czerwiński on Unsplash
5. Dominik Kundel | @dkundel | #s2techsession | console.log(` Hi! I’m Dominik Kundel `); dkundel.com @dkundel dkundel@twilio.com github/dkundel Developer Evangelist && JavaScript Hacker #whatTheAST
6. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST OnesieJS
7. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST How do we find / modify code?
8. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Ctrl/Cmd + F
9. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST grep / sed
10. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST /RegularExpressions/gi
11. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Unreliable & Hard to maintain
12. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Let’s talk about those tools again Linters / Code Formatters Frameworks Bundlers Compilers / Transpilers
13. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Code > ✨ > Output Linters / Code Formatters Frameworks Bundlers Compilers / Transpilers
14. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST What’s that Magic? ✨
15. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Picture by @_lrlna
16. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Lexical Analysis Tokenization
17. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST console.log('Hello');
18. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST isPanda('');
19. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST const { tokenize } = require('esprima'); const tokens = tokenize("isPanda('')");
20. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST [ { type: 'Identifier', value: 'isPanda' }, { type: 'Punctuator', value: '(' }, { type: 'String', value: "''" }, { type: 'Punctuator', value: ')' }, ];
21. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Practical Examples Lexical Analysis Practical Examples
22. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Syntax Highlighting
23. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST https://code.visualstudio.com/blogs/2017/02/08/syntax-highlighting-optimizations
24. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Abstract Syntax Trees ASTs
25. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST [ { type: 'Identifier', value: 'isPanda' }, { type: 'Punctuator', value: '(' }, { type: 'String', value: "''" }, { type: 'Punctuator', value: ')' }, ];
26. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST { type: 'Program', body: [ { type: 'ExpressionStatement', expression: { type: 'CallExpression', callee: { type: 'Identifier', name: 'isPanda' }, arguments: [{ type: 'Literal', value: '', raw: "''" }], }, }, ], sourceType: 'script', }
27. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST https://github.com/estree/estree ESTree Spec
28. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Official ESTree specification ESTree Spec Alterations Esprima Syntax Tree Format Babel AST Format Facebook JSX Extension
29. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Practical Examples ASTs Practical Examples
30. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Linting
31. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Code Analysis angular.io/guide/language-service
32. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Bundling
33. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Modifying ASTs
34. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST { type: 'Program', body: [ { type: 'ExpressionStatement', expression: { type: 'CallExpression', callee: { type: 'Identifier', name: 'isPanda' }, arguments: [{ type: 'Literal', value: '', raw: "''" }], }, }, ], sourceType: 'script', }
35. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST { type: 'Program', body: [ { type: 'ExpressionStatement', expression: { type: 'Literal', value: true, raw: 'true' }, }, ], sourceType: 'script', }
36. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Practical Examples Modifying ASTs Practical Examples
37. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Code Modifications
38. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Language Extensions <Hello jsx={true} />
39. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Language Transpiling
40. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Code Coverage https://istanbul.js.org/
41. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST nyc instrument sum.js ./output
42. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST sum.js function sum(a, b) { return a + b; } module.exports = sum;
43. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST ./output/sum.js var cov_2mh38x6lsj = (function () { /* ... */ })(); function sum(a, b) { cov_2mh38x6lsj.f[0] ++; cov_2mh38x6lsj.s[0] ++; return a + b; } cov_2mh38x6lsj.s[1] ++; module.exports = sum;
44. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST var cov_2mh38x6lsj = (function() { var path = 'sum.js', hash = 'e40ca58a296221aa5a52fed6ee0f25cc89e6447b', Function = function() {}.constructor, global = new Function('return this')(), gcv = ' __coverage __', coverageData = { /* ... */ }, coverage = global[gcv] || (global[gcv] = {}); if (coverage[path] && coverage[path].hash === hash) { return coverage[path]; } coverageData.hash = hash; return (coverage[path] = coverageData); })();
45. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST coverageData = { path: 'sum.js', statementMap: { '0': { start: { line: 2, column: 2 }, end: { … } }, '1': { start: { line: 4, column: 0 }, end: { … } }, }, fnMap: { '0': { name: 'sum', decl: { start: { … }, end: { … } }, loc: { start: { … }, end: { … } }, line: 1, }, }, branchMap: {}, s: { '0': 0, '1': 0 }, f: { '0': 0 }, b: {}, _coverageSchema: 'd34fc3e6b8297bcde183f5492bcb8fcb36775295', },
46. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Understanding Code Coverage engineering.semantics3.com/understanding-code-coverage-1074e8fccce0 More Info
47. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Running other languages in the Browser
48. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST codecombat.com
49. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST esper.js.org
50. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Rendering ASTs Rendering ASTs
51. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST { type: 'Program', body: [ { type: 'ExpressionStatement', expression: { type: 'Literal', value: true, raw: 'true' }, }, ], sourceType: 'script', }
52. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST const escodegen = require('escodegen'); escodegen.generate({ type: 'Program', body: [ { type: 'ExpressionStatement', expression: { type: 'Literal', value: true, raw: 'true' }, }, ], sourceType: 'script', });
53. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST true;
54. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Practical Examples Rendering ASTs Rendering ASTs Practical Examples
55. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Code Formatting
56. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Minification
57. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Why would I care?
58. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Babel Transform Plugins
59. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST grep / sed
60. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST /RegularExpressions/gi
61. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST /RegularExpressions/gi ( )
62. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Building a Babel Plugin Live Demo
63. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Summary Photo by Jan Kahánek on Unsplash
64. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST 1. You use Tokens & ASTs daily
65. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST 2. Tokens represent format
66. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST 3. ASTs represent structure
67. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST 4. ASTs are “easy” to manipulate
68. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST 5. AST for safer code alterations
69. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST 6. ESTree for interoperability
70. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST 7. Don’t reinvent the wheel
71. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Libraries to create ASTs • @babel/parser (aka. babylon) • espree (ESLint’s parser) • typescript-estree (Used by ESLint & prettier) • flow-parser • esprima • uglify-js • acorn (the foundation of a lot of parsers)
72. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Libraries to walk ASTs • @babel/traverse • burrito, falafel • rocambole • babylon-walk (lightweight alternative to @babel/ traverse) • acorn-walk • jscodeshift
73. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Libraries to render ASTs • @babel/generator • escodegen • astring
74. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST 8. Replace grep & sed
75. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST astexplorer.net
76. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST git.io/babel-handbook
77. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Everything Open Source
78. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST hacktoberfest.digitalocean.com
79. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST www.twilio.com/quest
80. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Stickers!!
81. console.log(` Thank You! `); dkundel.com @dkundel dkundel@twilio.com github/dkundel d-k.im/s2-ast Dominik Kundel | @dkundel | #s2techsession | #whatTheAST