Upgrade to Pro — share decks privately, control downloads, hide ads and more …

SinnerSchrader Tech Session '18 - What the AST?

SinnerSchrader Tech Session '18 - What the AST?

Two fundamental things that make the life of every developer easier are tokenizers and Abstract Syntax Trees. Whether it's your favorite syntax highlighting, linter, code formatting tool, or tools like the TypeScript compiler or Babel. All rely on these fundamental concepts. But how do they work?

In this session we'll look at what Tokenizer and Syntax parsers do, how an Abstract Syntax Tree looks like, why they are useful and other ways developers can benefit from them.

Dominik Kundel

October 11, 2018
Tweet

More Decks by Dominik Kundel

Other Decks in Programming

Transcript

  1. AST? Dominik Kundel | @dkundel | #s2techsession | #whatTheAST What

    the Photo by Paweł Czerwiński on Unsplash
  2. AST? Dominik Kundel | @dkundel | #s2techsession | #whatTheAST What

    the Photo by Paweł Czerwiński on Unsplash
  3. Dominik Kundel | @dkundel | #s2techsession | console.log(` Hi! I’m

    Dominik Kundel `); dkundel.com @dkundel [email protected] github/dkundel Developer Evangelist && JavaScript Hacker #whatTheAST
  4. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Let’s talk

    about those tools again Linters / Code Formatters Frameworks Bundlers Compilers / Transpilers
  5. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Code >

    ✨ > Output Linters / Code Formatters Frameworks Bundlers Compilers / Transpilers
  6. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST const {

    tokenize } = require('esprima'); const tokens = tokenize("isPanda('')");
  7. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST [ {

    type: 'Identifier', value: 'isPanda' }, { type: 'Punctuator', value: '(' }, { type: 'String', value: "''" }, { type: 'Punctuator', value: ')' }, ];
  8. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST [ {

    type: 'Identifier', value: 'isPanda' }, { type: 'Punctuator', value: '(' }, { type: 'String', value: "''" }, { type: 'Punctuator', value: ')' }, ];
  9. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST { type:

    'Program', body: [ { type: 'ExpressionStatement', expression: { type: 'CallExpression', callee: { type: 'Identifier', name: 'isPanda' }, arguments: [{ type: 'Literal', value: '', raw: "''" }], }, }, ], sourceType: 'script', }
  10. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Official ESTree

    specification ESTree Spec Alterations Esprima Syntax Tree Format Babel AST Format Facebook JSX Extension
  11. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST { type:

    'Program', body: [ { type: 'ExpressionStatement', expression: { type: 'CallExpression', callee: { type: 'Identifier', name: 'isPanda' }, arguments: [{ type: 'Literal', value: '', raw: "''" }], }, }, ], sourceType: 'script', }
  12. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST { type:

    'Program', body: [ { type: 'ExpressionStatement', expression: { type: 'Literal', value: true, raw: 'true' }, }, ], sourceType: 'script', }
  13. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST sum.js function

    sum(a, b) { return a + b; } module.exports = sum;
  14. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST ./output/sum.js var

    cov_2mh38x6lsj = (function () { /* ... */ })(); function sum(a, b) { cov_2mh38x6lsj.f[0] ++; cov_2mh38x6lsj.s[0] ++; return a + b; } cov_2mh38x6lsj.s[1] ++; module.exports = sum;
  15. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST var cov_2mh38x6lsj

    = (function() { var path = 'sum.js', hash = 'e40ca58a296221aa5a52fed6ee0f25cc89e6447b', Function = function() {}.constructor, global = new Function('return this')(), gcv = ' __coverage __', coverageData = { /* ... */ }, coverage = global[gcv] || (global[gcv] = {}); if (coverage[path] && coverage[path].hash === hash) { return coverage[path]; } coverageData.hash = hash; return (coverage[path] = coverageData); })();
  16. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST coverageData =

    { path: 'sum.js', statementMap: { '0': { start: { line: 2, column: 2 }, end: { … } }, '1': { start: { line: 4, column: 0 }, end: { … } }, }, fnMap: { '0': { name: 'sum', decl: { start: { … }, end: { … } }, loc: { start: { … }, end: { … } }, line: 1, }, }, branchMap: {}, s: { '0': 0, '1': 0 }, f: { '0': 0 }, b: {}, _coverageSchema: 'd34fc3e6b8297bcde183f5492bcb8fcb36775295', },
  17. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Understanding Code

    Coverage engineering.semantics3.com/understanding-code-coverage-1074e8fccce0 More Info
  18. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST { type:

    'Program', body: [ { type: 'ExpressionStatement', expression: { type: 'Literal', value: true, raw: 'true' }, }, ], sourceType: 'script', }
  19. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST const escodegen

    = require('escodegen'); escodegen.generate({ type: 'Program', body: [ { type: 'ExpressionStatement', expression: { type: 'Literal', value: true, raw: 'true' }, }, ], sourceType: 'script', });
  20. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Practical Examples

    Rendering ASTs Rendering ASTs Practical Examples
  21. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Libraries to

    create ASTs • @babel/parser (aka. babylon) • espree (ESLint’s parser) • typescript-estree (Used by ESLint & prettier) • flow-parser • esprima • uglify-js • acorn (the foundation of a lot of parsers)
  22. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Libraries to

    walk ASTs • @babel/traverse • burrito, falafel • rocambole • babylon-walk (lightweight alternative to @babel/ traverse) • acorn-walk • jscodeshift
  23. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Libraries to

    render ASTs • @babel/generator • escodegen • astring