SinnerSchrader Tech Session '18 - What the AST?

SinnerSchrader Tech Session '18 - What the AST?

Two fundamental things that make the life of every developer easier are tokenizers and Abstract Syntax Trees. Whether it's your favorite syntax highlighting, linter, code formatting tool, or tools like the TypeScript compiler or Babel. All rely on these fundamental concepts. But how do they work?

In this session we'll look at what Tokenizer and Syntax parsers do, how an Abstract Syntax Tree looks like, why they are useful and other ways developers can benefit from them.

0722ad084c65f6177d80cf793cfbd013?s=128

Dominik Kundel

October 11, 2018
Tweet

Transcript

  1. AST? Dominik Kundel | @dkundel | #s2techsession | #whatTheAST What

    the Photo by Paweł Czerwiński on Unsplash
  2. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST

  3. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Have you

    used these tools?
  4. AST? Dominik Kundel | @dkundel | #s2techsession | #whatTheAST What

    the Photo by Paweł Czerwiński on Unsplash
  5. Dominik Kundel | @dkundel | #s2techsession | console.log(` Hi! I’m

    Dominik Kundel `); dkundel.com @dkundel dkundel@twilio.com github/dkundel Developer Evangelist && JavaScript Hacker #whatTheAST
  6. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST OnesieJS

  7. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST How do

    we find / modify code?
  8. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Ctrl/Cmd +

    F
  9. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST grep /

    sed
  10. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST /RegularExpressions/gi

  11. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Unreliable &

    Hard to maintain
  12. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Let’s talk

    about those tools again Linters / Code Formatters Frameworks Bundlers Compilers / Transpilers
  13. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Code >

    ✨ > Output Linters / Code Formatters Frameworks Bundlers Compilers / Transpilers
  14. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST What’s that

    Magic? ✨
  15. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Picture by

    @_lrlna
  16. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Lexical Analysis

    Tokenization
  17. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST console.log('Hello');

  18. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST isPanda('');

  19. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST const {

    tokenize } = require('esprima'); const tokens = tokenize("isPanda('')");
  20. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST [ {

    type: 'Identifier', value: 'isPanda' }, { type: 'Punctuator', value: '(' }, { type: 'String', value: "''" }, { type: 'Punctuator', value: ')' }, ];
  21. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Practical Examples

    Lexical Analysis Practical Examples
  22. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Syntax Highlighting

  23. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST https://code.visualstudio.com/blogs/2017/02/08/syntax-highlighting-optimizations

  24. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Abstract Syntax

    Trees ASTs
  25. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST [ {

    type: 'Identifier', value: 'isPanda' }, { type: 'Punctuator', value: '(' }, { type: 'String', value: "''" }, { type: 'Punctuator', value: ')' }, ];
  26. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST { type:

    'Program', body: [ { type: 'ExpressionStatement', expression: { type: 'CallExpression', callee: { type: 'Identifier', name: 'isPanda' }, arguments: [{ type: 'Literal', value: '', raw: "''" }], }, }, ], sourceType: 'script', }
  27. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST https://github.com/estree/estree ESTree

    Spec
  28. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Official ESTree

    specification ESTree Spec Alterations Esprima Syntax Tree Format Babel AST Format Facebook JSX Extension
  29. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Practical Examples

    ASTs Practical Examples
  30. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Linting

  31. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Code Analysis

    angular.io/guide/language-service
  32. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Bundling

  33. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Modifying ASTs

  34. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST { type:

    'Program', body: [ { type: 'ExpressionStatement', expression: { type: 'CallExpression', callee: { type: 'Identifier', name: 'isPanda' }, arguments: [{ type: 'Literal', value: '', raw: "''" }], }, }, ], sourceType: 'script', }
  35. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST { type:

    'Program', body: [ { type: 'ExpressionStatement', expression: { type: 'Literal', value: true, raw: 'true' }, }, ], sourceType: 'script', }
  36. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Practical Examples

    Modifying ASTs Practical Examples
  37. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Code Modifications

  38. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Language Extensions

    <Hello jsx={true} />
  39. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Language Transpiling

  40. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Code Coverage

    https://istanbul.js.org/
  41. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST nyc instrument

    sum.js ./output
  42. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST sum.js function

    sum(a, b) { return a + b; } module.exports = sum;
  43. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST ./output/sum.js var

    cov_2mh38x6lsj = (function () { /* ... */ })(); function sum(a, b) { cov_2mh38x6lsj.f[0] ++; cov_2mh38x6lsj.s[0] ++; return a + b; } cov_2mh38x6lsj.s[1] ++; module.exports = sum;
  44. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST var cov_2mh38x6lsj

    = (function() { var path = 'sum.js', hash = 'e40ca58a296221aa5a52fed6ee0f25cc89e6447b', Function = function() {}.constructor, global = new Function('return this')(), gcv = ' __coverage __', coverageData = { /* ... */ }, coverage = global[gcv] || (global[gcv] = {}); if (coverage[path] && coverage[path].hash === hash) { return coverage[path]; } coverageData.hash = hash; return (coverage[path] = coverageData); })();
  45. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST coverageData =

    { path: 'sum.js', statementMap: { '0': { start: { line: 2, column: 2 }, end: { … } }, '1': { start: { line: 4, column: 0 }, end: { … } }, }, fnMap: { '0': { name: 'sum', decl: { start: { … }, end: { … } }, loc: { start: { … }, end: { … } }, line: 1, }, }, branchMap: {}, s: { '0': 0, '1': 0 }, f: { '0': 0 }, b: {}, _coverageSchema: 'd34fc3e6b8297bcde183f5492bcb8fcb36775295', },
  46. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Understanding Code

    Coverage engineering.semantics3.com/understanding-code-coverage-1074e8fccce0 More Info
  47. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Running other

    languages in the Browser
  48. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST codecombat.com

  49. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST esper.js.org

  50. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Rendering ASTs

    Rendering ASTs
  51. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST { type:

    'Program', body: [ { type: 'ExpressionStatement', expression: { type: 'Literal', value: true, raw: 'true' }, }, ], sourceType: 'script', }
  52. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST const escodegen

    = require('escodegen'); escodegen.generate({ type: 'Program', body: [ { type: 'ExpressionStatement', expression: { type: 'Literal', value: true, raw: 'true' }, }, ], sourceType: 'script', });
  53. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST true;

  54. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Practical Examples

    Rendering ASTs Rendering ASTs Practical Examples
  55. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Code Formatting

  56. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Minification

  57. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Why would

    I care?
  58. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Babel Transform

    Plugins
  59. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST grep /

    sed
  60. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST /RegularExpressions/gi

  61. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST /RegularExpressions/gi (

    )
  62. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Building a

    Babel Plugin Live Demo
  63. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Summary Photo

    by Jan Kahánek on Unsplash
  64. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST 1. You

    use Tokens & ASTs daily
  65. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST 2. Tokens

    represent format
  66. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST 3. ASTs

    represent structure
  67. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST 4. ASTs

    are “easy” to manipulate
  68. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST 5. AST

    for safer code alterations
  69. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST 6. ESTree

    for interoperability
  70. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST 7. Don’t

    reinvent the wheel
  71. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Libraries to

    create ASTs • @babel/parser (aka. babylon) • espree (ESLint’s parser) • typescript-estree (Used by ESLint & prettier) • flow-parser • esprima • uglify-js • acorn (the foundation of a lot of parsers)
  72. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Libraries to

    walk ASTs • @babel/traverse • burrito, falafel • rocambole • babylon-walk (lightweight alternative to @babel/ traverse) • acorn-walk • jscodeshift
  73. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Libraries to

    render ASTs • @babel/generator • escodegen • astring
  74. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST 8. Replace

    grep & sed
  75. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST astexplorer.net

  76. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST git.io/babel-handbook

  77. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Everything Open

    Source
  78. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST hacktoberfest.digitalocean.com

  79. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST www.twilio.com/quest

  80. Dominik Kundel | @dkundel | #s2techsession | #whatTheAST Stickers!!

  81. console.log(` Thank You! `); dkundel.com @dkundel dkundel@twilio.com github/dkundel d-k.im/s2-ast Dominik

    Kundel | @dkundel | #s2techsession | #whatTheAST