ASTs in JavaScript

ASTs in JavaScript

A presentation I held at JavaZone in 2017 about ASTs and tooling in JavaScript

6c51c14716e24bc1f1a3fb5ad234e773?s=128

Kim Joar Bekkelund

September 13, 2017
Tweet

Transcript

  1. ASTs

  2. We want to learn more about the internals of tools

    we use everyday
  3. Abstract syntax trees (AST) are at the core of all

    of these tools
  4. … but what are abstract syntax trees? … how do

    they work in JavaScript? … how can we use them effectively?
  5. const conference = "JavaZone";

  6. const conference = "JavaZone"; Lexical analysis (lexer)

  7. const conference = "JavaZone"; keyword Lexical analysis (lexer)

  8. const conference = "JavaZone"; keyword identifier Lexical analysis (lexer)

  9. const conference = "JavaZone"; keyword identifier punctuator string punctuator Lexical

    analysis (lexer)
  10. [ { type: 'Keyword', value: 'const' }, { type: 'Identifier',

    value: 'conference' }, { type: 'Punctuator', value: '=' }, { type: 'Numeric', value: 'JavaZone' }, { type: 'Punctuator', value: ';' } ] const conference = "JavaZone";
  11. Syntax analysis (parser) const conference = "JavaZone"; [ { type:

    'Keyword', value: 'const' }, { type: 'Identifier', value: 'conference' }, { type: 'Punctuator', value: '=' }, { type: 'Numeric', value: 'JavaZone' }, { type: 'Punctuator', value: ';' } ]
  12. const conference = "JavaZone"; Syntax analysis (parser) const conference =

    "JavaZone"; Program
  13. Syntax analysis (parser) const conference = "JavaZone"; VariableDeclaration Program

  14. Syntax analysis (parser) const conference = "JavaZone"; VariableDeclaration VariableDeclarator Program

    Kind
  15. const conference = "JavaZone"; VariableDeclaration VariableDeclarator Identifier Literal Program Kind

    Syntax analysis (parser)
  16. const conference = "JavaZone"; Program VariableDeclaration VariableDeclarator const Identifier Literal

    conference "JavaZone"
  17. None
  18. None
  19. None
  20. None
  21. Okey, so now we know what an AST is

  22. … but what can we do with it? Okey, so

    now we know what an AST is
  23. Use-cases Static analysis Syntax highlighting Code completion Minification Compile-time optimisations

    Source maps Code coverage Transpilers Code formatter One-time code transformations Build-time code transformations
  24. None
  25. Static analysis

  26. module.exports = { }; Creating our own ESLint plugin that

    disallows default exports export default myFunction
  27. module.exports = { meta: {}, create: function(context) { return {

    }; } }; ESLint helpers Creating our own ESLint plugin that disallows default exports
  28. Node type module.exports = { meta: {}, create: function(context) {

    return { ExportDefaultDeclaration: function(node) { } }; } }; (we’ll talk about how we find these later) ESLint calls this function to “visit” the node while traversing down the AST Creating our own ESLint plugin that disallows default exports
  29. module.exports = { meta: {}, create: function(context) { return {

    ExportDefaultDeclaration: function(node) { context.report(node, 'Default exports are not allowed.'); } }; } }; Publishes a warning or error Creating our own ESLint plugin that disallows default exports
  30. create: function(context) { const dependencies = new Set(); return {

    ImportDeclaration: function(node) { dependencies.add(node.source.value); }, 'Program:exit': function() { failIfTooManyDeps(dependencies, context); } } } ESLint example: fail if too many dependencies
  31. You’ll see the visitor pattern everywhere

  32. Build-time transformation

  33. None
  34. None
  35. None
  36. None
  37. None
  38. None
  39. None
  40. None
  41. None
  42. WHY!?!?

  43. WHY!?!? Legacy, JavaScript is evolving, tools have different needs

  44. WHY!?!? “NOTE: Only 'stage 4' (finalized) ECMAScript features are being

    implemented by Acorn.” “[Babylon has] support for experimental language proposals” "Unfortunately, ESLint relies on more than just the AST to do its job. It relies on […] tokens and comment attachment features to get a complete picture of the source code." “Experimental support for JSX”
  45. None
  46. Why isn’t there an AST spec?

  47. None
  48. Experimental features Early spec stages JSX, TypeScript, etc Not everyone

    agrees
  49. So … how do we solve this problem?

  50. None
  51. None
  52. None
  53. None
  54. Transform AST

  55. None
  56. None
  57. Specify parser, then transform ASTs Several almost compatible ASTs

  58. However, it's not easy to get the best of both

    worlds
  59. However, it's not easy to get the best of both

    worlds Only one can parse
  60. Working with AST Explorer

  61. None
  62. None
  63. None
  64. None
  65. None
  66. Codemods One-time transformation

  67. Codemods Using AST-to-AST transformations to enable large-scale, incremental changes

  68. Some of the changes we’ve done over time: AMD CommonJS

    ES6 modules var const / let Change test lib (Mocha to Jest) and assertion lib (e.g. to.be() to toBe()) Remove unused code, e.g. unused variables and functions Object.assign Object spread Introduce support for I18N
  69. Some of these might be easy to search-and-replace, but most

    of them are not
  70. Writing a codemod

  71. Object.assign to Object spread

  72. { type: 'dashboard', id: indexId, ...additionalOptions } Object.assign( { type:

    'dashboard', id: indexId }, additionalOptions ); Object.assign to Object spread
  73. Object.assign( {}, query, hashedQuery ); Object.assign to Object spread {

    ...query, ...hashedQuery };
  74. Object.assign( myVar, { type: 'dashboard', id: indexId } ); Object.assign

    to Object spread Don’t change as it’s mutating myVar
  75. // most parsers: visitor pattern const ast = parse(file.source); traverse(ast,

    { Identifier: function(node) { // do something with node } }); const ast = j(file.source); ast .find(j.Identifier) .forEach(function(path) { // do something with path }); // jscodeshift: fluent interface
  76. const ast = j(file.source); ast .find(j.Identifier) .forEach(function(path) { // do

    something with path }); Collection: Similar to jQuery's $(...) APIs to find, filter, map and remove AST nodes.
  77. Wraps an AST node and adds helpers to simplify working

    with it. Path: Collection: Similar to jQuery's $(...) APIs to find, filter, map and remove AST nodes. const ast = j(file.source); ast .find(j.Identifier) .forEach(function(path) { // do something with path });
  78. Let’s look at some code in slides (I’m not tough

    enough to live code this)
  79. Object.assign({}, obj);

  80. Object.assign({}, obj); AST { "type": "CallExpression", "callee": { "type": "MemberExpression",

    "object": { "type": "Identifier", "name": "Object" }, "property": { "type": "Identifier", "name": "assign" } }, "arguments": [ { "type": "ObjectExpression", "properties": [] }, { "type": "Identifier", "name": "obj" } ] }
  81. .find( ); Object.assign({}, obj); AST { "type": "CallExpression", "callee": {

    "type": "MemberExpression", "object": { "type": "Identifier", "name": "Object" }, "property": { "type": "Identifier", "name": "assign" } }, "arguments": [ { "type": "ObjectExpression", "properties": [] }, { "type": "Identifier", "name": "obj" } ] }
  82. .find(j.CallExpression ); Object.assign({}, obj); AST { "type": "CallExpression", "callee": {

    "type": "MemberExpression", "object": { "type": "Identifier", "name": "Object" }, "property": { "type": "Identifier", "name": "assign" } }, "arguments": [ { "type": "ObjectExpression", "properties": [] }, { "type": "Identifier", "name": "obj" } ] }
  83. Object.assign({}, obj); .find(j.CallExpression, { callee: { type: "MemberExpression", object: {

    name: "Object" }, property: { name: "assign" } } }); AST { "type": "CallExpression", "callee": { "type": "MemberExpression", "object": { "type": "Identifier", "name": "Object" }, "property": { "type": "Identifier", "name": "assign" } }, "arguments": [ { "type": "ObjectExpression", "properties": [] }, { "type": "Identifier", "name": "obj" } ] }
  84. Object.assign({}, obj); test(); Object.assign( {}, query, hashedQuery ); Object.keys({ hello:

    'world' }); console.log('hello world'); export default function transformer(file, api) { const j = api.jscodeshift; const ast = j(file.source); return ast.toSource(); } Input Transform
  85. Object.assign({}, obj); test(); Object.assign( {}, query, hashedQuery ); Object.keys({ hello:

    'world' }); console.log('hello world'); export default function transformer(file, api) { const j = api.jscodeshift; const ast = j(file.source); ast .find(j.CallExpression, { callee: { type: "MemberExpression", object: { name: "Object" }, property: { name: "assign" } } }) return ast.toSource(); } Input Transform
  86. Object.assign({}, obj); test(); Object.assign( {}, query, hashedQuery ); Object.keys({ hello:

    'world' }); console.log('hello world'); export default function transformer(file, api) { const j = api.jscodeshift; const ast = j(file.source); ast .find(j.CallExpression, { callee: { type: "MemberExpression", object: { name: "Object" }, property: { name: "assign" } } }) .forEach(path => transformToObject(path)); return ast.toSource(); } Input Transform
  87. { "type": "CallExpression", "callee": { "type": "MemberExpression", "object": { "type":

    "Identifier", "name": "Object" }, "property": { "type": "Identifier", "name": "assign" } }, "arguments": [ { "type": "ObjectExpression", "properties": [] }, { "type": "Identifier", "name": "obj" } ] } function transformToObject(path) { } Wraps the AST
  88. { "type": "CallExpression", "callee": { "type": "MemberExpression", "object": { "type":

    "Identifier", "name": "Object" }, "property": { "type": "Identifier", "name": "assign" } }, "arguments": [ { "type": "ObjectExpression", "properties": [] }, { "type": "Identifier", "name": "obj" } ] } function transformToObject(path) { const assignCall = path.value; }
  89. { "type": "CallExpression", "callee": { "type": "MemberExpression", "object": { "type":

    "Identifier", "name": "Object" }, "property": { "type": "Identifier", "name": "assign" } }, "arguments": [ { "type": "ObjectExpression", "properties": [] }, { "type": "Identifier", "name": "obj" } ] } function transformToObject(path) { const assignCall = path.value; const args = assignCall.arguments; }
  90. function transformToObject(path) { const assignCall = path.value; const args =

    assignCall.arguments; const newProps = createObjectProperties(args); }
  91. function transformToObject(path) { const assignCall = path.value; const args =

    assignCall.arguments; const newProps = createObjectProperties(args); path.replace( j.objectExpression(newProps) ); } Builders to generate AST nodes Replace Object.assign call with an object expression j.identifier("name") j.memberExpression(j.identifier("foo"), j.identifier("bar")) j.objectExpression([ j.property("init", j.literal("key"), j.literal("value")) ]) j.callExpression( j.identifier("foo"), [j.identifier("bar")] ) ...
  92. None
  93. None
  94. Then I copy the transform into a local file and

    run it on the full codebase (and fix all the edge-cases)
  95. The goal isn't always to solve 100% of the problems

    with this
  96. if (isStopped) { return <div className='is-warning'> <FormattedMessage id='cluster-status.stopped' defaultMessage='Stopped' />

    </div> } if (isStopped) { return <div className='is-warning'> Stopped </div> } Introducing I18N using React-intl and jscodeshift
  97. <div> <FormattedMessage id='hello-name' defaultMessage='Hello { name }' values={{ name: <strong>{

    name }</strong> }} /> </div> <div> Hello <strong>{ name }<strong> </div> Introducing I18N using React-intl and jscodeshift
  98. Took us a couple of days on a 30k+ line

    codebase to solve 80% of the problems
  99. Interesting use-cases: Libraries can include transforms that fix breaking changes,

    e.g. React deprecating PropTypes Transforms for switching test libraries, e.g. moving from Mocha to Jest Pre-built codemods for moving to ES6+ features, e.g. arrow functions and const/let
  100. None
  101. None