ASTs in JavaScript

A presentation I held at JavaZone in 2017 about ASTs and tooling in JavaScript

1. ASTs
2. We want to learn more about the internals of tools we use everyday
3. Abstract syntax trees (AST) are at the core of all of these tools
4. … but what are abstract syntax trees? … how do they work in JavaScript? … how can we use them effectively?
5. const conference = "JavaZone";
6. const conference = "JavaZone"; Lexical analysis (lexer)
7. const conference = "JavaZone"; keyword Lexical analysis (lexer)
8. const conference = "JavaZone"; keyword identifier Lexical analysis (lexer)
9. const conference = "JavaZone"; keyword identifier punctuator string punctuator Lexical analysis (lexer)
10. [ { type: 'Keyword', value: 'const' }, { type: 'Identifier', value: 'conference' }, { type: 'Punctuator', value: '=' }, { type: 'Numeric', value: 'JavaZone' }, { type: 'Punctuator', value: ';' } ] const conference = "JavaZone";
11. Syntax analysis (parser) const conference = "JavaZone"; [ { type: 'Keyword', value: 'const' }, { type: 'Identifier', value: 'conference' }, { type: 'Punctuator', value: '=' }, { type: 'Numeric', value: 'JavaZone' }, { type: 'Punctuator', value: ';' } ]
12. const conference = "JavaZone"; Syntax analysis (parser) const conference = "JavaZone"; Program
13. Syntax analysis (parser) const conference = "JavaZone"; VariableDeclaration Program
14. Syntax analysis (parser) const conference = "JavaZone"; VariableDeclaration VariableDeclarator Program Kind
15. const conference = "JavaZone"; VariableDeclaration VariableDeclarator Identifier Literal Program Kind Syntax analysis (parser)
16. const conference = "JavaZone"; Program VariableDeclaration VariableDeclarator const Identifier Literal conference "JavaZone"
21. Okey, so now we know what an AST is
22. … but what can we do with it? Okey, so now we know what an AST is
23. Use-cases Static analysis Syntax highlighting Code completion Minification Compile-time optimisations Source maps Code coverage Transpilers Code formatter One-time code transformations Build-time code transformations
25. Static analysis
26. module.exports = { }; Creating our own ESLint plugin that disallows default exports export default myFunction
27. module.exports = { meta: {}, create: function(context) { return { }; } }; ESLint helpers Creating our own ESLint plugin that disallows default exports
28. Node type module.exports = { meta: {}, create: function(context) { return { ExportDefaultDeclaration: function(node) { } }; } }; (we’ll talk about how we find these later) ESLint calls this function to “visit” the node while traversing down the AST Creating our own ESLint plugin that disallows default exports
29. module.exports = { meta: {}, create: function(context) { return { ExportDefaultDeclaration: function(node) { context.report(node, 'Default exports are not allowed.'); } }; } }; Publishes a warning or error Creating our own ESLint plugin that disallows default exports
30. create: function(context) { const dependencies = new Set(); return { ImportDeclaration: function(node) { dependencies.add(node.source.value); }, 'Program:exit': function() { failIfTooManyDeps(dependencies, context); } } } ESLint example: fail if too many dependencies
31. You’ll see the visitor pattern everywhere
32. Build-time transformation
42. WHY!?!?
43. WHY!?!? Legacy, JavaScript is evolving, tools have different needs
44. WHY!?!? “NOTE: Only 'stage 4' (finalized) ECMAScript features are being implemented by Acorn.” “[Babylon has] support for experimental language proposals” "Unfortunately, ESLint relies on more than just the AST to do its job. It relies on […] tokens and comment attachment features to get a complete picture of the source code." “Experimental support for JSX”
46. Why isn’t there an AST spec?
48. Experimental features Early spec stages JSX, TypeScript, etc Not everyone agrees
49. So … how do we solve this problem?
54. Transform AST
57. Specify parser, then transform ASTs Several almost compatible ASTs
58. However, it's not easy to get the best of both worlds
59. However, it's not easy to get the best of both worlds Only one can parse
60. Working with AST Explorer
66. Codemods One-time transformation
67. Codemods Using AST-to-AST transformations to enable large-scale, incremental changes
68. Some of the changes we’ve done over time: AMD CommonJS ES6 modules var const / let Change test lib (Mocha to Jest) and assertion lib (e.g. to.be() to toBe()) Remove unused code, e.g. unused variables and functions Object.assign Object spread Introduce support for I18N
69. Some of these might be easy to search-and-replace, but most of them are not
70. Writing a codemod
71. Object.assign to Object spread
72. { type: 'dashboard', id: indexId, ...additionalOptions } Object.assign( { type: 'dashboard', id: indexId }, additionalOptions ); Object.assign to Object spread
73. Object.assign( {}, query, hashedQuery ); Object.assign to Object spread { ...query, ...hashedQuery };
74. Object.assign( myVar, { type: 'dashboard', id: indexId } ); Object.assign to Object spread Don’t change as it’s mutating myVar
75. // most parsers: visitor pattern const ast = parse(file.source); traverse(ast, { Identifier: function(node) { // do something with node } }); const ast = j(file.source); ast .find(j.Identifier) .forEach(function(path) { // do something with path }); // jscodeshift: fluent interface
76. const ast = j(file.source); ast .find(j.Identifier) .forEach(function(path) { // do something with path }); Collection: Similar to jQuery's $(...) APIs to find, filter, map and remove AST nodes.
77. Wraps an AST node and adds helpers to simplify working with it. Path: Collection: Similar to jQuery's $(...) APIs to find, filter, map and remove AST nodes. const ast = j(file.source); ast .find(j.Identifier) .forEach(function(path) { // do something with path });
78. Let’s look at some code in slides (I’m not tough enough to live code this)
79. Object.assign({}, obj);
80. Object.assign({}, obj); AST { "type": "CallExpression", "callee": { "type": "MemberExpression", "object": { "type": "Identifier", "name": "Object" }, "property": { "type": "Identifier", "name": "assign" } }, "arguments": [ { "type": "ObjectExpression", "properties": [] }, { "type": "Identifier", "name": "obj" } ] }
81. .find( ); Object.assign({}, obj); AST { "type": "CallExpression", "callee": { "type": "MemberExpression", "object": { "type": "Identifier", "name": "Object" }, "property": { "type": "Identifier", "name": "assign" } }, "arguments": [ { "type": "ObjectExpression", "properties": [] }, { "type": "Identifier", "name": "obj" } ] }
82. .find(j.CallExpression ); Object.assign({}, obj); AST { "type": "CallExpression", "callee": { "type": "MemberExpression", "object": { "type": "Identifier", "name": "Object" }, "property": { "type": "Identifier", "name": "assign" } }, "arguments": [ { "type": "ObjectExpression", "properties": [] }, { "type": "Identifier", "name": "obj" } ] }
83. Object.assign({}, obj); .find(j.CallExpression, { callee: { type: "MemberExpression", object: { name: "Object" }, property: { name: "assign" } } }); AST { "type": "CallExpression", "callee": { "type": "MemberExpression", "object": { "type": "Identifier", "name": "Object" }, "property": { "type": "Identifier", "name": "assign" } }, "arguments": [ { "type": "ObjectExpression", "properties": [] }, { "type": "Identifier", "name": "obj" } ] }
84. Object.assign({}, obj); test(); Object.assign( {}, query, hashedQuery ); Object.keys({ hello: 'world' }); console.log('hello world'); export default function transformer(file, api) { const j = api.jscodeshift; const ast = j(file.source); return ast.toSource(); } Input Transform
85. Object.assign({}, obj); test(); Object.assign( {}, query, hashedQuery ); Object.keys({ hello: 'world' }); console.log('hello world'); export default function transformer(file, api) { const j = api.jscodeshift; const ast = j(file.source); ast .find(j.CallExpression, { callee: { type: "MemberExpression", object: { name: "Object" }, property: { name: "assign" } } }) return ast.toSource(); } Input Transform
86. Object.assign({}, obj); test(); Object.assign( {}, query, hashedQuery ); Object.keys({ hello: 'world' }); console.log('hello world'); export default function transformer(file, api) { const j = api.jscodeshift; const ast = j(file.source); ast .find(j.CallExpression, { callee: { type: "MemberExpression", object: { name: "Object" }, property: { name: "assign" } } }) .forEach(path => transformToObject(path)); return ast.toSource(); } Input Transform
87. { "type": "CallExpression", "callee": { "type": "MemberExpression", "object": { "type": "Identifier", "name": "Object" }, "property": { "type": "Identifier", "name": "assign" } }, "arguments": [ { "type": "ObjectExpression", "properties": [] }, { "type": "Identifier", "name": "obj" } ] } function transformToObject(path) { } Wraps the AST
88. { "type": "CallExpression", "callee": { "type": "MemberExpression", "object": { "type": "Identifier", "name": "Object" }, "property": { "type": "Identifier", "name": "assign" } }, "arguments": [ { "type": "ObjectExpression", "properties": [] }, { "type": "Identifier", "name": "obj" } ] } function transformToObject(path) { const assignCall = path.value; }
89. { "type": "CallExpression", "callee": { "type": "MemberExpression", "object": { "type": "Identifier", "name": "Object" }, "property": { "type": "Identifier", "name": "assign" } }, "arguments": [ { "type": "ObjectExpression", "properties": [] }, { "type": "Identifier", "name": "obj" } ] } function transformToObject(path) { const assignCall = path.value; const args = assignCall.arguments; }
90. function transformToObject(path) { const assignCall = path.value; const args = assignCall.arguments; const newProps = createObjectProperties(args); }
91. function transformToObject(path) { const assignCall = path.value; const args = assignCall.arguments; const newProps = createObjectProperties(args); path.replace( j.objectExpression(newProps) ); } Builders to generate AST nodes Replace Object.assign call with an object expression j.identifier("name") j.memberExpression(j.identifier("foo"), j.identifier("bar")) j.objectExpression([ j.property("init", j.literal("key"), j.literal("value")) ]) j.callExpression( j.identifier("foo"), [j.identifier("bar")] ) ...
94. Then I copy the transform into a local file and run it on the full codebase (and fix all the edge-cases)
95. The goal isn't always to solve 100% of the problems with this
96. if (isStopped) { return <div className='is-warning'> <FormattedMessage id='cluster-status.stopped' defaultMessage='Stopped' /> </div> } if (isStopped) { return <div className='is-warning'> Stopped </div> } Introducing I18N using React-intl and jscodeshift
97. <div> <FormattedMessage id='hello-name' defaultMessage='Hello { name }' values={{ name: <strong>{ name }</strong> }} /> </div> <div> Hello <strong>{ name }<strong> </div> Introducing I18N using React-intl and jscodeshift
98. Took us a couple of days on a 30k+ line codebase to solve 80% of the problems
99. Interesting use-cases: Libraries can include transforms that fix breaking changes, e.g. React deprecating PropTypes Transforms for switching test libraries, e.g. moving from Mocha to Jest Pre-built codemods for moving to ES6+ features, e.g. arrow functions and const/let