Share
Embed
Don't be Afraid of ASTs
An early guide to abstract syntax trees in JavaScript
Jamund Ferguson
November 18, 2014
Transcript
99 slides
-
Our Basic Plan 1. High-level overview 2. Static Analysis with
ASTs 3. Transforming and refactoring 4. A quick look at the Mozilla Parser API (de-facto standard AST format) -
An AST makes it easier to inspect and manipulate your
code with confidence. -
{ "type": "Program", "body": [ { "type": “VariableDeclaration", "kind": "var"
"declarations": [ { "type": "VariableDeclarator", "id": { "type": "Identifier", "name": "fullstack" }, "init": { "type": "BinaryExpression", "left": { "type": "Identifier", "name": "node" }, "operator": "+", "right": { "type": "Identifier", "name": "ui" } } } ], } ] } -
{ "type": "Program", "body": [ { "type": “VariableDeclaration", "kind": "var"
"declarations": [ { "type": "VariableDeclarator", "id": { "type": "Identifier", "name": "fullstack" }, "init": { "type": "BinaryExpression", "left": { "type": "Identifier", "name": "node" }, "operator": "+", "right": { "type": "Identifier", "name": "ui" } } } ], } ] } -
Things Built On ASTs • Syntax Highlighting • Code Completion
• Static Analysis (aka JSLint, etc.) • Code Coverage • Minification • JIT Compilation • Source Maps • Compile to JS Languages So much more… -
Write some really solid static analysis. Never write that same
type of bug again. -
function loadUser(req, res, next) { User.loadUser(function(err, user) { req.session.user =
user; next(); }); } Bad Example We forgot to handle the error! -
1. Each time a function is declared check if there
is an error* parameter 2. If so set a count to 0; 3. Increment count when error is used 4. At the end of the function warn when count is empty * the parameter name can be defined by the user handle-callback-err -
History Lesson • 1995: JavaScript • 2002: JSLint started by
Douglas Crockford • 2011: JSHint comes out as a fork of JSLint. Esprima AST parser released. • 2012: plato, escomplex, complexity-report • 2013: Nicholas Zakas releases ESLint. Marat Dulin releases JSCS. -
no-console return { "MemberExpression": function(node) { if (node.object.name === "console")
{ context.report(node, "Unexpected console statement.”); } } }; https://github.com/eslint/eslint/blob/master/lib/rules/no-console.js -
no-loop-func function checkForLoops(node) { var ancestors = context.getAncestors(); if (ancestors.some(function(ancestor)
{ return ancestor.type === "ForStatement" || ancestor.type === "WhileStatement" || ancestor.type === "DoWhileStatement"; })) { context.report(node, "Don't make functions within a loop"); } } return { "FunctionExpression": checkForLoops, "FunctionDeclaration": checkForLoops }; -
max-params var numParams = context.options[0] || 3; function checkParams(node) {
if (node.params.length > numParams) { context.report(node, "This function has too many parameters ({{count}}). Maximum allowed is {{max}}.", { count: node.params.length, max: numParams }); } } return { “FunctionDeclaration”: checkParams, “FunctionExpression”: checkParams } -
no-jquery function isjQuery(name) { return name === '$' || name
=== 'jquery' || name === 'jQuery'; } return { “CallExpression”: function(node) { var name = node.callee && node.callee.name; if (isjQuery(name)) { context.report(node, 'Please avoid using jQuery here.’); } } } -
More Complex Rules • indent • no-extend-native • no-next-next •
security • internationalization -
Other Areas for Static Analysis Code complexity and visualization is
another area where static analysis is really useful. Plato is an exciting start, but I believe there are tons of more interesting things that can be done in this area. -
Recap • Static Analysis can help you catch real bugs
and keep your code maintainable • ESLint and JSCS both use ASTs for inspecting your code to make it easy to cleanly to add new rules • Static analysis can also help you manage your code complexity as well • What exactly does a for loop sound like? -
Sometimes you want to step into the future, but something
is keeping you in the past. -
Tools like falafel and recast give you an API to
manipulate an AST and then convert that back into source code. -
Two Types of AST Transformations Regenerative Regenerate the full file
from the AST. Often losing comments and non-essential formatting. Fine for code not read by humans (i.e. browserify transforms). Partial-source transformation Regenerate only the parts of the source that have changed based on the AST modifications. Nicer for one-time changes in source. -
var fullstack = node + ui; var fullstack = node
+ browserify; -
4 Steps 1. Buffer up the stream of source code
2. Convert the source into an AST 3. Transform the AST 4. Re-generate and output the source -
Step 1 var through = require(‘through'); var buffer = [];
return through(function write(data) { buffer.push(data); }, function end () { var source = buffer.join(‘’); }); Use through to grab the source code -
Step 2 var falafel = require(‘falafel’); function end () {
var source = buffer.join(‘’); var out = falafel(source, parse).toString(); } Use falafel to transform create an AST -
Step 3 function parse(node) { if (node.type === 'Identifier' &&
node.value === ‘ui’) { node.update('browserify'); } } Use falafel to transform the AST -
Step 4 function end () { var source = buffer.join(‘’);
var out = falafel(source, parse).toString(); this.queue(out); this.queue(null); // end the stream } Stream the source with through and close the stream -
var through = require('through'); var falafel = require('falafel'); module.exports =
function() { var buffer = []; return through(function write(data) { buffer.push(data); }, function end() { var source = buffer.join('\n'); var out = falafel(source, parse).toString(); this.queue(out); this.queue(null); // close the stream }); }; function parse(node) { if (node.type === 'Identifier' && node.name === 'ui') { node.update('browserify'); } } -
It Works! browserify -t ./ui-to-browserify.js code.js (function e(t,n,r){function s(o,u){if(!n[o]){if(!t[o]){var a=typeof
require=="function"&&require;if(!u&&a)return a(o,!0);if(i)return i(o,!0);var f=new Error("Cannot find module '"+o+"'");throw f.code="MODULE_NOT_FOUND",f} var l=n[o]={exports:{}};t[o][0].call(l.exports,function(e){var n=t[o][1] [e];return s(n?n:e)},l,l.exports,e,t,n,r)}return n[o].exports}var i=typeof require=="function"&&require;for(var o=0;o<r.length;o++)s(r[o]);return s}) ({1:[function(require,module,exports){ var fullstack = node + browserify; },{}]},{},[1]); -
And it’s a building block for building a bunch of
cooler things. -
A Basic Map/Filter var a = [1, 2, 3]; var
b = a.filter(function(n) { return n > 1; }).map(function(k) { return k * 2; }); -
Faster Like This var a = [1, 2, 3]; var
b = []; for (var i = 0; i < a.length; i++) { if (a[i] > 1) { b.push(a[i] * 2); } } -
A Basic Recast Script var recast = require(‘recast’); var code
= fs.readFileSync(‘code.js', 'utf-8'); var ast = recast.parse(code); var faster = transform(ast); var output = recast.print(faster).code; -
function transform(ast) { var transformedAST = new MapFilterEater({ body: ast.program.body
}).visit(ast); return transformedAST; } var Visitor = recast.Visitor; var MapFilterEater = Visitor.extend({ init: function(options) {}, visitForStatement: function(ast) {}, visitIfStatement: function(ast) {}, visitCallExpression: function(ast) {}, visitVariableDeclarator: function(ast) {} }); -
How Does it Work? 1. Move the right side of
the b declaration into a for loop 2. Set b = [] 3. Place the .filter() contents inside of an if statement 4. Unwrap the .map contents and .push() them into b 5. Replace all of the local counters with a[_i] -
And Voila…. var a = [1, 2, 3]; var b
= []; for (var i = 0; i < a.length; i++) { if (a[i] > 1) { b.push(a[i] * 2); } } -
The most well-read documentation for how to engineer your app
is the current codebase. -
Parser 1. Read your raw JavaScript source. 2. Parse out
every single thing that’s happening. 3. Return an AST that represents your code -
Esprima is a very popular* parser that converts your code
into an abstract syntax tree. *FB recently forked it to add support for ES6 and JSX -
Esprima follows the Mozilla Parser API which is a well
documented AST format used internally by Mozilla (and now by basically everyone else*) -
{ "type": "Program", "body": [ { "type": “VariableDeclaration", "kind": "var"
"declarations": [ { "type": "VariableDeclarator", "id": { "type": "Identifier", "name": "fullstack" }, "init": { "type": "BinaryExpression", "left": { "type": "Identifier", "name": "node" }, "operator": "+", "right": { "type": "Identifier", "name": "ui" } } } ], } ] } -
{ "type": "Program", "body": [ { "type": “VariableDeclaration", "kind": "var"
"declarations": [ { "type": "VariableDeclarator", "id": { "type": "Identifier", "name": "fullstack" }, "init": { "type": "BinaryExpression", "left": { "type": "Identifier", "name": "node" }, "operator": "+", "right": { "type": "Identifier", "name": "ui" } } } ], } ] } -
Node Types SwitchCase (1) Property (1) Literal (1) Identifier (1)
Declaration (3) Expression (14) Statement (18) -
Expression Types • FunctionExpression • MemberExpression • CallExpression • NewExpression
• ConditionalExpression • LogicalExpression • UpdateExpression • AssignmentExpression • BinaryExpression • UnaryExpression • SequenceExpression • ObjectExpression • ArrayExpression • ThisExpression -
Statement Types • DebuggerStatement • ForInStatement • ForStatement • DoWhileStatement
• WhileStatement • CatchClause • TryStatement • ThrowStatement • ReturnStatement • SwitchStatement • WithStatement • ContinueStatement • BreakStatement • LabeledStatement • IfStatement • ExpressionStatement • BlockStatement • EmptyStatement -
• When debugging console.log(ast) will not print a large nested
AST properly. Instead you can use util.inspect: var util = require('util'); var tree = util.inspect(ast, { depth: null }); console.log(tree); • When transforming code start with the AST you want and then work backward. • Often this means pasting code using the Esprima online visualization tool or just outputting the trees into JS files and manually diffing them. -
Oftentimes it helps to print out the code representation of
a single node. In recast you can do: var source = recast.prettyPrint(ast, { tabWidth: 2 }).code; In ESLint you can get the current node with: var source = context.getSource(node) -
Related - http://pegjs.majda.cz/ - https://www.kickstarter.com/projects/michaelficarra/make-a-better-coffeescript-compiler - http://coffeescript.org/documentation/docs/grammar.html - https://github.com/padolsey/parsers-built-with-js -
https://github.com/zaach/jison - https://github.com/substack/node-falafel - eprima-based code modifier - https://github.com/substack/node-burrito - uglify-based AST walking code-modifier - https://github.com/jscs-dev/node-jscs/blob/e745ceb23c5f1587c3e43c0a9cfb05f5ad86b5ac/lib/js-file.js - JSCS’s way of walking the AST - https://www.npmjs.org/package/escodegen - converts an AST into real code again - https://www.npmjs.org/package/ast-types - esprima-ish parser - http://esprima.org/demo/parse.html - the most helpful tool https://github.com/RReverser/estemplate - AST-based search and replace https://www.npmjs.org/package/aster - build system thing Technical Papers http://aosd.net/2013/escodegen.html Videos / Slides http://slidedeck.io/benjamn/fluent2014-talk http://vimeo.com/93749422 https://speakerdeck.com/michaelficarra/spidermonkey-parser-api-a-standard-for-structured-js-representations https://www.youtube.com/watch?v=fF_jZ7ErwUY https://speakerdeck.com/ariya/bringing-javascript-code-analysis-to-the-next-level Just in Time Compilers http://blogs.msdn.com/b/ie/archive/2012/06/13/advances-in-javascript-performance-in-ie10-and-windows-8.aspx https://blog.mozilla.org/luke/2014/01/14/asm-js-aot-compilation-and-startup-performance/ https://blog.indutny.com/4.how-to-start-jitting Podcasts http://javascriptjabber.com/082-jsj-jshint-with-anton-kovalyov/ http://javascriptjabber.com/054-jsj-javascript-parsing-asts-and-language-grammar-w-david-herman-and-ariya-hidayat/ -
Static analysis tools like ESLint and JSCS provide an API
to let you inspect an AST to make sure it’s following certain patterns. -
function isEmptyObject( obj ) { for ( var name in
obj ) { return false; } return true; } -
function loadUser(req, res, next) { User.loadUser(function(err, user) { if (err)
{ next(err); } req.session.user = user; next(); }); } Another Example -
Tools like falafel and recast give you an API to
manipulate an AST and then convert that back into source code.