Tweet
Tweet

Slide 1

Slide 1 text

The Future of JavaScript Language Tooling @ariyahidayat October 6, 2013

Slide 2

Slide 2 text

/usr/bin/whoami

Slide 3

Slide 3 text

Front-end development team Single-page applications

Slide 4

Slide 4 text

Static analysis Inspection Dynamic analysis Transformation

Slide 5

Slide 5 text

Every tool is open-source Tweak/customize/run with it! There are links (everywhere) to detailed articles/blog posts

Slide 6

Slide 6 text

Multi-Layer Defense

Slide 7

Slide 7 text

Defensive = Paranoid http://ariya.ofilabs.com/2012/12/quality-code-via-multiple-layers-of-defense.html

Slide 8

Slide 8 text

Placement of Layers Code Editor CI Server VCS Hooks Smoke Tests Pull Request Build Tasks

Slide 9

Slide 9 text

Feedback is Important Engineer Tools Feedback Cycle Boring Repetitive Time- consuming

Slide 10

Slide 10 text

Track Quality Metrics Application revision Execution time Baseline

Slide 11

Slide 11 text

Metrics Quadrant http://ariya.ofilabs.com/2012/11/optimization-journey-vs-destination.html

Slide 12

Slide 12 text

Be Reasonable “First Things First”, Steven Covey “The Big Rock”

Slide 13

Slide 13 text

Composable Tools

Slide 14

Slide 14 text

This is the Unix Philosophy Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface. Doug McIlroy, the inventor of Unix pipes and one of the founders of the Unix tradition

Slide 15

Slide 15 text

JavaScript in the Browser User Interface Browser Engine Graphics Stack Data Persistence Render Engine JavaScript Engine Networking I/O

Slide 16

Slide 16 text

JavaScript Engine Virtual Machine/ Interpreter Parser Runtime Source Syntax Tree Built-in objects, host objects, ... Fast and conservative

Slide 17

Slide 17 text

Parser var answer = 42 keyword equal sign identifier number Variable Declaration Identifier Literal Constant Tokenization → Tokens Parsing → Syntax Tree

Slide 18

Slide 18 text

{ type: "Program", body: [ { type: "VariableDeclaration", declarations: [ { type: "VariableDeclarator", id: { type: "Identifier", name: "answer" }, init: { type: "Literal", value: 42, raw: "42" } } ], kind: "var" } ] } Syntax Tree Mozilla Parser API var answer = 42; Terms → ECMAScript 5.1 Specification https://developer.mozilla.org/en-US/docs/SpiderMonkey/Parser_API

Slide 19

Slide 19 text

Syntax Visualization http://esprima.org/demo/parse.html Try online! http://ariya.ofilabs.com/2012/04/javascript-syntax-tree-visualization-with-esprima.html

Slide 20

Slide 20 text

Execution Visualization http://int3.github.io/metajs/

Slide 21

Slide 21 text

Building Blocks → Tools Parser Code Generator Source Syntax Tree Source Obvious examples: minifier, obfuscator, ...

Slide 22

Slide 22 text

Another Case: CoffeeScriptRedux JavaScript Parser Code Generator JavaScript Syntax Tree Generated JavaScript JavaScript Source CoffeeScript Source CoffeeScript Parser + Transformer

Slide 23

Slide 23 text

Consistency Convention Scalability Bring Your Own Tool (BYOT)

Slide 24

Slide 24 text

Inspection

Slide 25

Slide 25 text

Syntax Validation http://ariya.ofilabs.com/2012/10/javascript-validator-with-esprima.html http://esprima.org/demo/validate.html Try online! Also available: Grunt plugin, Ant task, ...

Slide 26

Slide 26 text

Strict Mode Validator 'use strict'; block = { color: 'blue', height: 20, width: 10, color: 'red' }; Duplicate data property in object literal not allowed in strict mode http://ariya.ofilabs.com/2012/10/validating-strict-mode.html

Slide 27

Slide 27 text

Linter vs Validator Validator Linter Looks for specification violations Does not care about coding style Works well on generated/minified code Searches for suspicious pattern Warns on style inconsistencies Works well on hand-written code

Slide 28

Slide 28 text

Protection Layers, Again JUnit XML + Jenkins files=$(git diff-index --name- only HEAD | grep -P '\.js$') for file in $files; do esvalidate $file if [ $? -eq 1 ]; then echo "Syntax error: $file" exit 1 fi done Git Precommit Hook http://ariya.ofilabs.com/2012/03/git-pre-commit-hook-and-smoke-testing.html

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

Application Structure MyApp.create('MyApp.Person', { name: 'Joe Sixpack', age: 42, constructor: function(name) {}, walk: function(steps) {} run: function(steps) {} }); { objectName: 'MyApp.Person', functions: ['walk', 'run'], properties: ['name', 'age'] } Metadata

Slide 31

Slide 31 text

Editing Autocomplete http://ariya.ofilabs.com/2013/03/javascript-editing-with-autocomplete.html http://esprima.org/demo/autocomplete.html Try online!

Slide 32

Slide 32 text

Type Annotation http://esprima.org/demo/autocomplete.html Try online!

Slide 33

Slide 33 text

Scope Visualization http://mazurov.github.io/eslevels-demo/

Slide 34

Slide 34 text

Scope and Highlight http://ariya.ofilabs.com/2013/04/javascript-variable-scope-and-highlight.html http://esprima.org/demo/highlight.html Try online!

Slide 35

Slide 35 text

Static Analysis

Slide 36

Slide 36 text

Polluting Variables var height; // some fancy processing heigth = 200; Leaks to global test.js:3 heigth = 200; ^ LeakError: global leak detected: heigth https://github.com/kesla/node-leaky http://ariya.ofilabs.com/2012/11/polluting-and-unused-javascript-variables.html

Slide 37

Slide 37 text

Unused Variables var height; // some fancy processing heigth = 200; http://ariya.ofilabs.com/2012/11/polluting-and-unused-javascript-variables.html Declared but not used test.js height - on line 1 https://github.com/Kami/node-unused

Slide 38

Slide 38 text

Stray Logging function detect_console(code) { function check(node) { if (node.type === 'CallExpression') { if (node.callee.type === 'MemberExpression') { if (node.callee.object.name === 'console') { alert('console call at line', node.loc.start.line); } } } } var tree = esprima.parse(code, { loc: true }); estraverse.traverse(tree, { enter:check }); } var answer = 42; console.log(answer); http://ariya.ofilabs.com/2013/04/automagic-removal-of-javascript-logging.html

Slide 39

Slide 39 text

Nested Ternary Conditionals var str = (age < 1) ? "baby" : (age < 5) ? "toddler" : (age < 18) ? "child": "adult"; http://ariya.ofilabs.com/2012/10/detecting-nested-ternary-conditionals.html

Slide 40

Slide 40 text

“Boolean Trap” Finder Can you make up your mind? treeItem.setState(true, false); event.initKeyEvent("keypress", true, true, null, null, false, false, false, false, 9, 0); The more the merrier? Obfuscated choice var volumeSlider = new Slider(false); Double-negative component.setHidden(false); filter.setCaseInsensitive(false); http://ariya.ofilabs.com/2012/06/detecting-boolean-traps-with-esprima.html

Slide 41

Slide 41 text

Code Complexity http://jscomplexity.org/ if (true) "foo"; else "bar"; Control Flow Graph 6 edges 6 nodes 1 exit Cyclomatic Complexity = 2

Slide 42

Slide 42 text

Most Complex Function http://ariya.ofilabs.com/2013/05/continuous-monitoring-of-javascript-code-complexity.html var cr = require('complexity-report'), content = require('fs').readFileSync('index.js', 'utf-8'), list = []; cr.run(content).functions.forEach(function (entry) { list.push({ name: entry.name, value: entry.complexity.cyclomatic }); }); list.sort(function (x, y) { return y.value - x.value; }); console.log('Most cyclomatic-complex functions:'); list.slice(0, 6).forEach(function (entry) { console.log(' ', entry.name, entry.value); });

Slide 43

Slide 43 text

Continuous Monitoring of Complexity http://ariya.ofilabs.com/2013/05/continuous-monitoring-of-javascript-code-complexity.html

Slide 44

Slide 44 text

Complexity Visualization with Plato https://ariya.ofilabs.com/2013/01/javascript-code-complexity-visualization.html https://github.com/jsoverson/plato

Slide 45

Slide 45 text

Dynamic Analysis

Slide 46

Slide 46 text

Dynamic Code Coverage via Instrumentation var answer = 42; alert(answer); var __cov_l$4m7m$L464yvav5F$qhNA = __coverage__['hello.js']; __cov_l$4m7m$L464yvav5F$qhNA.s['1']++; var answer = 42; __cov_l$4m7m$L464yvav5F$qhNA.s['2']++; alert(answer); http://gotwarlost.github.io/istanbul/

Slide 47

Slide 47 text

Latent Trap of Statement Coverage function inc(p, q) { if (q == undefined) q = 1; return p + q/q; } assert("inc(4) must give 5", inc(4) == 5); Does not catch the missing code sequence http://ariya.ofilabs.com/2012/09/the-hidden-trap-of-code-coverage.html

Slide 48

Slide 48 text

Workaround: Explicit Path function inc(p, q) { if (q == undefined) return p + 1; return p + q/q; } assert("inc(4) must give 5", inc(4) == 5);

Slide 49

Slide 49 text

Branch Coverage http://ariya.ofilabs.com/2012/12/javascript-code-coverage-with-istanbul.html function inc(p, q) { if (q == undefined) q = 1; return p + q/q; } assert("inc(4) must give 5", inc(4) == 5); E = Else is not taken

Slide 50

Slide 50 text

Hard Thresholds on Code Coverage http://ariya.ofilabs.com/2013/05/hard-thresholds-on-javascript-code-coverage.html istanbul check-coverage --statement -5 --branch -3 --function 100

Slide 51

Slide 51 text

Buster.JS buster-istanbul Mocha mocha-istanbul Karma (née Testacular) built-in Intern built-in Grover built-in Istanbul vs Test Libraries

Slide 52

Slide 52 text

Code Coverage for Jasmine Tests Specs Karma Istanbul http://ariya.ofilabs.com/2013/10/code-coverage-of-jasmine-tests-using-istanbul-and-karma.html INFO [karma]: Karma v0.10.2 server started at http://localhost:9876/ INFO [launcher]: Starting browser PhantomJS INFO [PhantomJS 1.9.2 (Linux)]: Connected on socket N9nDnhJ0Np92NTSPGx-X PhantomJS 1.9.2 (Linux): Executed 1 of 1 SUCCESS (0.029 secs / 0.003 secs)

Slide 53

Slide 53 text

If you think JSLint hurts your feelings, wait until you use Istanbul. @davglass

Slide 54

Slide 54 text

...I gave you my heart But the very next day you gave it away...

Slide 55

Slide 55 text

Performance Tweaks

Slide 56

Slide 56 text

http://ariya.ofilabs.com/2012/12/javascript-performance-analysis-sampling-tracing-and-timing.html Timing Prepare the stopwatch Mark the start and the end Sampling Periodically check which function is being executed Tracing Track all function calls and exits

Slide 57

Slide 57 text

Measurement Confidence Accuracy Precision http://ariya.ofilabs.com/2013/06/javascript-timing-accuracy-vs-precision.html

Slide 58

Slide 58 text

What you write Optimized by the engine function test() { var a = 0, b = 0; for (var i = 0; i < 100; i++) { a += i; b += i * i; } return a; } function test() { var a = 0; for (var i = 0; i < 100; i++) { a += i; } return a; } Dead Code Elimination

Slide 59

Slide 59 text

Fast = Enough? Alice Bob Chuck Dan ... Bob Alice Dan Chuck ... Address Book Application Sort How’s the speed? 2 ms to sort 10 contacts

Slide 60

Slide 60 text

Array.prototype.swap = function (i, j) { var k = this[i]; this[i] = this[j]; this[j] = k; } function sort(list) { var items = list.slice(0), swapped = false, p, q; for (p = 1; p < items.length; ++p) { for (q = 0; q < items.length - p; ++q) { if (items[q + 1] < items[q]) { items.swap(q, q + 1); swapped =true; } } if (!swapped) break; } return items; } Bubble Sort ???

Slide 61

Slide 61 text

Profile-Guide Optimization function isDigit(ch) { return '0123456789'.indexOf(ch) >= 0; } function isDigit(ch) { return ch !==' ' && '0123456789'.indexOf(ch) >= 0; } Letters Digits Spaces http://ariya.ofilabs.com/2013/07/profile-guided-javascript-optimization.html

Slide 62

Slide 62 text

No content

Slide 63

Slide 63 text

Object in a Set http://ariya.ofilabs.com/2012/08/determining-objects-in-a-set-examples-in-javascript.html var valid_words = { 'foobar': true, 'bar': true, 'baz': true, 'quux': true }; function is_valid(word) { return valid_words.hasOwnProperty(word); } is_valid('fox'); // false Dictionary Spell checker Alternatives: array lookup, switch case, prefix/suffix tree, perfect hash

Slide 64

Slide 64 text

Tiered Conditionals function is_valid(word) { switch (word.length) { case 3: return word === 'bar' || word === 'baz'; case 4: return word === 'quux'; case 6: return word === 'foobar'; } return false; } Filter #1: Length http://ariya.ofilabs.com/2012/08/determining-objects-in-a-set-examples-in-javascript.html

Slide 65

Slide 65 text

Empirical Run-time Complexity Array.prototype.swap = function (i, j) { var k = this[i]; this[i] = this[j]; this[j] = k; } Array.prototype.swap = function (i, j) { Log({ name: 'Array.prototype.swap', lineNumber: 1, range: [23, 94] }); var k = this[i]; this[i] = this[j]; this[j] = k; } http://esprima.org/demo/functiontrace.html Try online!

Slide 66

Slide 66 text

Run-time Analysis 0 250000 500000 0 500 1000 Calls to swap() Input Size http://ariya.ofilabs.com/2012/01/scalable-web-apps-the-complexity-issue.html

Slide 67

Slide 67 text

Execution Tracing http://ariya.ofilabs.com/2012/02/tracking-javascript-execution-during-startup.html https://gist.github.com/1823129 jQuery Mobile startup log 4640 function calls jquery.js 26 jQuery jquery.js 103 init undefined, undefined, [object Object] jquery.js 274 each (Function) jquery.js 631 each [object Object], (Function), undefined jquery.js 495 isFunction [object Object] jquery.js 512 type [object Object] jquery.mobile.js 1857 [Anonymous] jquery.mobile.js 642 [Anonymous] jquery.mobile.js 624 enableMouseBindings jquery.mobile.js 620 disableTouchBindings

Slide 68

Slide 68 text

Transformation

Slide 69

Slide 69 text

Source Transformation Parser Code Generator Source Syntax Tree Source In-place Modification Modified Source Regenerative Non-Destructive http://ariya.ofilabs.com/2013/06/javascript-source-transformation-non-destructive-vs-regenerative.html

Slide 70

Slide 70 text

String Literal Quotes [ { type: "Identifier", value: "console", range: [0, 7] }, { type: "Punctuator", value: ".", range: [7, 8] }, { type: "Identifier", value: "log", range: [8, 11] }, { type: "Punctuator", value: "(", range: [11, 12] }, { type: "String", value: "\"Hello\"", range: [12, 19] }, { type: "Punctuator", value: ")", range: [19, 19] } ] console.log('Hello') console.log("Hello") List of tokens http://ariya.ofilabs.com/2012/02/from-double-quotes-to-single-quotes.html

Slide 71

Slide 71 text

Style Formatter CodePainter Source Sample code Formatted Infer coding styles Indentation Quote for string literal Whitespace esformatter Source Formatted Style options Indentation Line breaks Whitespaces https://github.com/millermedeiros/esformatter

Slide 72

Slide 72 text

Rename Refactoring Assistant http://ariya.ofilabs.com/2013/04/rename-refactoring-for-javascript-code.html http://esprima.org/demo/rename.html Try online!

Slide 73

Slide 73 text

function f() { var j = data.length; console.log(j, 'items'); for (var i = 0; i < j; ++i) { var j$0 = data[i] * data[i]; console.log(j$0); // squares } } Lexical Block Scope function f() { let j = data.length; console.log(j, 'items'); for (let i = 0; i < j; ++i) { let j = data[i] * data[i]; console.log(j); // squares } } https://github.com/olov/defs New in ECMAScript 6 http://ariya.ofilabs.com/2013/05/es6-and-block-scope.html

Slide 74

Slide 74 text

Parametrized Coding http://nornagon.github.com/scrubby/

Slide 75

Slide 75 text

Embrace the Future

Slide 76

Slide 76 text

Adaptive Tools Explicit Implicit Customize analysis options Define new sets of rules Infer from high-quality sample Observe the engineer’s behavior

Slide 77

Slide 77 text

Tools: The Final Frontier To boldly analyze what no man has analyzed before...

Slide 78

Slide 78 text

Thank You ariya.hidayat@gmail.com @ariyahidayat ariya.ofilabs.com/highlights Credits: Some artworks are from http://openclipart.org speakerdeck.com/ariya