Dynamic Code Analysis for JavaScript

Presented at jQuery Conference 2014 in San Diego: http://events.jquery.org/2014/san-diego/.

These days, publishing a project without a comprehensive test suite is frown upon. However, the tests themselves do not always tell the full story. We need to level up and provide a more confident level of dynamic code testing. In this talk, two types of such analysis will be covered: code coverage and run-time complexity profiling. Code coverage is instrumental in deciding the quality of the existing sets of unit tests. An important aspect of coverage testing, branch coverage, plays an important role in discovering possible latent bugs due to untested code flow. In addition to code coverage, complexity profiling permits selective instrumentation of a particular code block for the purpose of performance analysis. More than just measuring elapsed time, run-time profiling also needs to reveal the algorithmic complexity which can be tracked via this empirical approach.

Additional details: http://ariya.ofilabs.com/2014/02/the-majestic-jquery-conference-san-diego-2014.html.

Transcript

1. Dynamic Code Analysis for JavaScript @ariyahidayat Feb 12, 2014
2. /usr/bin/whoami shapesecurity.com
3. “Software Provocateur” PhantomJS Esprima
5. “YOU SHALL NOT PASS!” — Darth Vader
6. Utility Belt
7. Parser Code Coverage Code Generator Function Instrumentation +
8. Syntax Parser var answer = 42 keyword equal sign identifier number Variable Declaration Identifier Literal Constant Tokenization → Tokens Parsing → Syntax Tree
9. { type: "Program", body: [ { type: "VariableDeclaration", declarations: [ { type: "VariableDeclarator", id: { type: "Identifier", name: "answer" }, init: { type: "Literal", value: 42, raw: "42" } } ], kind: "var" } ] } Syntax Tree var answer = 42; Terms → ECMAScript 5.1 Specification https://developer.mozilla.org/en-US/docs/SpiderMonkey/Parser_API
10. Syntax Visualization http://esprima.org/demo/parse.html
12. Source Transformation Parser Code Generator In-place Modification Non-Destructive Regenerative http://ariya.ofilabs.com/2013/06/javascript-source-transformation-non-destructive-vs-regenerative.html
13. Code Coverage
14. Regenerative Transformation Parser Code Generator http://ariya.ofilabs.com/2013/06/javascript-source-transformation-non-destructive-vs-regenerative.html
15. Istanbul
16. Code Instrumentation var answer = 42; alert(answer); var __cov_l$4m7m$L464yvav5F$qhNA = __coverage__['hello.js']; __cov_l$4m7m$L464yvav5F$qhNA.s['1']++; var answer = 42; __cov_l$4m7m$L464yvav5F$qhNA.s['2']++; alert(answer);
17. Statement/Expression Coverage http://ariya.ofilabs.com/2012/12/javascript-code-coverage-with-istanbul.html function fibo(n) { return (n < 2) ? n : fibo(n-1) + fibo(n-2); } assert("fibo(1) must give 1", fibo(1) == 1);
18. Branch Coverage function inc(p, q) { if (q == undefined) q = 1; return p + q/q; } assert("inc(4) must give 5", inc(4) == 5); http://ariya.ofilabs.com/2012/12/javascript-code-coverage-with-istanbul.html
19. http://ariya.ofilabs.com/2012/09/the-hidden-trap-of-code-coverage.html Does not catch the missing code sequence
21. Profile-Guided Optimization http://ariya.ofilabs.com/2013/07/profile-guided-javascript-optimization.html function isDigit(ch) { return '0123456789'.indexOf(ch) >= 0; } function isDigit(ch) { return (ch !== ' ') && '0123456789'.indexOf(ch) >= 0; }
22. Istanbul, QUnit, Karma http://ariya.ofilabs.com/2013/10/code-coverage-of-qunit-tests-using-istanbul-and-karma.html test("sqrt", function() { deepEqual(My.sqrt(4), 2, "square root of 4 is 2"); });
23. “If you think JSLint hurts your feelings, wait until you use Istanbul.” @davglass
24. ...I gave you my heart But the very next day you gave it away...
25. Function Instrumentation
26. Non-Destructive Transformation Parser In-place Modification http://ariya.ofilabs.com/2013/06/javascript-source-transformation-non-destructive-vs-regenerative.html
27. String Literal Quotes http://ariya.ofilabs.com/2012/02/from-double-quotes-to-single-quotes.html console.log('Hello') console.log("Hello")
28. Timing Prepare the stopwatch Mark the start and the end Sampling Periodically check which function is being executed Tracing Track all function calls and exits http://ariya.ofilabs.com/2012/12/javascript-performance-analysis-sampling-tracing-and-timing.html
29. Fast = Enough? Alice Bob Chuck Dan ... Bob Alice Dan Chuck ... Sort How’s the speed? 2 ms to sort 10 contacts
30. Array.prototype.swap = function (i, j) { var k = this[i]; this[i] = this[j]; this[j] = k; } function sort(list) { var items = list.slice(0), swapped = false, p, q; for (p = 1; p < items.length; ++p) { for (q = 0; q < items.length - p; ++q) { if (items[q + 1] < items[q]) { items.swap(q, q + 1); swapped =true; } } if (!swapped) break; } return items; } Bubble Sort ???
31. Run-time Complexity http://ariya.ofilabs.com/2012/01/scalable-web-apps-the-complexity-issue.html Array.prototype.swap = function (i, j) { var k = this[i]; this[i] = this[j]; this[j] = k; } Array.prototype.swap = function (i, j) { Log({ name: 'Array.prototype.swap', lineNumber: 1, range: [23, 94] }); var k = this[i]; this[i] = this[j]; this[j] = k; } http://esprima.org/demo/functiontrace.html
33. “First Things First”, Steven Covey “The Big Rock” http://calendar.perfplanet.com/2013/javascript-performance-big-picture/
34. Layers of Defense
35. Being Defensive http://ariya.ofilabs.com/2012/12/quality-code-via-multiple-layers-of-defense.html
36. Being Objective Mr. Reviewer Your code sucks! Your code sucks!
37. Tests Madness T D D B D D
38. Coverage Thresholds http://ariya.ofilabs.com/2013/05/hard-thresholds-on-javascript-code-coverage.html istanbul check-coverage --statement -5 --branch -3 --function 100
39. Tools separate us
40. Final Words Dynamic code analysis for quality: • Code coverage (statement, branch,...) • Function instrumentation Build, write, tweak code analysis tools
41. Thank You @ariyahidayat ariya.ofilabs.com speakerdeck.com/ariya Some artworks are from http://openclipart.org.