Dynamic Code Analysis for JavaScript

Presented at jQuery Conference 2014 in San Diego: http://events.jquery.org/2014/san-diego/.

These days, publishing a project without a comprehensive test suite is frown upon. However, the tests themselves do not always tell the full story. We need to level up and provide a more confident level of dynamic code testing. In this talk, two types of such analysis will be covered: code coverage and run-time complexity profiling. Code coverage is instrumental in deciding the quality of the existing sets of unit tests. An important aspect of coverage testing, branch coverage, plays an important role in discovering possible latent bugs due to untested code flow. In addition to code coverage, complexity profiling permits selective instrumentation of a particular code block for the purpose of performance analysis. More than just measuring elapsed time, run-time profiling also needs to reveal the algorithmic complexity which can be tracked via this empirical approach.

Additional details: http://ariya.ofilabs.com/2014/02/the-majestic-jquery-conference-san-diego-2014.html.

Transcript

1. Dynamic Code Analysis
   for JavaScript
   @ariyahidayat
   Feb 12, 2014
   

   View Slide

2. /usr/bin/whoami
   shapesecurity.com
   

   View Slide

3. “Software Provocateur”
   PhantomJS Esprima
   

   View Slide

4. View Slide

5. “YOU SHALL NOT PASS!”
   — Darth Vader
   

   View Slide

6. Utility Belt
   

   View Slide

7. Parser
   Code
   Coverage
   Code
   Generator
   Function
   Instrumentation
   +
   

   View Slide

8. Syntax Parser
   var answer = 42
   keyword equal sign
   identifier number
   Variable
   Declaration
   Identifier Literal
   Constant
   Tokenization → Tokens
   Parsing → Syntax Tree
   

   View Slide

9. {
   type: "Program",
   body: [
   {
   type: "VariableDeclaration",
   declarations: [
   {
   type: "VariableDeclarator",
   id: {
   type: "Identifier",
   name: "answer"
   },
   init: {
   type: "Literal",
   value: 42,
   raw: "42"
   }
   }
   ],
   kind: "var"
   }
   ]
   }
   Syntax Tree
   var answer = 42;
   Terms → ECMAScript 5.1 Specification
   https://developer.mozilla.org/en-US/docs/SpiderMonkey/Parser_API
   

   View Slide

10. Syntax Visualization
    http://esprima.org/demo/parse.html
    

    View Slide

11. View Slide

12. Source Transformation
    Parser
    Code
    Generator
    In-place
    Modification
    Non-Destructive Regenerative
    http://ariya.ofilabs.com/2013/06/javascript-source-transformation-non-destructive-vs-regenerative.html
    

    View Slide

13. Code Coverage
    

    View Slide

14. Regenerative Transformation
    Parser
    Code
    Generator
    http://ariya.ofilabs.com/2013/06/javascript-source-transformation-non-destructive-vs-regenerative.html
    

    View Slide

15. Istanbul
    

    View Slide

16. Code Instrumentation
    var answer = 42;
    alert(answer);
    var __cov_l$4m7m$L464yvav5F$qhNA = __coverage__['hello.js'];
    __cov_l$4m7m$L464yvav5F$qhNA.s['1']++;
    var answer = 42;
    __cov_l$4m7m$L464yvav5F$qhNA.s['2']++;
    alert(answer);
    

    View Slide

17. Statement/Expression Coverage
    http://ariya.ofilabs.com/2012/12/javascript-code-coverage-with-istanbul.html
    function fibo(n) {
    return (n < 2) ? n : fibo(n-1) + fibo(n-2);
    }
    assert("fibo(1) must give 1", fibo(1) == 1);
    

    View Slide

18. Branch Coverage
    function inc(p, q) {
    if (q == undefined) q = 1;
    return p + q/q;
    }
    assert("inc(4) must give 5", inc(4) == 5);
    http://ariya.ofilabs.com/2012/12/javascript-code-coverage-with-istanbul.html
    

    View Slide

19. http://ariya.ofilabs.com/2012/09/the-hidden-trap-of-code-coverage.html
    Does not catch the missing
    code sequence
    

    View Slide

20. View Slide

21. Profile-Guided Optimization
    http://ariya.ofilabs.com/2013/07/profile-guided-javascript-optimization.html
    function isDigit(ch) {
    return '0123456789'.indexOf(ch) >= 0;
    }
    function isDigit(ch) {
    return (ch !== ' ') && '0123456789'.indexOf(ch) >= 0;
    }
    

    View Slide

22. Istanbul, QUnit, Karma
    http://ariya.ofilabs.com/2013/10/code-coverage-of-qunit-tests-using-istanbul-and-karma.html
    test("sqrt", function() {
    deepEqual(My.sqrt(4), 2, "square root of 4 is 2");
    });
    

    View Slide

23. “If you think JSLint hurts
    your feelings, wait until you
    use Istanbul.”
    @davglass
    

    View Slide

24. ...I gave you my heart
    But the very next day you gave it away...
    

    View Slide

25. Function Instrumentation
    

    View Slide

26. Non-Destructive Transformation
    Parser
    In-place
    Modification
    http://ariya.ofilabs.com/2013/06/javascript-source-transformation-non-destructive-vs-regenerative.html
    

    View Slide

27. String Literal Quotes
    http://ariya.ofilabs.com/2012/02/from-double-quotes-to-single-quotes.html
    console.log('Hello')
    console.log("Hello")
    

    View Slide

28. Timing
    Prepare the stopwatch
    Mark the start and the end
    Sampling
    Periodically check which function
    is being executed
    Tracing Track all function calls and exits
    http://ariya.ofilabs.com/2012/12/javascript-performance-analysis-sampling-tracing-and-timing.html
    

    View Slide

29. Fast = Enough?
    Alice
    Bob
    Chuck
    Dan
    ...
    Bob
    Alice
    Dan
    Chuck
    ...
    Sort
    How’s the speed?
    2 ms to sort 10
    contacts
    

    View Slide

30. Array.prototype.swap = function (i, j) {
    var k = this[i]; this[i] = this[j]; this[j] = k;
    }
    function sort(list) {
    var items = list.slice(0), swapped = false, p, q;
    for (p = 1; p < items.length; ++p) {
    for (q = 0; q < items.length - p; ++q) {
    if (items[q + 1] < items[q]) {
    items.swap(q, q + 1);
    swapped =true;
    }
    }
    if (!swapped) break;
    }
    return items;
    }
    Bubble Sort
    ???
    

    View Slide

31. Run-time Complexity
    http://ariya.ofilabs.com/2012/01/scalable-web-apps-the-complexity-issue.html
    Array.prototype.swap = function (i, j) {
    var k = this[i]; this[i] = this[j]; this[j] = k;
    }
    Array.prototype.swap = function (i, j) {
    Log({ name: 'Array.prototype.swap', lineNumber: 1, range: [23, 94] });
    var k = this[i]; this[i] = this[j]; this[j] = k;
    }
    http://esprima.org/demo/functiontrace.html
    

    View Slide

32. View Slide

33. “First Things First”, Steven Covey
    “The Big Rock”
    http://calendar.perfplanet.com/2013/javascript-performance-big-picture/
    

    View Slide

34. Layers of Defense
    

    View Slide

35. Being Defensive
    http://ariya.ofilabs.com/2012/12/quality-code-via-multiple-layers-of-defense.html
    

    View Slide

36. Being Objective
    Mr. Reviewer
    Your code sucks!
    Your code sucks!
    

    View Slide

37. Tests Madness
    T D D B D D
    

    View Slide

38. Coverage Thresholds
    http://ariya.ofilabs.com/2013/05/hard-thresholds-on-javascript-code-coverage.html
    istanbul check-coverage --statement -5 --branch -3 --function 100
    

    View Slide

39. Tools separate us
    

    View Slide

40. Final Words
    Dynamic code analysis for quality:
    ● Code coverage (statement, branch,...)
    ● Function instrumentation
    Build, write, tweak code analysis tools
    

    View Slide

41. Thank You
    @ariyahidayat
    ariya.ofilabs.com
    speakerdeck.com/ariya
    Some artworks are from http://openclipart.org.
    

    View Slide