Unpacking the Black Box: Benchmarking JS Parsing and Execution on Mobile Devices

Optimizing the experience on mobile devices is a major priority for our industry, but many client performance characteristics are not well understood. In pursuit of the 1000ms time-to-glass holy grail, we wrote a tool to benchmark the initial parse and execute times of our JavaScript files. While high end devices are expectedly fast, midrange and lower end ones are surprisingly slow. We’ll share our full suite of results, the methodology used and some optimization techniques – plus we released the tool used to do these tests: github.com/etsy/DeviceTiming

Transcript

1. Daniel Espeset
   @danielespeset
   Velocity NYC – September 17, 2014
   Benchmarking JS Parsing and
   Execution on Mobile Devices
   Unpacking the Black Box
   

   View Slide

2. talks.desp.in/unpacking-the-black-box
   Resources for this talk
   

   View Slide

3. 3
   https://www.flickr.com/photos/tiarescott/
   

   View Slide

4. Daniel Espeset @danielespeset
   Frontend Infrastructure
   4
   Seth Walker Daniel Espeset Daniel Na Takashi Mizohata
   Etsy
   

   View Slide

5. Daniel Espeset @danielespeset
   Performance
   5
   Lara Hogan Natalya Hoota Jonathan Klein Allison McKnight
   Etsy
   

   View Slide

6. Daniel Espeset @danielespeset
   >50%
   Mobile Traffic
   10
   Languages
   6
   

   View Slide

7. Daniel Espeset @danielespeset 7
   

   View Slide

8. 8
   Thinking about Mobile
   Performance
   

   View Slide

9. Daniel Espeset @danielespeset 9
   1. The Network
   

   View Slide

10. Daniel Espeset @danielespeset 10
    2. The Browser
    

    View Slide

11. Daniel Espeset @danielespeset 11
    degraded render performance
    

    View Slide

12. Daniel Espeset @danielespeset 12
    JavaScript initializing
    

    View Slide

13. Daniel Espeset @danielespeset 13
    jquery.js
    

    View Slide

14. Daniel Espeset @danielespeset 14
    

    View Slide

15. Daniel Espeset @danielespeset 15
    

    View Slide

16. Daniel Espeset @danielespeset 16
    ?
    

    View Slide

17. Daniel Espeset @danielespeset 17
    parse execute
    

    View Slide

18. Daniel Espeset @danielespeset
    How long does our JavaScript payload
    take to parse and execute, once it hits
    the browser?
    18
    

    View Slide

19. Daniel Espeset @danielespeset
    To the lab!
    19
    

    View Slide

20. Let’s Add Some Timers!
    

    View Slide

21. Adding naive timers to our JS bundle
    var start = new Date().getTime();
    // ... bulk of your code
    var duration = new Date().getTime() - start;
    1
    -
    600
    21
    

    View Slide

22. Daniel Espeset @danielespeset
    Results!
    271ms for our main
    shared JS bundle
    22
    

    View Slide

23. Daniel Espeset @danielespeset
    This is awesome. Do this if you don’t already.
    23
    

    View Slide

24. var start = new Date().getTime();
    // ... bulk of your code
    var duration = new Date().getTime() - start;
    1
    -
    600
    24
    A bunch of work happens before this runs.
    Adding naive timers to our JS bundle
    

    View Slide

25. Daniel Espeset @danielespeset 25
    parse execute
    

    View Slide

26. Daniel Espeset @danielespeset
    Next questions
    26
    1. How long does it take to parse?
    2. Can we measure this on every device in our lab?
    

    View Slide

27. What Are Our Options?
    

    View Slide

28. Daniel Espeset @danielespeset
    Remote Debugging
    28
    

    View Slide

29. Daniel Espeset @danielespeset 29
    

    View Slide

30. Daniel Espeset @danielespeset 30
    

    View Slide

31. Daniel Espeset @danielespeset
    Pure JS Solutions
    31
    

    View Slide

32. We could wrap timers around a tag<br/><script>var beforeJquery = new Date().getTime()
    
    var took = new Date().getTime() - beforeJquery<script><br/>20<br/>21<br/>22<br/>32<br/>

    View Slide

33. Daniel Espeset @danielespeset
    This doesn’t work, it may include the
    network or exclude parse time.
    33
    

    View Slide

34. Same idea, but in a loop
    var runs = 1000;
    results = [];
    while (runs--) {
    d.write(“var start = Date.now();”);
    d.write(“”);
    d.write(“”);<br/>d.write(“results.push(Date.now() - start);”);<br/>d.write(“var e = d.getElementById(‘test’);”);<br/>d.write(“e.parentNode.removeChild(e);”);<br/>d.write(“”);
    }
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    34
    

    View Slide

35. Daniel Espeset @danielespeset
    Carlos Bueno did this in 2010
    “Measuring Javascript Parse and Load”
    bit.ly/js-parse-and-load
    35
    

    View Slide

36. Daniel Espeset @danielespeset
    Runtime is complicated
    JIT Optimizations
    Garbage Collection
    36
    bit.ly/js-parse-and-load
    

    View Slide

37. Our approach
    

    View Slide

38. Step 1. “window scoper” transforms implicit globals to explicit ones.
    var foo = {};
    (function() {
    // ...
    });
    1
    2
    -
    99
    38
    window.foo = {};
    (function() {
    // ...
    });
    1
    2
    -
    99
    

    View Slide

39. Step 2. wrap the contents in a named function
    39
    window.foo = {};
    (function() {
    // ...
    });
    1
    2
    -
    99
    function sourceCode() {
    window.foo = {};
    (function() {
    // ...
    });
    }
    1
    2
    3
    -
    100
    101
    

    View Slide

40. Step 3. turn the function into a string, eval and execute it
    var start = new Date().getTime();
    eval(“function sourceCode() { window.foo = {}; ... ”);
    var parse = new Date().getTime();
    var parseTook = start - parse;
    sourceCode();
    var execTook = new Date().getTime() - parse;
    1
    2
    3
    4
    5
    6
    40
    

    View Slide

41. Step 3. turn the function into a string, eval and execute it
    var start = new Date().getTime();
    eval(“function sourceCode() { window.foo = {}; ... ”);
    var parse = new Date().getTime();
    var parseTook = start - parse;
    sourceCode();
    var execTook = new Date().getTime() - parse;
    1
    2
    3
    4
    5
    6
    41
    

    View Slide

42. Step 3. turn the function into a string, eval and execute it
    var start = new Date().getTime();
    eval(“function sourceCode() { window.foo = {}; ... ”);
    var parse = new Date().getTime();
    var parseTook = start - parse;
    sourceCode();
    var execTook = new Date().getTime() - parse;
    1
    2
    3
    4
    5
    6
    42
    

    View Slide

43. Daniel Espeset @danielespeset 43
    parse execute
    

    View Slide

44. Daniel Espeset @danielespeset 44
    results for jquery 1.8.2 + almond
    Apple (Macbook Pro)
    Chrome 31
    RAM: 16GB CPU: 2.4GHz GPU: 1058MHz
    36ms total
    

    View Slide

45. Daniel Espeset @danielespeset 45
    results for jquery 1.8.2 + almond
    319ms total
    Samsung Galaxy S3
    Mobile Chrome 31
    RAM: 1GB CPU: 1.4GHz GPU: 400MHz
    

    View Slide

46. Daniel Espeset @danielespeset 46
    results for jquery 1.8.2 + almond
    1389ms total
    LG Optimus V (VM670)
    Android Browser 4.0
    RAM: 512MB CPU: 600MHz GPU: 128MHz
    

    View Slide

47. Daniel Espeset @danielespeset 47
    1. How long to eval a big function with our code
    2. How long to execute it
    What does this measure?
    

    View Slide

48. Daniel Espeset @danielespeset 48
    1. Eval is a reasonable proxy for <br/>2. This effectively separates parse and exec<br/>Assumptions<br/>

    View Slide

49. Add timers to the top and bottom of the big wrapper function
    var start = new Date().getTime(), innerStart, innerEnd;
    eval(“function sourceCode() {
    innerStart = new Date().getTime();
    window.foo = {}; ...
    }”);
    var execStart = new Date().getTime();
    sourceCode();
    var execEnd = new Date().getTime() - parse;
    1
    2
    3
    4
    5
    6
    7
    8
    49
    

    View Slide

50. Add a timer to the top of the big wrapper function
    50
    var start = new Date().getTime(), innerStart, innerEnd;
    eval(“function sourceCode() {
    innerStart = new Date().getTime();
    window.foo = {}; ...
    }”);
    var execStart = new Date().getTime();
    sourceCode();
    var execEnd = new Date().getTime() - parse;
    1
    2
    3
    4
    5
    6
    7
    8
    

    View Slide

51. (what this looks like at runtime)
    51
    var execStart = new Date().getTime();
    (function sourceCode() {
    innerStart = new Date().getTime();
    window.foo = {};
    // ...
    })();
    var execEnd = new Date().getTime() - parse;
    1
    2
    3
    4
    5
    6
    7
    8
    The time between
    these should be
    ~0ms
    

    View Slide

52. (what this looks like at runtime)
    52
    var execStart = new Date().getTime();
    (function sourceCode() {
    innerStart = new Date().getTime();
    window.foo = {};
    // ...
    })();
    var execEnd = new Date().getTime() - parse;
    1
    2
    3
    4
    5
    6
    7
    8
    Chrome on Samsung
    Galaxy S3
    50–100ms
    

    View Slide

53. Daniel Espeset @danielespeset 53
    Runtime is complicated
    JIT Optimizations
    Garbage Collection
    Chrome on Samsung
    Galaxy S3
    50–100ms
    

    View Slide

54. Daniel Espeset @danielespeset 54
    parse execute
    

    View Slide

55. Step 3 REDUX: turn the function into a string, eval it
    var start = new Date().getTime();
    eval(“var parse = new Date().getTime(); !function(){ ... }()”);
    var execTook = new Date().getTime() - parse;
    var parseTook = start - parse;
    1
    2
    3
    4
    55
    

    View Slide

56. Parse: the time between start, and the first line of the eval running.
    var start = new Date().getTime();
    eval(“var parse = new Date().getTime(); !function(){ ... }()”);
    var execTook = new Date().getTime() - parse;
    var parseTook = start - parse;
    1
    2
    3
    4
    56
    

    View Slide

57. Exec: the time between the first line of the eval, and the line after
    var start = new Date().getTime();
    eval(“var parse = new Date().getTime(); !function(){ ... }()”);
    var execTook = new Date().getTime() - parse;
    var parseTook = start - parse;
    1
    2
    3
    4
    57
    

    View Slide

58. Daniel Espeset @danielespeset
    DeviceTiming
    58
    

    View Slide

59. Daniel Espeset @danielespeset 59
    DeviceTiming
    Server
    

    View Slide

60. Daniel Espeset @danielespeset 60
    DeviceTiming
    Server
    Test Devices
    

    View Slide

61. demo.desp.in
    DEMO TIME!
    The backbone example from todomvc.com
    

    View Slide

62. Daniel Espeset @danielespeset
    Cutting The Mustard
    Going Vanilla
    You could be blocked on CSS
    62
    Optimizations
    

    View Slide

63. codeascraft.com
    etsy.com/careers
    

    View Slide