Unpacking the Black Box: Benchmarking JS Parsing and Execution on Mobile Devices

757c233176d3ad14136df4a547ac4234?s=47 Daniel Espeset
September 17, 2014

Unpacking the Black Box: Benchmarking JS Parsing and Execution on Mobile Devices

Optimizing the experience on mobile devices is a major priority for our industry, but many client performance characteristics are not well understood. In pursuit of the 1000ms time-to-glass holy grail, we wrote a tool to benchmark the initial parse and execute times of our JavaScript files. While high end devices are expectedly fast, midrange and lower end ones are surprisingly slow. We’ll share our full suite of results, the methodology used and some optimization techniques – plus we released the tool used to do these tests: github.com/etsy/DeviceTiming

757c233176d3ad14136df4a547ac4234?s=128

Daniel Espeset

September 17, 2014
Tweet

Transcript

  1. Daniel Espeset @danielespeset Velocity NYC – September 17, 2014 Benchmarking

    JS Parsing and Execution on Mobile Devices Unpacking the Black Box
  2. talks.desp.in/unpacking-the-black-box Resources for this talk

  3. 3 https://www.flickr.com/photos/tiarescott/

  4. Daniel Espeset @danielespeset Frontend Infrastructure 4 Seth Walker Daniel Espeset

    Daniel Na Takashi Mizohata Etsy
  5. Daniel Espeset @danielespeset Performance 5 Lara Hogan Natalya Hoota Jonathan

    Klein Allison McKnight Etsy
  6. Daniel Espeset @danielespeset >50% Mobile Traffic 10 Languages 6

  7. Daniel Espeset @danielespeset 7

  8. 8 Thinking about Mobile Performance

  9. Daniel Espeset @danielespeset 9 1. The Network

  10. Daniel Espeset @danielespeset 10 2. The Browser

  11. Daniel Espeset @danielespeset 11 degraded render performance

  12. Daniel Espeset @danielespeset 12 JavaScript initializing

  13. Daniel Espeset @danielespeset 13 jquery.js

  14. Daniel Espeset @danielespeset 14

  15. Daniel Espeset @danielespeset 15

  16. Daniel Espeset @danielespeset 16 ?

  17. Daniel Espeset @danielespeset 17 parse execute

  18. Daniel Espeset @danielespeset How long does our JavaScript payload take

    to parse and execute, once it hits the browser? 18
  19. Daniel Espeset @danielespeset To the lab! 19

  20. Let’s Add Some Timers!

  21. Adding naive timers to our JS bundle var start =

    new Date().getTime(); // ... bulk of your code var duration = new Date().getTime() - start; 1 - 600 21
  22. Daniel Espeset @danielespeset Results! 271ms for our main shared JS

    bundle 22
  23. Daniel Espeset @danielespeset This is awesome. Do this if you

    don’t already. 23
  24. var start = new Date().getTime(); // ... bulk of your

    code var duration = new Date().getTime() - start; 1 - 600 24 A bunch of work happens before this runs. Adding naive timers to our JS bundle
  25. Daniel Espeset @danielespeset 25 parse execute

  26. Daniel Espeset @danielespeset Next questions 26 1. How long does

    it take to parse? 2. Can we measure this on every device in our lab?
  27. What Are Our Options?

  28. Daniel Espeset @danielespeset Remote Debugging 28

  29. Daniel Espeset @danielespeset 29

  30. Daniel Espeset @danielespeset 30

  31. Daniel Espeset @danielespeset Pure JS Solutions 31

  32. We could wrap timers around a <script> tag <script>var beforeJquery

    = new Date().getTime()</script> <script src=”jquery.js”></script> <script>var took = new Date().getTime() - beforeJquery<script> 20 21 22 32
  33. Daniel Espeset @danielespeset This doesn’t work, it may include the

    network or exclude parse time. 33
  34. Same idea, but in a loop var runs = 1000;

    results = []; while (runs--) { d.write(“<script>var start = Date.now();</script>”); d.write(“<script src=‘jQuery.js’></script>”); d.write(“<script>”); d.write(“results.push(Date.now() - start);”); d.write(“var e = d.getElementById(‘test’);”); d.write(“e.parentNode.removeChild(e);”); d.write(“</script>”); } 22 23 24 25 26 27 28 29 30 31 32 34
  35. Daniel Espeset @danielespeset Carlos Bueno did this in 2010 “Measuring

    Javascript Parse and Load” bit.ly/js-parse-and-load 35
  36. Daniel Espeset @danielespeset Runtime is complicated JIT Optimizations Garbage Collection

    36 bit.ly/js-parse-and-load
  37. Our approach

  38. Step 1. “window scoper” transforms implicit globals to explicit ones.

    var foo = {}; (function() { // ... }); 1 2 - 99 38 window.foo = {}; (function() { // ... }); 1 2 - 99
  39. Step 2. wrap the contents in a named function 39

    window.foo = {}; (function() { // ... }); 1 2 - 99 function sourceCode() { window.foo = {}; (function() { // ... }); } 1 2 3 - 100 101
  40. Step 3. turn the function into a string, eval and

    execute it var start = new Date().getTime(); eval(“function sourceCode() { window.foo = {}; ... ”); var parse = new Date().getTime(); var parseTook = start - parse; sourceCode(); var execTook = new Date().getTime() - parse; 1 2 3 4 5 6 40
  41. Step 3. turn the function into a string, eval and

    execute it var start = new Date().getTime(); eval(“function sourceCode() { window.foo = {}; ... ”); var parse = new Date().getTime(); var parseTook = start - parse; sourceCode(); var execTook = new Date().getTime() - parse; 1 2 3 4 5 6 41
  42. Step 3. turn the function into a string, eval and

    execute it var start = new Date().getTime(); eval(“function sourceCode() { window.foo = {}; ... ”); var parse = new Date().getTime(); var parseTook = start - parse; sourceCode(); var execTook = new Date().getTime() - parse; 1 2 3 4 5 6 42
  43. Daniel Espeset @danielespeset 43 parse execute

  44. Daniel Espeset @danielespeset 44 results for jquery 1.8.2 + almond

    Apple (Macbook Pro) Chrome 31 RAM: 16GB CPU: 2.4GHz GPU: 1058MHz 36ms total
  45. Daniel Espeset @danielespeset 45 results for jquery 1.8.2 + almond

    319ms total Samsung Galaxy S3 Mobile Chrome 31 RAM: 1GB CPU: 1.4GHz GPU: 400MHz
  46. Daniel Espeset @danielespeset 46 results for jquery 1.8.2 + almond

    1389ms total LG Optimus V (VM670) Android Browser 4.0 RAM: 512MB CPU: 600MHz GPU: 128MHz
  47. Daniel Espeset @danielespeset 47 1. How long to eval a

    big function with our code 2. How long to execute it What does this measure?
  48. Daniel Espeset @danielespeset 48 1. Eval is a reasonable proxy

    for <script> 2. This effectively separates parse and exec Assumptions
  49. Add timers to the top and bottom of the big

    wrapper function var start = new Date().getTime(), innerStart, innerEnd; eval(“function sourceCode() { innerStart = new Date().getTime(); window.foo = {}; ... }”); var execStart = new Date().getTime(); sourceCode(); var execEnd = new Date().getTime() - parse; 1 2 3 4 5 6 7 8 49
  50. Add a timer to the top of the big wrapper

    function 50 var start = new Date().getTime(), innerStart, innerEnd; eval(“function sourceCode() { innerStart = new Date().getTime(); window.foo = {}; ... }”); var execStart = new Date().getTime(); sourceCode(); var execEnd = new Date().getTime() - parse; 1 2 3 4 5 6 7 8
  51. (what this looks like at runtime) 51 var execStart =

    new Date().getTime(); (function sourceCode() { innerStart = new Date().getTime(); window.foo = {}; // ... })(); var execEnd = new Date().getTime() - parse; 1 2 3 4 5 6 7 8 The time between these should be ~0ms
  52. (what this looks like at runtime) 52 var execStart =

    new Date().getTime(); (function sourceCode() { innerStart = new Date().getTime(); window.foo = {}; // ... })(); var execEnd = new Date().getTime() - parse; 1 2 3 4 5 6 7 8 Chrome on Samsung Galaxy S3 50–100ms
  53. Daniel Espeset @danielespeset 53 Runtime is complicated JIT Optimizations Garbage

    Collection Chrome on Samsung Galaxy S3 50–100ms
  54. Daniel Espeset @danielespeset 54 parse execute

  55. Step 3 REDUX: turn the function into a string, eval

    it var start = new Date().getTime(); eval(“var parse = new Date().getTime(); !function(){ ... }()”); var execTook = new Date().getTime() - parse; var parseTook = start - parse; 1 2 3 4 55
  56. Parse: the time between start, and the first line of

    the eval running. var start = new Date().getTime(); eval(“var parse = new Date().getTime(); !function(){ ... }()”); var execTook = new Date().getTime() - parse; var parseTook = start - parse; 1 2 3 4 56
  57. Exec: the time between the first line of the eval,

    and the line after var start = new Date().getTime(); eval(“var parse = new Date().getTime(); !function(){ ... }()”); var execTook = new Date().getTime() - parse; var parseTook = start - parse; 1 2 3 4 57
  58. Daniel Espeset @danielespeset DeviceTiming 58

  59. Daniel Espeset @danielespeset 59 DeviceTiming Server

  60. Daniel Espeset @danielespeset 60 DeviceTiming Server Test Devices

  61. demo.desp.in DEMO TIME! The backbone example from todomvc.com

  62. Daniel Espeset @danielespeset Cutting The Mustard Going Vanilla You could

    be blocked on CSS 62 Optimizations
  63. codeascraft.com etsy.com/careers