Unpacking the Black Box: Benchmarking JS Parsing and Execution on Mobile Devices

Optimizing the experience on mobile devices is a major priority for our industry, but many client performance characteristics are not well understood. In pursuit of the 1000ms time-to-glass holy grail, we wrote a tool to benchmark the initial parse and execute times of our JavaScript files. While high end devices are expectedly fast, midrange and lower end ones are surprisingly slow. We’ll share our full suite of results, the methodology used and some optimization techniques – plus we released the tool used to do these tests: github.com/etsy/DeviceTiming

Transcript

1. Daniel Espeset @danielespeset Velocity NYC – September 17, 2014 Benchmarking JS Parsing and Execution on Mobile Devices Unpacking the Black Box
2. talks.desp.in/unpacking-the-black-box Resources for this talk
3. 3 https://www.flickr.com/photos/tiarescott/
4. Daniel Espeset @danielespeset Frontend Infrastructure 4 Seth Walker Daniel Espeset Daniel Na Takashi Mizohata Etsy
5. Daniel Espeset @danielespeset Performance 5 Lara Hogan Natalya Hoota Jonathan Klein Allison McKnight Etsy
6. Daniel Espeset @danielespeset >50% Mobile Traffic 10 Languages 6
7. Daniel Espeset @danielespeset 7
8. 8 Thinking about Mobile Performance
9. Daniel Espeset @danielespeset 9 1. The Network
10. Daniel Espeset @danielespeset 10 2. The Browser
11. Daniel Espeset @danielespeset 11 degraded render performance
12. Daniel Espeset @danielespeset 12 JavaScript initializing
13. Daniel Espeset @danielespeset 13 jquery.js
14. Daniel Espeset @danielespeset 14
15. Daniel Espeset @danielespeset 15
16. Daniel Espeset @danielespeset 16 ?
17. Daniel Espeset @danielespeset 17 parse execute
18. Daniel Espeset @danielespeset How long does our JavaScript payload take to parse and execute, once it hits the browser? 18
19. Daniel Espeset @danielespeset To the lab! 19
20. Let’s Add Some Timers!
21. Adding naive timers to our JS bundle var start = new Date().getTime(); // ... bulk of your code var duration = new Date().getTime() - start; 1 - 600 21
22. Daniel Espeset @danielespeset Results! 271ms for our main shared JS bundle 22
23. Daniel Espeset @danielespeset This is awesome. Do this if you don’t already. 23
24. var start = new Date().getTime(); // ... bulk of your code var duration = new Date().getTime() - start; 1 - 600 24 A bunch of work happens before this runs. Adding naive timers to our JS bundle
25. Daniel Espeset @danielespeset 25 parse execute
26. Daniel Espeset @danielespeset Next questions 26 1. How long does it take to parse? 2. Can we measure this on every device in our lab?
27. What Are Our Options?
28. Daniel Espeset @danielespeset Remote Debugging 28
29. Daniel Espeset @danielespeset 29
30. Daniel Espeset @danielespeset 30
31. Daniel Espeset @danielespeset Pure JS Solutions 31
32. We could wrap timers around a <script> tag <script>var beforeJquery = new Date().getTime()</script> <script src=”jquery.js”></script> <script>var took = new Date().getTime() - beforeJquery<script> 20 21 22 32
33. Daniel Espeset @danielespeset This doesn’t work, it may include the network or exclude parse time. 33
34. Same idea, but in a loop var runs = 1000; results = []; while (runs--) { d.write(“<script>var start = Date.now();</script>”); d.write(“<script src=‘jQuery.js’></script>”); d.write(“<script>”); d.write(“results.push(Date.now() - start);”); d.write(“var e = d.getElementById(‘test’);”); d.write(“e.parentNode.removeChild(e);”); d.write(“</script>”); } 22 23 24 25 26 27 28 29 30 31 32 34
35. Daniel Espeset @danielespeset Carlos Bueno did this in 2010 “Measuring Javascript Parse and Load” bit.ly/js-parse-and-load 35
36. Daniel Espeset @danielespeset Runtime is complicated JIT Optimizations Garbage Collection 36 bit.ly/js-parse-and-load
37. Our approach
38. Step 1. “window scoper” transforms implicit globals to explicit ones. var foo = {}; (function() { // ... }); 1 2 - 99 38 window.foo = {}; (function() { // ... }); 1 2 - 99
39. Step 2. wrap the contents in a named function 39 window.foo = {}; (function() { // ... }); 1 2 - 99 function sourceCode() { window.foo = {}; (function() { // ... }); } 1 2 3 - 100 101
40. Step 3. turn the function into a string, eval and execute it var start = new Date().getTime(); eval(“function sourceCode() { window.foo = {}; ... ”); var parse = new Date().getTime(); var parseTook = start - parse; sourceCode(); var execTook = new Date().getTime() - parse; 1 2 3 4 5 6 40
41. Step 3. turn the function into a string, eval and execute it var start = new Date().getTime(); eval(“function sourceCode() { window.foo = {}; ... ”); var parse = new Date().getTime(); var parseTook = start - parse; sourceCode(); var execTook = new Date().getTime() - parse; 1 2 3 4 5 6 41
42. Step 3. turn the function into a string, eval and execute it var start = new Date().getTime(); eval(“function sourceCode() { window.foo = {}; ... ”); var parse = new Date().getTime(); var parseTook = start - parse; sourceCode(); var execTook = new Date().getTime() - parse; 1 2 3 4 5 6 42
43. Daniel Espeset @danielespeset 43 parse execute
44. Daniel Espeset @danielespeset 44 results for jquery 1.8.2 + almond Apple (Macbook Pro) Chrome 31 RAM: 16GB CPU: 2.4GHz GPU: 1058MHz 36ms total
45. Daniel Espeset @danielespeset 45 results for jquery 1.8.2 + almond 319ms total Samsung Galaxy S3 Mobile Chrome 31 RAM: 1GB CPU: 1.4GHz GPU: 400MHz
46. Daniel Espeset @danielespeset 46 results for jquery 1.8.2 + almond 1389ms total LG Optimus V (VM670) Android Browser 4.0 RAM: 512MB CPU: 600MHz GPU: 128MHz
47. Daniel Espeset @danielespeset 47 1. How long to eval a big function with our code 2. How long to execute it What does this measure?
48. Daniel Espeset @danielespeset 48 1. Eval is a reasonable proxy for <script> 2. This effectively separates parse and exec Assumptions
49. Add timers to the top and bottom of the big wrapper function var start = new Date().getTime(), innerStart, innerEnd; eval(“function sourceCode() { innerStart = new Date().getTime(); window.foo = {}; ... }”); var execStart = new Date().getTime(); sourceCode(); var execEnd = new Date().getTime() - parse; 1 2 3 4 5 6 7 8 49
50. Add a timer to the top of the big wrapper function 50 var start = new Date().getTime(), innerStart, innerEnd; eval(“function sourceCode() { innerStart = new Date().getTime(); window.foo = {}; ... }”); var execStart = new Date().getTime(); sourceCode(); var execEnd = new Date().getTime() - parse; 1 2 3 4 5 6 7 8
51. (what this looks like at runtime) 51 var execStart = new Date().getTime(); (function sourceCode() { innerStart = new Date().getTime(); window.foo = {}; // ... })(); var execEnd = new Date().getTime() - parse; 1 2 3 4 5 6 7 8 The time between these should be ~0ms
52. (what this looks like at runtime) 52 var execStart = new Date().getTime(); (function sourceCode() { innerStart = new Date().getTime(); window.foo = {}; // ... })(); var execEnd = new Date().getTime() - parse; 1 2 3 4 5 6 7 8 Chrome on Samsung Galaxy S3 50–100ms
53. Daniel Espeset @danielespeset 53 Runtime is complicated JIT Optimizations Garbage Collection Chrome on Samsung Galaxy S3 50–100ms
54. Daniel Espeset @danielespeset 54 parse execute
55. Step 3 REDUX: turn the function into a string, eval it var start = new Date().getTime(); eval(“var parse = new Date().getTime(); !function(){ ... }()”); var execTook = new Date().getTime() - parse; var parseTook = start - parse; 1 2 3 4 55
56. Parse: the time between start, and the first line of the eval running. var start = new Date().getTime(); eval(“var parse = new Date().getTime(); !function(){ ... }()”); var execTook = new Date().getTime() - parse; var parseTook = start - parse; 1 2 3 4 56
57. Exec: the time between the first line of the eval, and the line after var start = new Date().getTime(); eval(“var parse = new Date().getTime(); !function(){ ... }()”); var execTook = new Date().getTime() - parse; var parseTook = start - parse; 1 2 3 4 57
58. Daniel Espeset @danielespeset DeviceTiming 58
59. Daniel Espeset @danielespeset 59 DeviceTiming Server
60. Daniel Espeset @danielespeset 60 DeviceTiming Server Test Devices
61. demo.desp.in DEMO TIME! The backbone example from todomvc.com
62. Daniel Espeset @danielespeset Cutting The Mustard Going Vanilla You could be blocked on CSS 62 Optimizations
63. codeascraft.com etsy.com/careers