Measuring Web Performance Using Selenium

You are the Selenium master and automate tests in your sleep. You make sure the quality of your companies web page is 100%. But then the team wants you to measure the performance of your site. How do you do that? Is that even possible with Selenium?

In this talk we will look at how we use Selenium in Browsertime & sitespeed.io to measure the speed of your web application and how you can implement it yourself in your own Selenium tests. We will also briefly look at strategies to make sure you can catch regressions in your sites performance.

You can checkout the talk at Youtube: https://www.youtube.com/watch?v=cLlZhnZvi1M

http://2016.seleniumconf.co.uk/sessions

Transcript

1. Peter Hedenskog @soulislove peter@soulgalore.com Measuring Web Performance Using Selenium SeleniumConf UK 2016 Peter Hedenskog @soulislove peter@soulgalore.com
2. https://www.mediawiki.org/wiki/ Wikimedia_Performance_Team
3. Wikipedia =! Wikileaks stupid!
5. “Fika”
6. Tyrell
7. Pippi
8. ABBA
9. ABBA!!
10. ABBA!!!!!!!!!
12. “Do not use Selenium for performance testing”
13. Yeah but you can!
14. Today
15. - Web performance metrics Today
16. - Web performance metrics - Selenium Today
17. - Web performance metrics - Selenium - Performance vs functional testing Today
18. - Web performance metrics - Selenium - Performance vs functional testing - Strategies for catching performance regression Today
19. Load testing?
20. The golden rule of Web Performance
21. “80-90% of the end-user response time is spent on the frontend.”
22. Web Performance Metrics
23. Navigation Timing API
24. Navigation Timing API
27. User Timing API
31. Selenium
32. const script = 'var wp = window.performance.timing;' + 'return wp.domComplete - wp.navigationStart;'; driver.get('http://2016.seleniumconf.co.uk/') .then(() => { return driver.executeScript(script) }) .then((result) => { console.log('DOMComplete: %d ms', result); }) .finally(driver.quit); Javascript
33. NAVIGATION TIMING
34. Navigation timing is a Titanic success!
35. Navigation timings != user experience Navigation timings != user experience
36. Navigation timings != user experience User Timing API != user experience
37. // Chrome if (window.chrome && window.chrome.loadTimes) { // Convert to ms var firstPaint = window.chrome.loadTimes().firstPaintTime*1000; var startTime = window.chrome.loadTimes().startLoadTime*1000; console.log(firstPaint - startTime); } First paint https://github.com/addyosmani/timing.js/blob/ master/timing.js#L47-L56
38. // IE if (typeof window.performance.timing.msFirstPaint === 'number') { var firstPaint = window.performance.timing.msFirstPaint; var startTime = window.performance.timing.navigationStart; console.log(firstPaint - startTime); } https://github.com/addyosmani/timing.js/blob/ master/timing.js#L47-L56 First paint
39. First paint
40. First paint SpeedIndex
41. FFMpeg VisualMetrics SpeedIndex
46. Coming soon
47. firstMeaningfulPaint firstTextPaint firstImagePaint https://yoavweiss.github.io/webperfwg_keynote_velocity_ams/#20
48. Chrome meaningfulPaint chromeOptions.setLoggingPrefs(logPrefs); chromeOptions.setPerfLoggingPrefs({enableNetwork: true, enablePage: true, traceCategories: "blink.user_timing"}); ... case 'Tracing.dataCollected': // Here we can catch things as // firstImagePaint // firstContentfulPaint // firstTextPaint // firstMeaningfulPaint
49. You also want a HAR! Ändra title Something is missing
51. BrowserMob Proxy?
52. Firefox HAR
54. Firefox HAR 'use strict'; const webdriver = require('selenium-webdriver'); const firefox = require('selenium-webdriver/firefox'); const profile = new firefox.Profile(); // HAR export - see http://www.softwareishard.com/blog/har-export-trigger/ profile.setPreference('extensions.netmonitor.har.enableAutomation', true); profile.setPreference('extensions.netmonitor.har.contentAPIToken', 'supersecrettoken'); profile.setPreference('extensions.netmonitor.har.autoConnect', true); profile.setPreference('devtools.netmonitor.har.includeResponseBodies', false); // Download from the version page, the default URL shows wrong latest version // https://addons.mozilla.org/sv-se/firefox/addon/har-export-trigger/versions/? page=1#version-0.5.0-beta.10 profile.addExtension('har_export_trigger-0.5.0-beta.10-fx.xpi'); 1/3
55. const script = ` var callback = arguments[arguments.length - 1]; function triggerExport() { HAR.triggerExport({'token':'supersecrettoken', 'getData':true}) .then((result) => { var har = JSON.parse(result.data); har.log.pages[0].title = document.title; return callback({'har': JSON.stringify(har)}); }) .catch((e) => callback({'error': e})); }; if (typeof HAR === 'undefined') { addEventListener('har-api-ready', triggerExport, false); } else { triggerExport(); }`; 2/3 Firefox HAR
56. driver.get('http://2016.seleniumconf.co.uk/') .then(() => driver.executeAsyncScript(script) ) .then((result) => { if (result.error) { console.error('Could not get the HAR %s', result.error); } else { console.log(result.har); } }) .finally(driver.quit); 3/3 Firefox HAR
57. Chrome HAR
58. Chrome HAR const logPrefs = new webdriver.logging.Preferences(); logPrefs.setLevel(webdriver.logging.Type.PERFORMANCE, webdriver.logging.Level.INFO); const chromeOptions = new chrome.Options(); chromeOptions.setLoggingPrefs(logPrefs); chromeOptions.setPerfLoggingPrefs({enableNetwork: true, enablePage: true});
59. https://github.com/ sitespeedio/ browsertime/blob/ master/lib/support/ chromePerflogParser.js Chrome HAR
60. Performance vs functional testing
61. Ten One commandments of performance testing
62. You shall make sure the metrics are repeatable!
63. 1. Setup the browser like a boss
64. Firefox setup const defaultFirefoxPreferences = { 'browser.safebrowsing.enabled': false, 'browser.safebrowsing.malware.enabled': false, 'browser.safebrowsing.remotelookups': false, 'browser.shell.checkDefaultBrowser': false, 'browser.startup.homepage': 'about:blank' // IRL you need more! } const profile = new firefox.Profile(); Object.keys(defaultFirefoxPreferences).forEach(function(pref) { profile.setPreference(pref, defaultFirefoxPreferences[pref]); });
65. const defaultChromeOptions = [ '--disable-background-networking', '--no-default-browser-check', '--disable-translate', '--disable-desktop-notifications', '--disable-save-password-bubble', '--no-default-browser-check' // and more ... ]; const chromeOptions = new chrome.Options(); chromeOptions.addArguments(defaultChromeOptions); Chrome setup
66. 2. Use the same connectivity
68. Also tc Also tc, dummynet & tylertreat/ comcast
69. Chrome TSProxy // Configure SOCKS proxy, see https://www.chromium.org/ developers/design-documents/network-stack/socks-proxy chromeOptions.addArguments('--proxy-server=socks5:// localhost:1080'); chromeOptions.addArguments('--host-resolver-rules="MAP * ~NOTFOUND , EXCLUDE localhost"');
70. profile.setPreference("network.proxy.socks", "localhost"); profile.setPreference("network.proxy.socks_port", 1080); profile.setPreference("network.proxy.type", 1); Firefox TSProxy
71. tc - simulating 3g sudo tc qdisc add dev eth0 root netem delay 300ms loss 0% rate 1600kbps
72. 3. Run many iterations
73. 4. Run it alone
75. Nooooo!!!!
76. Catch regressions
81. CI + RUM + synthetic = love
82. IRL
83. https://grafana.wikimedia.org/dashboard/db/ navigation-timing
84. https://grafana.wikimedia.org/dashboard/db/ navigation-timing
85. https://grafana.wikimedia.org/dashboard/db/ navigation-timing-by-browser
86. https://grafana.wikimedia.org/dashboard/db/ webpagetest
87. But what do you do when you have different stories?
88. Summary
89. Navigation Timing User Timing
90. Navigation Timing User Timing
91. SpeedIndex
92. Before, RUM and synthetic
94. https://github.com/soulgalore/seleniumconf16
95. Connectivity: https://www.flickr.com/photos/cogdog/15638928284/ Catch regression: https://www.flickr.com/photos/testlab/21496317363/ Fika: https://www.flickr.com/photos/andreasivarsson/5745405973/ Boris 1: https://www.flickr.com/photos/53797600@N04/6849855824 Broken heart: https://www.flickr.com/photos/miguelpdl/4356975474 Queen: https://www.flickr.com/photos/foreignoffice/8283323803 Ikea: https://www.flickr.com/photos/dahlstroms/4406947248 Apple/Orange: https://www.flickr.com/photos/nubobo/8226113305 Snowflakes: https://www.flickr.com/photos/dan1984/15678152054 Crowd: https://www.flickr.com/photos/vin60/15386141803 Synthetic: https://www.flickr.com/photos/-adam/4674856117 Queen: https://www.flickr.com/photos/aftab/5578122981/ Credits
96. Traffic: https://www.flickr.com/photos/griff69 Safetybelt: https://www.flickr.com/photos/torsteinsaltvedt/2460156876 Free hugs: https://www.flickr.com/photos/clement127/13661779374 Alone: https://www.flickr.com/photos/thomashawk/98867886 Golden: https://www.flickr.com/photos/haeresis07/16191841366 Ten commandments: https://www.flickr.com/photos/vancityscapes/512782411 Angry: https://www.flickr.com/photos/stevendepolo/4605621230 Puzzle: https://www.flickr.com/photos/acefrenzy/8163097 Super heroes: https://www.flickr.com/photos/chuckles396/7145919879 Credits
97. Thank you! @soulislove peter@soulgalore.com