Measuring Web Performance Using Selenium

Measuring Web Performance Using Selenium

You are the Selenium master and automate tests in your sleep. You make sure the quality of your companies web page is 100%. But then the team wants you to measure the performance of your site. How do you do that? Is that even possible with Selenium?

In this talk we will look at how we use Selenium in Browsertime & sitespeed.io to measure the speed of your web application and how you can implement it yourself in your own Selenium tests. We will also briefly look at strategies to make sure you can catch regressions in your sites performance.

You can checkout the talk at Youtube: https://www.youtube.com/watch?v=cLlZhnZvi1M

http://2016.seleniumconf.co.uk/sessions

Dfb236c2d968fbbf6fa4dd7d0541b6b6?s=128

Peter Hedenskog

November 16, 2016
Tweet

Transcript

  1. Peter Hedenskog @soulislove peter@soulgalore.com Measuring Web Performance Using Selenium SeleniumConf

    UK 2016 Peter Hedenskog @soulislove peter@soulgalore.com
  2. https://www.mediawiki.org/wiki/ Wikimedia_Performance_Team

  3. Wikipedia =! Wikileaks stupid!

  4. None
  5. “Fika”

  6. Tyrell

  7. Pippi

  8. ABBA

  9. ABBA!!

  10. ABBA!!!!!!!!!

  11. None
  12. “Do not use Selenium for performance testing”

  13. Yeah but you can!

  14. Today

  15. - Web performance metrics Today

  16. - Web performance metrics - Selenium Today

  17. - Web performance metrics - Selenium - Performance vs functional

    testing Today
  18. - Web performance metrics - Selenium - Performance vs functional

    testing - Strategies for catching performance regression Today
  19. Load testing?

  20. The golden rule of Web Performance

  21. “80-90% of the end-user response time is spent on the

    frontend.”
  22. Web Performance Metrics

  23. Navigation Timing API

  24. Navigation Timing API

  25. None
  26. None
  27. User Timing API

  28. None
  29. None
  30. None
  31. Selenium

  32. const script = 'var wp = window.performance.timing;' + 'return wp.domComplete

    - wp.navigationStart;'; driver.get('http://2016.seleniumconf.co.uk/') .then(() => { return driver.executeScript(script) }) .then((result) => { console.log('DOMComplete: %d ms', result); }) .finally(driver.quit); Javascript
  33. NAVIGATION TIMING

  34. Navigation timing is a Titanic success!

  35. Navigation timings != user experience Navigation timings != user experience

  36. Navigation timings != user experience User Timing API != user

    experience
  37. // Chrome if (window.chrome && window.chrome.loadTimes) { // Convert to

    ms var firstPaint = window.chrome.loadTimes().firstPaintTime*1000; var startTime = window.chrome.loadTimes().startLoadTime*1000; console.log(firstPaint - startTime); } First paint https://github.com/addyosmani/timing.js/blob/ master/timing.js#L47-L56
  38. // IE if (typeof window.performance.timing.msFirstPaint === 'number') { var firstPaint

    = window.performance.timing.msFirstPaint; var startTime = window.performance.timing.navigationStart; console.log(firstPaint - startTime); } https://github.com/addyosmani/timing.js/blob/ master/timing.js#L47-L56 First paint
  39. First paint

  40. First paint SpeedIndex

  41. FFMpeg VisualMetrics SpeedIndex

  42. None
  43. None
  44. None
  45. None
  46. Coming soon

  47. firstMeaningfulPaint firstTextPaint firstImagePaint https://yoavweiss.github.io/webperfwg_keynote_velocity_ams/#20

  48. Chrome meaningfulPaint chromeOptions.setLoggingPrefs(logPrefs); chromeOptions.setPerfLoggingPrefs({enableNetwork: true, enablePage: true, traceCategories: "blink.user_timing"}); ...

    case 'Tracing.dataCollected': // Here we can catch things as // firstImagePaint // firstContentfulPaint // firstTextPaint // firstMeaningfulPaint
  49. You also want a HAR! Ändra title Something is missing

  50. None
  51. BrowserMob Proxy?

  52. Firefox HAR

  53. None
  54. Firefox HAR 'use strict'; const webdriver = require('selenium-webdriver'); const firefox

    = require('selenium-webdriver/firefox'); const profile = new firefox.Profile(); // HAR export - see http://www.softwareishard.com/blog/har-export-trigger/ profile.setPreference('extensions.netmonitor.har.enableAutomation', true); profile.setPreference('extensions.netmonitor.har.contentAPIToken', 'supersecrettoken'); profile.setPreference('extensions.netmonitor.har.autoConnect', true); profile.setPreference('devtools.netmonitor.har.includeResponseBodies', false); // Download from the version page, the default URL shows wrong latest version // https://addons.mozilla.org/sv-se/firefox/addon/har-export-trigger/versions/? page=1#version-0.5.0-beta.10 profile.addExtension('har_export_trigger-0.5.0-beta.10-fx.xpi'); 1/3
  55. const script = ` var callback = arguments[arguments.length - 1];

    function triggerExport() { HAR.triggerExport({'token':'supersecrettoken', 'getData':true}) .then((result) => { var har = JSON.parse(result.data); har.log.pages[0].title = document.title; return callback({'har': JSON.stringify(har)}); }) .catch((e) => callback({'error': e})); }; if (typeof HAR === 'undefined') { addEventListener('har-api-ready', triggerExport, false); } else { triggerExport(); }`; 2/3 Firefox HAR
  56. driver.get('http://2016.seleniumconf.co.uk/') .then(() => driver.executeAsyncScript(script) ) .then((result) => { if (result.error)

    { console.error('Could not get the HAR %s', result.error); } else { console.log(result.har); } }) .finally(driver.quit); 3/3 Firefox HAR
  57. Chrome HAR

  58. Chrome HAR const logPrefs = new webdriver.logging.Preferences(); logPrefs.setLevel(webdriver.logging.Type.PERFORMANCE, webdriver.logging.Level.INFO); const

    chromeOptions = new chrome.Options(); chromeOptions.setLoggingPrefs(logPrefs); chromeOptions.setPerfLoggingPrefs({enableNetwork: true, enablePage: true});
  59. https://github.com/ sitespeedio/ browsertime/blob/ master/lib/support/ chromePerflogParser.js Chrome HAR

  60. Performance vs functional testing

  61. Ten One commandments of performance testing

  62. You shall make sure the metrics are repeatable!

  63. 1. Setup the browser like a boss

  64. Firefox setup const defaultFirefoxPreferences = { 'browser.safebrowsing.enabled': false, 'browser.safebrowsing.malware.enabled': false,

    'browser.safebrowsing.remotelookups': false, 'browser.shell.checkDefaultBrowser': false, 'browser.startup.homepage': 'about:blank' // IRL you need more! } const profile = new firefox.Profile(); Object.keys(defaultFirefoxPreferences).forEach(function(pref) { profile.setPreference(pref, defaultFirefoxPreferences[pref]); });
  65. const defaultChromeOptions = [ '--disable-background-networking', '--no-default-browser-check', '--disable-translate', '--disable-desktop-notifications', '--disable-save-password-bubble', '--no-default-browser-check'

    // and more ... ]; const chromeOptions = new chrome.Options(); chromeOptions.addArguments(defaultChromeOptions); Chrome setup
  66. 2. Use the same connectivity

  67. None
  68. Also tc Also tc, dummynet & tylertreat/ comcast

  69. Chrome TSProxy // Configure SOCKS proxy, see https://www.chromium.org/ developers/design-documents/network-stack/socks-proxy chromeOptions.addArguments('--proxy-server=socks5://

    localhost:1080'); chromeOptions.addArguments('--host-resolver-rules="MAP * ~NOTFOUND , EXCLUDE localhost"');
  70. profile.setPreference("network.proxy.socks", "localhost"); profile.setPreference("network.proxy.socks_port", 1080); profile.setPreference("network.proxy.type", 1); Firefox TSProxy

  71. tc - simulating 3g sudo tc qdisc add dev eth0

    root netem delay 300ms loss 0% rate 1600kbps
  72. 3. Run many iterations

  73. 4. Run it alone

  74. None
  75. Nooooo!!!!

  76. Catch regressions

  77. None
  78. None
  79. None
  80. None
  81. CI + RUM + synthetic = love

  82. IRL

  83. https://grafana.wikimedia.org/dashboard/db/ navigation-timing

  84. https://grafana.wikimedia.org/dashboard/db/ navigation-timing

  85. https://grafana.wikimedia.org/dashboard/db/ navigation-timing-by-browser

  86. https://grafana.wikimedia.org/dashboard/db/ webpagetest

  87. But what do you do when you have different stories?

  88. Summary

  89. Navigation Timing User Timing

  90. Navigation Timing User Timing

  91. SpeedIndex

  92. Before, RUM and synthetic

  93. None
  94. https://github.com/soulgalore/seleniumconf16

  95. Connectivity: https://www.flickr.com/photos/cogdog/15638928284/ Catch regression: https://www.flickr.com/photos/testlab/21496317363/ Fika: https://www.flickr.com/photos/andreasivarsson/5745405973/ Boris 1: https://www.flickr.com/photos/53797600@N04/6849855824

    Broken heart: https://www.flickr.com/photos/miguelpdl/4356975474 Queen: https://www.flickr.com/photos/foreignoffice/8283323803 Ikea: https://www.flickr.com/photos/dahlstroms/4406947248 Apple/Orange: https://www.flickr.com/photos/nubobo/8226113305 Snowflakes: https://www.flickr.com/photos/dan1984/15678152054 Crowd: https://www.flickr.com/photos/vin60/15386141803 Synthetic: https://www.flickr.com/photos/-adam/4674856117 Queen: https://www.flickr.com/photos/aftab/5578122981/ Credits
  96. Traffic: https://www.flickr.com/photos/griff69 Safetybelt: https://www.flickr.com/photos/torsteinsaltvedt/2460156876 Free hugs: https://www.flickr.com/photos/clement127/13661779374 Alone: https://www.flickr.com/photos/thomashawk/98867886 Golden:

    https://www.flickr.com/photos/haeresis07/16191841366 Ten commandments: https://www.flickr.com/photos/vancityscapes/512782411 Angry: https://www.flickr.com/photos/stevendepolo/4605621230 Puzzle: https://www.flickr.com/photos/acefrenzy/8163097 Super heroes: https://www.flickr.com/photos/chuckles396/7145919879 Credits
  97. Thank you! @soulislove peter@soulgalore.com