Tweet
Tweet

Slide 1

Slide 1 text

Peter Hedenskog @soulislove [email protected] Measuring Web Performance Using Selenium SeleniumConf UK 2016 Peter Hedenskog @soulislove [email protected]

Slide 2

Slide 2 text

https://www.mediawiki.org/wiki/ Wikimedia_Performance_Team

Slide 3

Slide 3 text

Wikipedia =! Wikileaks stupid!

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

“Fika”

Slide 6

Slide 6 text

Tyrell

Slide 7

Slide 7 text

Pippi

Slide 8

Slide 8 text

ABBA

Slide 9

Slide 9 text

ABBA!!

Slide 10

Slide 10 text

ABBA!!!!!!!!!

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

“Do not use Selenium for performance testing”

Slide 13

Slide 13 text

Yeah but you can!

Slide 14

Slide 14 text

Today

Slide 15

Slide 15 text

- Web performance metrics Today

Slide 16

Slide 16 text

- Web performance metrics - Selenium Today

Slide 17

Slide 17 text

- Web performance metrics - Selenium - Performance vs functional testing Today

Slide 18

Slide 18 text

- Web performance metrics - Selenium - Performance vs functional testing - Strategies for catching performance regression Today

Slide 19

Slide 19 text

Load testing?

Slide 20

Slide 20 text

The golden rule of Web Performance

Slide 21

Slide 21 text

“80-90% of the end-user response time is spent on the frontend.”

Slide 22

Slide 22 text

Web Performance Metrics

Slide 23

Slide 23 text

Navigation Timing API

Slide 24

Slide 24 text

Navigation Timing API

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

User Timing API

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

Selenium

Slide 32

Slide 32 text

const script = 'var wp = window.performance.timing;' + 'return wp.domComplete - wp.navigationStart;'; driver.get('http://2016.seleniumconf.co.uk/') .then(() => { return driver.executeScript(script) }) .then((result) => { console.log('DOMComplete: %d ms', result); }) .finally(driver.quit); Javascript

Slide 33

Slide 33 text

NAVIGATION TIMING

Slide 34

Slide 34 text

Navigation timing is a Titanic success!

Slide 35

Slide 35 text

Navigation timings != user experience Navigation timings != user experience

Slide 36

Slide 36 text

Navigation timings != user experience User Timing API != user experience

Slide 37

Slide 37 text

// Chrome if (window.chrome && window.chrome.loadTimes) { // Convert to ms var firstPaint = window.chrome.loadTimes().firstPaintTime*1000; var startTime = window.chrome.loadTimes().startLoadTime*1000; console.log(firstPaint - startTime); } First paint https://github.com/addyosmani/timing.js/blob/ master/timing.js#L47-L56

Slide 38

Slide 38 text

// IE if (typeof window.performance.timing.msFirstPaint === 'number') { var firstPaint = window.performance.timing.msFirstPaint; var startTime = window.performance.timing.navigationStart; console.log(firstPaint - startTime); } https://github.com/addyosmani/timing.js/blob/ master/timing.js#L47-L56 First paint

Slide 39

Slide 39 text

First paint

Slide 40

Slide 40 text

First paint SpeedIndex

Slide 41

Slide 41 text

FFMpeg VisualMetrics SpeedIndex

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

Coming soon

Slide 47

Slide 47 text

firstMeaningfulPaint firstTextPaint firstImagePaint https://yoavweiss.github.io/webperfwg_keynote_velocity_ams/#20

Slide 48

Slide 48 text

Chrome meaningfulPaint chromeOptions.setLoggingPrefs(logPrefs); chromeOptions.setPerfLoggingPrefs({enableNetwork: true, enablePage: true, traceCategories: "blink.user_timing"}); ... case 'Tracing.dataCollected': // Here we can catch things as // firstImagePaint // firstContentfulPaint // firstTextPaint // firstMeaningfulPaint

Slide 49

Slide 49 text

You also want a HAR! Ändra title Something is missing

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

BrowserMob Proxy?

Slide 52

Slide 52 text

Firefox HAR

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

Firefox HAR 'use strict'; const webdriver = require('selenium-webdriver'); const firefox = require('selenium-webdriver/firefox'); const profile = new firefox.Profile(); // HAR export - see http://www.softwareishard.com/blog/har-export-trigger/ profile.setPreference('extensions.netmonitor.har.enableAutomation', true); profile.setPreference('extensions.netmonitor.har.contentAPIToken', 'supersecrettoken'); profile.setPreference('extensions.netmonitor.har.autoConnect', true); profile.setPreference('devtools.netmonitor.har.includeResponseBodies', false); // Download from the version page, the default URL shows wrong latest version // https://addons.mozilla.org/sv-se/firefox/addon/har-export-trigger/versions/? page=1#version-0.5.0-beta.10 profile.addExtension('har_export_trigger-0.5.0-beta.10-fx.xpi'); 1/3

Slide 55

Slide 55 text

const script = ` var callback = arguments[arguments.length - 1]; function triggerExport() { HAR.triggerExport({'token':'supersecrettoken', 'getData':true}) .then((result) => { var har = JSON.parse(result.data); har.log.pages[0].title = document.title; return callback({'har': JSON.stringify(har)}); }) .catch((e) => callback({'error': e})); }; if (typeof HAR === 'undefined') { addEventListener('har-api-ready', triggerExport, false); } else { triggerExport(); }`; 2/3 Firefox HAR

Slide 56

Slide 56 text

driver.get('http://2016.seleniumconf.co.uk/') .then(() => driver.executeAsyncScript(script) ) .then((result) => { if (result.error) { console.error('Could not get the HAR %s', result.error); } else { console.log(result.har); } }) .finally(driver.quit); 3/3 Firefox HAR

Slide 57

Slide 57 text

Chrome HAR

Slide 58

Slide 58 text

Chrome HAR const logPrefs = new webdriver.logging.Preferences(); logPrefs.setLevel(webdriver.logging.Type.PERFORMANCE, webdriver.logging.Level.INFO); const chromeOptions = new chrome.Options(); chromeOptions.setLoggingPrefs(logPrefs); chromeOptions.setPerfLoggingPrefs({enableNetwork: true, enablePage: true});

Slide 59

Slide 59 text

https://github.com/ sitespeedio/ browsertime/blob/ master/lib/support/ chromePerflogParser.js Chrome HAR

Slide 60

Slide 60 text

Performance vs functional testing

Slide 61

Slide 61 text

Ten One commandments of performance testing

Slide 62

Slide 62 text

You shall make sure the metrics are repeatable!

Slide 63

Slide 63 text

1. Setup the browser like a boss

Slide 64

Slide 64 text

Firefox setup const defaultFirefoxPreferences = { 'browser.safebrowsing.enabled': false, 'browser.safebrowsing.malware.enabled': false, 'browser.safebrowsing.remotelookups': false, 'browser.shell.checkDefaultBrowser': false, 'browser.startup.homepage': 'about:blank' // IRL you need more! } const profile = new firefox.Profile(); Object.keys(defaultFirefoxPreferences).forEach(function(pref) { profile.setPreference(pref, defaultFirefoxPreferences[pref]); });

Slide 65

Slide 65 text

const defaultChromeOptions = [ '--disable-background-networking', '--no-default-browser-check', '--disable-translate', '--disable-desktop-notifications', '--disable-save-password-bubble', '--no-default-browser-check' // and more ... ]; const chromeOptions = new chrome.Options(); chromeOptions.addArguments(defaultChromeOptions); Chrome setup

Slide 66

Slide 66 text

2. Use the same connectivity

Slide 67

Slide 67 text

No content

Slide 68

Slide 68 text

Also tc Also tc, dummynet & tylertreat/ comcast

Slide 69

Slide 69 text

Chrome TSProxy // Configure SOCKS proxy, see https://www.chromium.org/ developers/design-documents/network-stack/socks-proxy chromeOptions.addArguments('--proxy-server=socks5:// localhost:1080'); chromeOptions.addArguments('--host-resolver-rules="MAP * ~NOTFOUND , EXCLUDE localhost"');

Slide 70

Slide 70 text

profile.setPreference("network.proxy.socks", "localhost"); profile.setPreference("network.proxy.socks_port", 1080); profile.setPreference("network.proxy.type", 1); Firefox TSProxy

Slide 71

Slide 71 text

tc - simulating 3g sudo tc qdisc add dev eth0 root netem delay 300ms loss 0% rate 1600kbps

Slide 72

Slide 72 text

3. Run many iterations

Slide 73

Slide 73 text

4. Run it alone

Slide 74

Slide 74 text

No content

Slide 75

Slide 75 text

Nooooo!!!!

Slide 76

Slide 76 text

Catch regressions

Slide 77

Slide 77 text

No content

Slide 78

Slide 78 text

No content

Slide 79

Slide 79 text

No content

Slide 80

Slide 80 text

No content

Slide 81

Slide 81 text

CI + RUM + synthetic = love

Slide 82

Slide 82 text

IRL

Slide 83

Slide 83 text

https://grafana.wikimedia.org/dashboard/db/ navigation-timing

Slide 84

Slide 84 text

https://grafana.wikimedia.org/dashboard/db/ navigation-timing

Slide 85

Slide 85 text

https://grafana.wikimedia.org/dashboard/db/ navigation-timing-by-browser

Slide 86

Slide 86 text

https://grafana.wikimedia.org/dashboard/db/ webpagetest

Slide 87

Slide 87 text

But what do you do when you have different stories?

Slide 88

Slide 88 text

Summary

Slide 89

Slide 89 text

Navigation Timing User Timing

Slide 90

Slide 90 text

Navigation Timing User Timing

Slide 91

Slide 91 text

SpeedIndex

Slide 92

Slide 92 text

Before, RUM and synthetic

Slide 93

Slide 93 text

No content

Slide 94

Slide 94 text

https://github.com/soulgalore/seleniumconf16

Slide 95

Slide 95 text

Connectivity: https://www.flickr.com/photos/cogdog/15638928284/ Catch regression: https://www.flickr.com/photos/testlab/21496317363/ Fika: https://www.flickr.com/photos/andreasivarsson/5745405973/ Boris 1: https://www.flickr.com/photos/53797600@N04/6849855824 Broken heart: https://www.flickr.com/photos/miguelpdl/4356975474 Queen: https://www.flickr.com/photos/foreignoffice/8283323803 Ikea: https://www.flickr.com/photos/dahlstroms/4406947248 Apple/Orange: https://www.flickr.com/photos/nubobo/8226113305 Snowflakes: https://www.flickr.com/photos/dan1984/15678152054 Crowd: https://www.flickr.com/photos/vin60/15386141803 Synthetic: https://www.flickr.com/photos/-adam/4674856117 Queen: https://www.flickr.com/photos/aftab/5578122981/ Credits

Slide 96

Slide 96 text

Traffic: https://www.flickr.com/photos/griff69 Safetybelt: https://www.flickr.com/photos/torsteinsaltvedt/2460156876 Free hugs: https://www.flickr.com/photos/clement127/13661779374 Alone: https://www.flickr.com/photos/thomashawk/98867886 Golden: https://www.flickr.com/photos/haeresis07/16191841366 Ten commandments: https://www.flickr.com/photos/vancityscapes/512782411 Angry: https://www.flickr.com/photos/stevendepolo/4605621230 Puzzle: https://www.flickr.com/photos/acefrenzy/8163097 Super heroes: https://www.flickr.com/photos/chuckles396/7145919879 Credits

Slide 97

Slide 97 text

Thank you! @soulislove [email protected]