$30 off During Our Annual Pro Sale. View Details »

Measuring Web Performance Using Selenium

Measuring Web Performance Using Selenium

You are the Selenium master and automate tests in your sleep. You make sure the quality of your companies web page is 100%. But then the team wants you to measure the performance of your site. How do you do that? Is that even possible with Selenium?

In this talk we will look at how we use Selenium in Browsertime & sitespeed.io to measure the speed of your web application and how you can implement it yourself in your own Selenium tests. We will also briefly look at strategies to make sure you can catch regressions in your sites performance.

You can checkout the talk at Youtube: https://www.youtube.com/watch?v=cLlZhnZvi1M

http://2016.seleniumconf.co.uk/sessions

Peter Hedenskog

November 16, 2016
Tweet

More Decks by Peter Hedenskog

Other Decks in Technology

Transcript

  1. Peter Hedenskog
    @soulislove
    [email protected]
    Measuring Web Performance
    Using Selenium
    SeleniumConf UK 2016
    Peter Hedenskog
    @soulislove
    [email protected]

    View Slide

  2. https://www.mediawiki.org/wiki/
    Wikimedia_Performance_Team

    View Slide

  3. Wikipedia
    =! Wikileaks
    stupid!

    View Slide

  4. View Slide

  5. “Fika”

    View Slide

  6. Tyrell

    View Slide

  7. Pippi

    View Slide

  8. ABBA

    View Slide

  9. ABBA!!

    View Slide

  10. ABBA!!!!!!!!!

    View Slide

  11. View Slide

  12. “Do not use Selenium for
    performance testing”

    View Slide

  13. Yeah but you can!

    View Slide

  14. Today

    View Slide

  15. - Web performance metrics
    Today

    View Slide

  16. - Web performance metrics
    - Selenium
    Today

    View Slide

  17. - Web performance metrics
    - Selenium
    - Performance vs functional testing
    Today

    View Slide

  18. - Web performance metrics
    - Selenium
    - Performance vs functional testing
    - Strategies for catching performance
    regression
    Today

    View Slide

  19. Load
    testing?

    View Slide

  20. The golden
    rule of Web
    Performance

    View Slide

  21. “80-90% of the end-user response
    time is spent on the frontend.”

    View Slide

  22. Web Performance
    Metrics

    View Slide

  23. Navigation
    Timing API

    View Slide

  24. Navigation Timing API

    View Slide

  25. View Slide

  26. View Slide

  27. User Timing API

    View Slide

  28. View Slide

  29. View Slide

  30. View Slide

  31. Selenium

    View Slide

  32. const script = 'var wp = window.performance.timing;' +
    'return wp.domComplete - wp.navigationStart;';
    driver.get('http://2016.seleniumconf.co.uk/')
    .then(() => {
    return driver.executeScript(script)
    })
    .then((result) => {
    console.log('DOMComplete: %d ms', result);
    })
    .finally(driver.quit);
    Javascript

    View Slide

  33. NAVIGATION TIMING

    View Slide

  34. Navigation
    timing is a Titanic
    success!

    View Slide

  35. Navigation
    timings != user
    experience
    Navigation
    timings != user
    experience

    View Slide

  36. Navigation
    timings != user
    experience
    User Timing
    API != user
    experience

    View Slide

  37. // Chrome
    if (window.chrome && window.chrome.loadTimes) {
    // Convert to ms
    var firstPaint = window.chrome.loadTimes().firstPaintTime*1000;
    var startTime = window.chrome.loadTimes().startLoadTime*1000;
    console.log(firstPaint - startTime);
    }
    First paint
    https://github.com/addyosmani/timing.js/blob/
    master/timing.js#L47-L56

    View Slide

  38. // IE
    if (typeof window.performance.timing.msFirstPaint === 'number') {
    var firstPaint = window.performance.timing.msFirstPaint;
    var startTime = window.performance.timing.navigationStart;
    console.log(firstPaint - startTime);
    }
    https://github.com/addyosmani/timing.js/blob/
    master/timing.js#L47-L56
    First paint

    View Slide

  39. First paint

    View Slide

  40. First paint
    SpeedIndex

    View Slide

  41. FFMpeg
    VisualMetrics
    SpeedIndex

    View Slide

  42. View Slide

  43. View Slide

  44. View Slide

  45. View Slide

  46. Coming soon

    View Slide

  47. firstMeaningfulPaint
    firstTextPaint
    firstImagePaint
    https://yoavweiss.github.io/webperfwg_keynote_velocity_ams/#20

    View Slide

  48. Chrome meaningfulPaint
    chromeOptions.setLoggingPrefs(logPrefs);
    chromeOptions.setPerfLoggingPrefs({enableNetwork: true,
    enablePage: true, traceCategories: "blink.user_timing"});
    ...
    case 'Tracing.dataCollected':
    // Here we can catch things as
    // firstImagePaint
    // firstContentfulPaint
    // firstTextPaint
    // firstMeaningfulPaint

    View Slide

  49. You also want a
    HAR! Ändra title
    Something is missing

    View Slide

  50. View Slide

  51. BrowserMob
    Proxy?

    View Slide

  52. Firefox HAR

    View Slide

  53. View Slide

  54. Firefox HAR
    'use strict';
    const webdriver = require('selenium-webdriver');
    const firefox = require('selenium-webdriver/firefox');
    const profile = new firefox.Profile();
    // HAR export - see http://www.softwareishard.com/blog/har-export-trigger/
    profile.setPreference('extensions.netmonitor.har.enableAutomation', true);
    profile.setPreference('extensions.netmonitor.har.contentAPIToken',
    'supersecrettoken');
    profile.setPreference('extensions.netmonitor.har.autoConnect', true);
    profile.setPreference('devtools.netmonitor.har.includeResponseBodies', false);
    // Download from the version page, the default URL shows wrong latest version
    // https://addons.mozilla.org/sv-se/firefox/addon/har-export-trigger/versions/?
    page=1#version-0.5.0-beta.10
    profile.addExtension('har_export_trigger-0.5.0-beta.10-fx.xpi');
    1/3

    View Slide

  55. const script = `
    var callback = arguments[arguments.length - 1];
    function triggerExport() {
    HAR.triggerExport({'token':'supersecrettoken', 'getData':true})
    .then((result) => {
    var har = JSON.parse(result.data);
    har.log.pages[0].title = document.title;
    return callback({'har': JSON.stringify(har)});
    })
    .catch((e) => callback({'error': e}));
    };
    if (typeof HAR === 'undefined') {
    addEventListener('har-api-ready', triggerExport, false);
    } else {
    triggerExport();
    }`;
    2/3
    Firefox HAR

    View Slide

  56. driver.get('http://2016.seleniumconf.co.uk/')
    .then(() =>
    driver.executeAsyncScript(script)
    )
    .then((result) => {
    if (result.error) {
    console.error('Could not get the HAR %s', result.error);
    } else {
    console.log(result.har);
    }
    })
    .finally(driver.quit);
    3/3
    Firefox HAR

    View Slide

  57. Chrome HAR

    View Slide

  58. Chrome HAR
    const logPrefs = new webdriver.logging.Preferences();
    logPrefs.setLevel(webdriver.logging.Type.PERFORMANCE,
    webdriver.logging.Level.INFO);
    const chromeOptions = new chrome.Options();
    chromeOptions.setLoggingPrefs(logPrefs);
    chromeOptions.setPerfLoggingPrefs({enableNetwork: true,
    enablePage: true});

    View Slide

  59. https://github.com/
    sitespeedio/
    browsertime/blob/
    master/lib/support/
    chromePerflogParser.js
    Chrome HAR

    View Slide

  60. Performance vs
    functional
    testing

    View Slide

  61. Ten One
    commandments
    of performance
    testing

    View Slide

  62. You shall make
    sure the metrics
    are repeatable!

    View Slide

  63. 1. Setup the
    browser like a
    boss

    View Slide

  64. Firefox setup
    const defaultFirefoxPreferences = {
    'browser.safebrowsing.enabled': false,
    'browser.safebrowsing.malware.enabled': false,
    'browser.safebrowsing.remotelookups': false,
    'browser.shell.checkDefaultBrowser': false,
    'browser.startup.homepage': 'about:blank'
    // IRL you need more!
    }
    const profile = new firefox.Profile();
    Object.keys(defaultFirefoxPreferences).forEach(function(pref)
    {
    profile.setPreference(pref, defaultFirefoxPreferences[pref]);
    });

    View Slide

  65. const defaultChromeOptions = [
    '--disable-background-networking',
    '--no-default-browser-check',
    '--disable-translate',
    '--disable-desktop-notifications',
    '--disable-save-password-bubble',
    '--no-default-browser-check'
    // and more ...
    ];
    const chromeOptions = new chrome.Options();
    chromeOptions.addArguments(defaultChromeOptions);
    Chrome setup

    View Slide

  66. 2. Use the same
    connectivity

    View Slide

  67. View Slide

  68. Also tc
    Also tc,
    dummynet &
    tylertreat/
    comcast

    View Slide

  69. Chrome TSProxy
    // Configure SOCKS proxy, see https://www.chromium.org/
    developers/design-documents/network-stack/socks-proxy
    chromeOptions.addArguments('--proxy-server=socks5://
    localhost:1080');
    chromeOptions.addArguments('--host-resolver-rules="MAP
    * ~NOTFOUND , EXCLUDE localhost"');

    View Slide

  70. profile.setPreference("network.proxy.socks", "localhost");
    profile.setPreference("network.proxy.socks_port", 1080);
    profile.setPreference("network.proxy.type", 1);
    Firefox TSProxy

    View Slide

  71. tc - simulating 3g
    sudo tc qdisc add dev eth0 root netem
    delay 300ms loss 0% rate 1600kbps

    View Slide

  72. 3. Run many
    iterations

    View Slide

  73. 4. Run it alone

    View Slide

  74. View Slide

  75. Nooooo!!!!

    View Slide

  76. Catch
    regressions

    View Slide

  77. View Slide

  78. View Slide

  79. View Slide

  80. View Slide

  81. CI + RUM +
    synthetic = love

    View Slide

  82. IRL

    View Slide

  83. https://grafana.wikimedia.org/dashboard/db/
    navigation-timing

    View Slide

  84. https://grafana.wikimedia.org/dashboard/db/
    navigation-timing

    View Slide

  85. https://grafana.wikimedia.org/dashboard/db/
    navigation-timing-by-browser

    View Slide

  86. https://grafana.wikimedia.org/dashboard/db/
    webpagetest

    View Slide

  87. But what do you
    do when you
    have different
    stories?

    View Slide

  88. Summary

    View Slide

  89. Navigation
    Timing
    User Timing

    View Slide

  90. Navigation Timing
    User Timing

    View Slide

  91. SpeedIndex

    View Slide

  92. Before, RUM and
    synthetic

    View Slide

  93. View Slide

  94. https://github.com/soulgalore/seleniumconf16

    View Slide

  95. Connectivity: https://www.flickr.com/photos/cogdog/15638928284/
    Catch regression: https://www.flickr.com/photos/testlab/21496317363/
    Fika: https://www.flickr.com/photos/andreasivarsson/5745405973/
    Boris 1: https://www.flickr.com/photos/53797600@N04/6849855824
    Broken heart: https://www.flickr.com/photos/miguelpdl/4356975474
    Queen: https://www.flickr.com/photos/foreignoffice/8283323803
    Ikea: https://www.flickr.com/photos/dahlstroms/4406947248
    Apple/Orange: https://www.flickr.com/photos/nubobo/8226113305
    Snowflakes: https://www.flickr.com/photos/dan1984/15678152054
    Crowd: https://www.flickr.com/photos/vin60/15386141803
    Synthetic: https://www.flickr.com/photos/-adam/4674856117
    Queen: https://www.flickr.com/photos/aftab/5578122981/
    Credits

    View Slide

  96. Traffic: https://www.flickr.com/photos/griff69
    Safetybelt: https://www.flickr.com/photos/torsteinsaltvedt/2460156876
    Free hugs: https://www.flickr.com/photos/clement127/13661779374
    Alone: https://www.flickr.com/photos/thomashawk/98867886
    Golden: https://www.flickr.com/photos/haeresis07/16191841366
    Ten commandments: https://www.flickr.com/photos/vancityscapes/512782411
    Angry: https://www.flickr.com/photos/stevendepolo/4605621230
    Puzzle: https://www.flickr.com/photos/acefrenzy/8163097
    Super heroes: https://www.flickr.com/photos/chuckles396/7145919879
    Credits

    View Slide

  97. Thank you!
    @soulislove
    [email protected]

    View Slide