Measuring Web Performance Using Selenium

You are the Selenium master and automate tests in your sleep. You make sure the quality of your companies web page is 100%. But then the team wants you to measure the performance of your site. How do you do that? Is that even possible with Selenium?

In this talk we will look at how we use Selenium in Browsertime & sitespeed.io to measure the speed of your web application and how you can implement it yourself in your own Selenium tests. We will also briefly look at strategies to make sure you can catch regressions in your sites performance.

You can checkout the talk at Youtube: https://www.youtube.com/watch?v=cLlZhnZvi1M

http://2016.seleniumconf.co.uk/sessions

Transcript

1. Peter Hedenskog
   @soulislove
   peter@soulgalore.com
   Measuring Web Performance
   Using Selenium
   SeleniumConf UK 2016
   Peter Hedenskog
   @soulislove
   peter@soulgalore.com
   

   View Slide

2. https://www.mediawiki.org/wiki/
   Wikimedia_Performance_Team
   

   View Slide

3. Wikipedia
   =! Wikileaks
   stupid!
   

   View Slide

4. View Slide

5. “Fika”
   

   View Slide

6. Tyrell
   

   View Slide

7. Pippi
   

   View Slide

8. ABBA
   

   View Slide

9. ABBA!!
   

   View Slide

10. ABBA!!!!!!!!!
    

    View Slide

11. View Slide

12. “Do not use Selenium for
    performance testing”
    

    View Slide

13. Yeah but you can!
    

    View Slide

14. Today
    

    View Slide

15. - Web performance metrics
    Today
    

    View Slide

16. - Web performance metrics
    - Selenium
    Today
    

    View Slide

17. - Web performance metrics
    - Selenium
    - Performance vs functional testing
    Today
    

    View Slide

18. - Web performance metrics
    - Selenium
    - Performance vs functional testing
    - Strategies for catching performance
    regression
    Today
    

    View Slide

19. Load
    testing?
    

    View Slide

20. The golden
    rule of Web
    Performance
    

    View Slide

21. “80-90% of the end-user response
    time is spent on the frontend.”
    

    View Slide

22. Web Performance
    Metrics
    

    View Slide

23. Navigation
    Timing API
    

    View Slide

24. Navigation Timing API
    

    View Slide

25. View Slide

26. View Slide

27. User Timing API
    

    View Slide

28. View Slide

29. View Slide

30. View Slide

31. Selenium
    

    View Slide

32. const script = 'var wp = window.performance.timing;' +
    'return wp.domComplete - wp.navigationStart;';
    driver.get('http://2016.seleniumconf.co.uk/')
    .then(() => {
    return driver.executeScript(script)
    })
    .then((result) => {
    console.log('DOMComplete: %d ms', result);
    })
    .finally(driver.quit);
    Javascript
    

    View Slide

33. NAVIGATION TIMING
    

    View Slide

34. Navigation
    timing is a Titanic
    success!
    

    View Slide

35. Navigation
    timings != user
    experience
    Navigation
    timings != user
    experience
    

    View Slide

36. Navigation
    timings != user
    experience
    User Timing
    API != user
    experience
    

    View Slide

37. // Chrome
    if (window.chrome && window.chrome.loadTimes) {
    // Convert to ms
    var firstPaint = window.chrome.loadTimes().firstPaintTime*1000;
    var startTime = window.chrome.loadTimes().startLoadTime*1000;
    console.log(firstPaint - startTime);
    }
    First paint
    https://github.com/addyosmani/timing.js/blob/
    master/timing.js#L47-L56
    

    View Slide

38. // IE
    if (typeof window.performance.timing.msFirstPaint === 'number') {
    var firstPaint = window.performance.timing.msFirstPaint;
    var startTime = window.performance.timing.navigationStart;
    console.log(firstPaint - startTime);
    }
    https://github.com/addyosmani/timing.js/blob/
    master/timing.js#L47-L56
    First paint
    

    View Slide

39. First paint
    

    View Slide

40. First paint
    SpeedIndex
    

    View Slide

41. FFMpeg
    VisualMetrics
    SpeedIndex
    

    View Slide

42. View Slide

43. View Slide

44. View Slide

45. View Slide

46. Coming soon
    

    View Slide

47. firstMeaningfulPaint
    firstTextPaint
    firstImagePaint
    https://yoavweiss.github.io/webperfwg_keynote_velocity_ams/#20
    

    View Slide

48. Chrome meaningfulPaint
    chromeOptions.setLoggingPrefs(logPrefs);
    chromeOptions.setPerfLoggingPrefs({enableNetwork: true,
    enablePage: true, traceCategories: "blink.user_timing"});
    ...
    case 'Tracing.dataCollected':
    // Here we can catch things as
    // firstImagePaint
    // firstContentfulPaint
    // firstTextPaint
    // firstMeaningfulPaint
    

    View Slide

49. You also want a
    HAR! Ändra title
    Something is missing
    

    View Slide

50. View Slide

51. BrowserMob
    Proxy?
    

    View Slide

52. Firefox HAR
    

    View Slide

53. View Slide

54. Firefox HAR
    'use strict';
    const webdriver = require('selenium-webdriver');
    const firefox = require('selenium-webdriver/firefox');
    const profile = new firefox.Profile();
    // HAR export - see http://www.softwareishard.com/blog/har-export-trigger/
    profile.setPreference('extensions.netmonitor.har.enableAutomation', true);
    profile.setPreference('extensions.netmonitor.har.contentAPIToken',
    'supersecrettoken');
    profile.setPreference('extensions.netmonitor.har.autoConnect', true);
    profile.setPreference('devtools.netmonitor.har.includeResponseBodies', false);
    // Download from the version page, the default URL shows wrong latest version
    // https://addons.mozilla.org/sv-se/firefox/addon/har-export-trigger/versions/?
    page=1#version-0.5.0-beta.10
    profile.addExtension('har_export_trigger-0.5.0-beta.10-fx.xpi');
    1/3
    

    View Slide

55. const script = `
    var callback = arguments[arguments.length - 1];
    function triggerExport() {
    HAR.triggerExport({'token':'supersecrettoken', 'getData':true})
    .then((result) => {
    var har = JSON.parse(result.data);
    har.log.pages[0].title = document.title;
    return callback({'har': JSON.stringify(har)});
    })
    .catch((e) => callback({'error': e}));
    };
    if (typeof HAR === 'undefined') {
    addEventListener('har-api-ready', triggerExport, false);
    } else {
    triggerExport();
    }`;
    2/3
    Firefox HAR
    

    View Slide

56. driver.get('http://2016.seleniumconf.co.uk/')
    .then(() =>
    driver.executeAsyncScript(script)
    )
    .then((result) => {
    if (result.error) {
    console.error('Could not get the HAR %s', result.error);
    } else {
    console.log(result.har);
    }
    })
    .finally(driver.quit);
    3/3
    Firefox HAR
    

    View Slide

57. Chrome HAR
    

    View Slide

58. Chrome HAR
    const logPrefs = new webdriver.logging.Preferences();
    logPrefs.setLevel(webdriver.logging.Type.PERFORMANCE,
    webdriver.logging.Level.INFO);
    const chromeOptions = new chrome.Options();
    chromeOptions.setLoggingPrefs(logPrefs);
    chromeOptions.setPerfLoggingPrefs({enableNetwork: true,
    enablePage: true});
    

    View Slide

59. https://github.com/
    sitespeedio/
    browsertime/blob/
    master/lib/support/
    chromePerflogParser.js
    Chrome HAR
    

    View Slide

60. Performance vs
    functional
    testing
    

    View Slide

61. Ten One
    commandments
    of performance
    testing
    

    View Slide

62. You shall make
    sure the metrics
    are repeatable!
    

    View Slide

63. 1. Setup the
    browser like a
    boss
    

    View Slide

64. Firefox setup
    const defaultFirefoxPreferences = {
    'browser.safebrowsing.enabled': false,
    'browser.safebrowsing.malware.enabled': false,
    'browser.safebrowsing.remotelookups': false,
    'browser.shell.checkDefaultBrowser': false,
    'browser.startup.homepage': 'about:blank'
    // IRL you need more!
    }
    const profile = new firefox.Profile();
    Object.keys(defaultFirefoxPreferences).forEach(function(pref)
    {
    profile.setPreference(pref, defaultFirefoxPreferences[pref]);
    });
    

    View Slide

65. const defaultChromeOptions = [
    '--disable-background-networking',
    '--no-default-browser-check',
    '--disable-translate',
    '--disable-desktop-notifications',
    '--disable-save-password-bubble',
    '--no-default-browser-check'
    // and more ...
    ];
    const chromeOptions = new chrome.Options();
    chromeOptions.addArguments(defaultChromeOptions);
    Chrome setup
    

    View Slide

66. 2. Use the same
    connectivity
    

    View Slide

67. View Slide

68. Also tc
    Also tc,
    dummynet &
    tylertreat/
    comcast
    

    View Slide

69. Chrome TSProxy
    // Configure SOCKS proxy, see https://www.chromium.org/
    developers/design-documents/network-stack/socks-proxy
    chromeOptions.addArguments('--proxy-server=socks5://
    localhost:1080');
    chromeOptions.addArguments('--host-resolver-rules="MAP
    * ~NOTFOUND , EXCLUDE localhost"');
    

    View Slide

70. profile.setPreference("network.proxy.socks", "localhost");
    profile.setPreference("network.proxy.socks_port", 1080);
    profile.setPreference("network.proxy.type", 1);
    Firefox TSProxy
    

    View Slide

71. tc - simulating 3g
    sudo tc qdisc add dev eth0 root netem
    delay 300ms loss 0% rate 1600kbps
    

    View Slide

72. 3. Run many
    iterations
    

    View Slide

73. 4. Run it alone
    

    View Slide

74. View Slide

75. Nooooo!!!!
    

    View Slide

76. Catch
    regressions
    

    View Slide

77. View Slide

78. View Slide

79. View Slide

80. View Slide

81. CI + RUM +
    synthetic = love
    

    View Slide

82. IRL
    

    View Slide

83. https://grafana.wikimedia.org/dashboard/db/
    navigation-timing
    

    View Slide

84. https://grafana.wikimedia.org/dashboard/db/
    navigation-timing
    

    View Slide

85. https://grafana.wikimedia.org/dashboard/db/
    navigation-timing-by-browser
    

    View Slide

86. https://grafana.wikimedia.org/dashboard/db/
    webpagetest
    

    View Slide

87. But what do you
    do when you
    have different
    stories?
    

    View Slide

88. Summary
    

    View Slide

89. Navigation
    Timing
    User Timing
    

    View Slide

90. Navigation Timing
    User Timing
    

    View Slide

91. SpeedIndex
    

    View Slide

92. Before, RUM and
    synthetic
    

    View Slide

93. View Slide

94. https://github.com/soulgalore/seleniumconf16
    

    View Slide

95. Connectivity: https://www.flickr.com/photos/cogdog/15638928284/
    Catch regression: https://www.flickr.com/photos/testlab/21496317363/
    Fika: https://www.flickr.com/photos/andreasivarsson/5745405973/
    Boris 1: https://www.flickr.com/photos/53797600@N04/6849855824
    Broken heart: https://www.flickr.com/photos/miguelpdl/4356975474
    Queen: https://www.flickr.com/photos/foreignoffice/8283323803
    Ikea: https://www.flickr.com/photos/dahlstroms/4406947248
    Apple/Orange: https://www.flickr.com/photos/nubobo/8226113305
    Snowflakes: https://www.flickr.com/photos/dan1984/15678152054
    Crowd: https://www.flickr.com/photos/vin60/15386141803
    Synthetic: https://www.flickr.com/photos/-adam/4674856117
    Queen: https://www.flickr.com/photos/aftab/5578122981/
    Credits
    

    View Slide

96. Traffic: https://www.flickr.com/photos/griff69
    Safetybelt: https://www.flickr.com/photos/torsteinsaltvedt/2460156876
    Free hugs: https://www.flickr.com/photos/clement127/13661779374
    Alone: https://www.flickr.com/photos/thomashawk/98867886
    Golden: https://www.flickr.com/photos/haeresis07/16191841366
    Ten commandments: https://www.flickr.com/photos/vancityscapes/512782411
    Angry: https://www.flickr.com/photos/stevendepolo/4605621230
    Puzzle: https://www.flickr.com/photos/acefrenzy/8163097
    Super heroes: https://www.flickr.com/photos/chuckles396/7145919879
    Credits
    

    View Slide

97. Thank you!
    @soulislove
    peter@soulgalore.com
    

    View Slide