UX & Performance: Metrics that Matter

In this talk, we’ll look at some of the new user experience metrics that the boomerang team has been collecting. We’ll find out how to measure page responsiveness, smoothness, jank, and usability. We’ll learn about things like Rage Clicks, Missed Clicks, and Dead Clicks. We’ll also look at real user data that we’ve collected showing how these aspects of the page affect user behaviour.

JavaScript provides us with many hooks into measuring performance and user experience. Let’s learn to collect them and understand what to expect when we optimise for them.

Transcript

1. UX & Performance
   Metrics that Matter
   

   View Slide

2. Philip Tellis
   Principal RUM Distiller @ Akamai
   Author of the OpenSource boomerang RUM
   library
   twitter:@bluesmoon * github:@bluesmoon
   

   View Slide

3. The Art & Science of Real User Measurement
   

   View Slide

4. What should we
   measure?
   

   View Slide

5. How should we
   measure it?
   

   View Slide

6. A user’s emotional response to latency and tolerance of
   errors appears to correlate with their emotional state.
   Measuring Emotion...
   https://speakerdeck.com/bluesmoon/sciencing-data
   

   View Slide

7. Methods of measuring Emotion
   ● Ask the user
   ● Bounce / Conversion Rate / LD
   50
   ● Rage Clicks / Cursor Thrashing
   ● Facial analysis
   ● Wireless Brain Interface
   

   View Slide

8. Rage Clicks occur when users rapid-fire click
   (or tap) on your site or app.
   Rage clicking is the digital equivalent of
   cursing to release frustration.
   https://www.fullstory.com/resources/guide-to-understanding-frustrating-user-experiences-online/
   https://www.psychologytoday.com/us/blog/hide-and-seek/201205/hell-yes-the-7-best-reasons-swearing
   Rage Clicks
   

   View Slide

9. People who are angry are more likely to use the mouse
   in a jerky and sudden, but surprisingly slow fashion.
   People who feel frustrated, confused or sad are less
   precise in their mouse movements and move it at
   different speeds.
   https://www.telegraph.co.uk/technology/news/12050481/Websites-could-read-emotions-by-seeing-how-fast-you-move-your-mouse.html
   Cursor Thrashing
   

   View Slide

10. https://webgazer.cs.brown.edu/
    Facial Analysis Mind Reading
    https://www.emotiv.com/
    We do NOT do either of these
    

    View Slide

11. Ask the User
    We do NOT do this either
    ● Selection Bias
    ● Hawthorne Effect
    ● Intrusive action (boomerang is passive)
    

    View Slide

12. Rage Clicks are measured in JavaScript by looking for more than 2 clicks
    within the same 10 pixels in a short period of time.
    onMouse*, event.timestamp, and an array
    The code is in boomerang’s Continuity Plugin.
    Measuring Rage Clicks
    https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html
    

    View Slide

13. Rage Click likelihood depends on Latency of Page Usability
    ● Interacting with a page before onload OR interactive is
    the most common cause of Rage Clicks
    ● Some Rage Clicks happen after the page is usable,
    possibly due to JavaScript errors or CPU intensive
    tasks; they could also be false positives.
    ● In over 30% of cases, a page is interactive after onload
    fires, and in 15% of cases, users try interacting between
    onload and interactive.
    Data collected and analysed with boomerang and Akamai mPulse
    

    View Slide

14. Data collected and analysed with boomerang and Akamai mPulse
    Rage Clicking is fairly consistent if first Interaction is before the page
    becomes Interactive
    Steady drop off if first Interaction is after the page becomes
    Interactive
    Rage Clicks are most likely if first Interaction is just before onload,
    possibly because DOM Ready event handlers are hogging CPU.
    (This is true even if the page becomes Interactive before onload)
    

    View Slide

15. Data collected and analysed with boomerang and Akamai mPulse
    

    View Slide

16. ● We found that the majority of users who Rage Click
    attempt to interact with a page between 1.25 and 1.5x
    the visually ready time.
    ● We don’t know why, but users expect a page to be
    interactive at 1.3x the time when it is visual.
    ● So, make sure your page is interactive and loaded
    before 1.3x the visually ready time.
    What is the optimum time to reduce Rage Clicking?
    Data collected and analysed with boomerang and Akamai mPulse
    Ratio of First Interaction to Visually Ready
    Rage Clicks
    

    View Slide

17. ● There is a clear linear correlation
    ● The slope varies between 1.2 and 1.5
    Rage Clicks by First Interaction & Visually Ready
    Data collected and analysed with boomerang and Akamai mPulse
    

    View Slide

18. Cursor Thrashing is measured in JavaScript by looking at mouse movements
    over time, both in absolute terms and as a fraction of screen size.
    onMouseMove, setTimeout, event.clientX, event.clientY, ...
    The code is in boomerang’s Continuity Plugin.
    Measuring Cursor Thrashing
    https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html
    

    View Slide

19. Rage Clicks are
    ● A user’s response to lack of
    responsiveness on your site.
    ● Indicative of design or performance
    issues with your site.
    Key differences between Rage Clicks & Cursor Thrashing...
    Cursor Thrashing is
    ● Indicative of the user’s emotional state
    before they got to your site
    ● A signal you can use to change how
    you present your site.
    

    View Slide

20. Definitions of our
    Metrics that Matter
    https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html
    

    View Slide

21. Timers
    ●
    Visually Ready: When did the user feel like they could interact with the site?
    ●
    Interactive: After the page was Visually Ready, when was the first time the user could
    have interacted with the site, and had a good experience?
    ●
    First Interaction: When was the first time the user tried to interact with the site?
    ●
    First Input Delay: For the first interaction on the page, how responsive was it?
    

    View Slide

22. ●
    Delayed Interactions: How often was the user's interaction delayed more than 50ms?
    ●
    Rage Clicks: Did the user repeatedly click on the same element/region?
    ●
    Mouse Movements: How did the user move the mouse around the screen?
    Interaction Metrics
    

    View Slide

23. Page Performance Metrics
    ●
    Frame Rate data: FPS during page load, minimum FPS, number of long frames
    ●
    Long Task data: Number of Long Tasks, how much time they took, attribution to what
    caused them
    ●
    Page Busy: Measurement of the page's “busyness”
    

    View Slide

24. How do we measure what matters?
    ●
    Use boomerang!
    ●
    Read the docs of boomerang’s Continuity Plugin
    ●
    Watch the talk by Nic Jansma and Philip Tellis at Velocity
    . . .
    

    View Slide

25. ● First Paint, First Contentful Paint, DOMContentLoaded
    ● Hero Images (eg: class="hero-image")
    ● Framework Ready Event
    Pick the last event from the above, that is when the user thinks the page is visually ready
    Visually Ready
    

    View Slide

26. ● Measure Frame Rate using requestAnimationFrame.
    ● Measure Long Tasks using PerformanceObserver (if available).
    ● Measure Page Busy using setInterval (if Long Tasks not available).
    ● Measure Interaction Latency
    First period of 500ms after Visually Ready with No Long Tasks, FPS > 20, and Page Busy < 10%
    Time to Interactive
    

    View Slide

27. ● Instrument Mouse, Keyboard, and Scroll Events in passive mode
    ● Measure latency between event.timestamp, and event handler being called
    performance.now().
    Delayed Interactions
    

    View Slide

28. ● Use requestAnimationFrame
    ● Measure time between consecutive calls to rAF
    ● 16ms for the best experience
    This measurement is expensive on the CPU and cause excessive battery usage when the page is
    idle, so best to use it only during the page load process.
    Frame Rate
    

    View Slide

29. var observer = new PerformanceObserver(function(list) {
    var perfEntries = list.getEntries();
    for (var i = 0; i < perfEntries.length; i++) {
    // Process long task notifications:
    }
    });
    // register observer for long task notifications
    observer.observe({entryTypes: ["longtask"]});
    Long Tasks: Use PerformanceObserver
    

    View Slide

30. Summary
    ● Measure what affects your user’s
    emotional state.
    ● Reduce the amount of work you do in
    onload and readyState handlers.
    ● Measure Responsiveness and
    Smoothness of your UI.
    Pacific Islander Navigation Map, Museum of Fine Arts, Boston
    https://www.flickr.com/photos/bluesmoon/1266590108/
    

    View Slide

31. Thank You
    Check out boomerang on github for much more detail
    

    View Slide

32. References
    ● Boomerang’s Continuity Plugin
    https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html
    ● Fullstory’s guide to User Frustration
    https://www.fullstory.com/resources/guide-to-understanding-frustrating-user-experiences-online/
    ● Reading emotion through measuring mouse movements
    https://www.telegraph.co.uk/technology/news/12050481/Websites-could-read-emotions-by-seeing-how...
    ● Inferring Emotion through HCI Devices
    https://www.researchgate.net/publication/283422711_How_Is_Your_User_Feeling_Inferring_Emotion...
    ● Correlation between emotional state and browsing behaviour
    https://speakerdeck.com/bluesmoon/sciencing-data
    ● Scroll behaviour across the web
    http://blog.chartbeat.com/2013/08/12/scroll-behavior-across-the-web/
    ● Debouncing & Throttling Events
    https://css-tricks.com/debouncing-throttling-explained-examples/
    ● Measuring Web Page Jank
    https://webperf.ninja/2015/jank-meter/
    ● Scroll Performance with Passive Event Listeners
    https://developers.google.com/web/updates/2016/06/passive-event-listeners
    ● What Every Browser knows about You
    http://webkay.robinlinus.com/
    ● An Audit of Boomerang’s Performance
    https://nicj.net/an-audit-of-boomerangs-performance/
    ● The Long Tasks API
    https://w3c.github.io/longtasks/
    ● The Paint Timing API
    https://www.w3.org/TR/paint-timing/
    ● Resource Timing Level 2
    https://www.w3.org/TR/resource-timing-2/
    ● Intersection Observer API
    https://developer.mozilla.org/en-US/docs/Web/API/Intersection_Observer_API
    ● requestIdleCallback
    https://developers.google.com/web/updates/2015/08/using-requestidlecallback
    ● requestAnimationFrame
    https://developer.mozilla.org/en-US/docs/Web/API/window/requestAnimationFrame
    ● Last Painted Hero
    https://speedcurve.com/blog/last-painted-hero/
    ● The Runtime Performance Checklist
    https://calendar.perfplanet.com/2013/the-runtime-performance-checklist/
    

    View Slide