Metrics that Matter

Whenever we speak about measuring web performance and user experience, we typically refer to static events in the browsing experience. Modern websites, however, are far from static, and user interactions with these websites have continuous aspects that cannot be represented by distinct events.

* How do you measure user frustration?
* How do you tell how fast your users expect your site to be?
* How do you tell if users were frustrated or annoyed before they got to your site or if your site caused that frustration?
* What are TTVR, TTI, TTFI, FID, and which of them is the most important?
* How do you tell if your page locks up during user interaction?

In this talk, we’ll look at some new user experience metrics. We’ll find out how to measure page responsiveness, smoothness, jank, and usability. We’ll learn about things like Rage Clicks, Missed Clicks, Dead Clicks, and Cursor Thrashing. We’ll also look at real user data that we’ve collected showing how these aspects of the page affect user behaviour.

Transcript

1. Metrics that Matter
   Measuring the effects of website performance on users
   

   View Slide

2. Philip Tellis
   Principal RUM Distiller @ Akamai
   Author of the OpenSource boomerang RUM
   library
   twitter:@bluesmoon ⦿ github:@bluesmoon
   speakerdeck:@bluesmoon
   

   View Slide

3. Abbé Jean-Antoine Nollet
   1700 - 1770
   French Clergyman & Budding
   Electrician
   Invented one of the first Electroscopes
   (we now call them beacon collectors)
   L'Abbé Jean Antoine Nollet — Maurice Quentin de La Tour
   Alte Pinakothek, Munich, Germany Public Domain
   

   View Slide

4. In 1746, he conducted the first ever RUM Experiment
   He shot an electric current
   through 200 monks, and
   checked how quickly they
   jumped; thereby measuring
   the latency of an electric
   signal with…
   Real Users!
   

   View Slide

5. Fortunately, our methods have gotten
   a lot less intrusive
   

   View Slide

6. What should we
   measure?
   What are the metrics that matter to users?
   

   View Slide

7. Loading a Webpage...
   https://addyosmani.com/blog/usability/
   

   View Slide

8. These moments are related and trigger
   different expectations with the user
   

   View Slide

9. Is it happening?
   What visual cues does the user
   have that this page might actually
   show up?
   That initial period of a page
   load when the screen goes
   blank causes anxiety & doubt
   for the user…
   ● Did I click correctly?
   ● Is it loading?
   ● Has is already billed my
   credit card?
   

   View Slide

10. First Paint
    Framework Ready
    Hero Images
    DOM Content Loaded
    Visually Ready
    https://www.slideshare.net/nicjansma/reliably-measuring-responsiveness
    

    View Slide

11. When does the first thing show up on screen?
    ● Register a PerformanceObserver before the event happens
    ● OR use performance.getEntriesByType("paint") after the event
    https://developer.mozilla.org/en-US/docs/Web/API/PerformancePaintTiming
    Currently only available in Chrome and Chrome based browsers
    

    View Slide

12. What about other browsers?
    Internet Explorer & Pre-Chrome Edge
    NavigationTiming on IE includes a
    proprietary msFirstPaint property.
    DOM Events
    We can look at when domInteractive fires,
    using NavigationTiming or by listening
    for readystatechange.
    Hero Images/Elements
    We could use a MutationObserver to
    determine when a particular image or
    element shows up in the DOM, or use
    ResourceTiming to check for it later.
    UserTiming
    Page or framework authors can use the
    UserTiming API to specify when
    important items are available.
    

    View Slide

13. Last Painted Hero
    max(h1, (biggest_img || bg_img))
    ● Developed by Steve Souders
    ● Named by Paul Irish
    It’s a synthetic measure of when
    the last of the hero elements was
    drawn on screen.
    https://twitter.com/vazac/status/979147191896784897
    https://speedcurve.com/blog/last-painted-hero/
    

    View Slide

14. Is it Useful?
    Is this the page I’m looking for?
    This stage…
    ● Removes doubt
    ● Confirms that the user is
    on the right path
    

    View Slide

15. Signals to detect usefulness
    ● First Meaningful Paint – Lighthouse
    only at the moment
    ● Hero Element timing is something
    that can be set by page developers
    using User Timing or Resource
    Timing.
    ● Boomerang aggregates all of these
    into a measure called Time to
    Visually Ready or ttvr.
    https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html
    https://developers.google.com/web/fundamentals/performance/user-centric-performance-metrics
    

    View Slide

16. Is it Usable?
    If the user tried to interact with
    the page, would they have a good
    experience?
    The user is ready to be
    delighted at this point, they’ve
    come to the right place...
    ● But is the page responsive?
    ● Is there jank?
    ● Are there random lockups?
    ● Is the interface confusing?
    

    View Slide

17. Not Interactive Until Here
    User Sees Content
    Time to Interactive
    https://www.slideshare.net/nicjansma/reliably-measuring-responsiveness
    

    View Slide

18. Signals to detect usability
    After Visually Ready (ie, user thinks they can use the page), check...
    ● Frame Rate using requestAnimationFrame
    ● Long Tasks using PerformanceObserver or use setInterval on older browsers
    ● Measure latency of responding to any user input (click, keypress, scroll)
    First period of 500ms after VR with no Long Tasks, FPS > 20, and Page
    Busy < 10%
    https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html
    https://developers.google.com/web/updates/2018/05/first-input-delay
    

    View Slide

19. Short Tasks Long Tasks
    https://developers.google.com/web/fundamentals/performance/user-centric-performance-metrics
    

    View Slide

20. Long Task Duration
    https://www.slideshare.net/nicjansma/reliably-measuring-responsiveness
    

    View Slide

21. LongTasks as % of Front End Load Time
    https://www.slideshare.net/nicjansma/reliably-measuring-responsiveness
    

    View Slide

22. Check Frame Rate using requestAnimationFrame
    var frames = 0;
    function measureFps() {
    frames++;
    // request a callback before the next frame
    window.requestAnimationFrame(measureFps);
    }
    // start measuring
    window.requestAnimationFrame(measureFps);
    // report on frame rate (FPS) once a second
    setInterval(function() {
    console.log("FPS: " + frames);
    frames = 0;
    }, 1000);
    https://www.slideshare.net/nicjansma/measuring-continuity
    

    View Slide

23. Other important continuity metrics
    ● Measure Battery Level using navigator.getBattery(...)
    ● Measure memory usage using
    performance.memory.usedJSHeapSize
    ● Check visibilityState and use IntersectionObserver to stop
    expensive tasks when not visible.
    ● Capture JavaScript errors
    

    View Slide

24. Is it delightful?
    When the user does interact, is
    that interaction pleasant?
    ● Were interactions smooth?
    ● Was the interface responsive?
    ● Was the interface consistent?
    

    View Slide

25. Signals to detect Delight
    Developer Signals
    ● Long Tasks come up again
    ● As does Frame Rate and
    input latency
    ● Measure these continuously
    while the page is active
    User Signals
    ● Rage Clicks
    ● Cursor Thrashing / Wild
    Mouse
    ● Scrandom
    

    View Slide

26. ● Ask the user
    ● Bounce / Conversion Rate /
    LD
    50
    ● Rage Clicks / Cursor Thrashing
    / Wild Mouse
    ● Facial analysis
    ● Wireless Brain Interface
    Methods of measuring Emotion
    (with JavaScript)
    

    View Slide

27. Rage Clicks occur when users rapid-fire click
    (or tap) on your site or app.
    Rage clicking is the digital equivalent of
    cursing to release frustration.
    https://www.fullstory.com/resources/guide-to-understanding-frustrating-user-experiences-online/
    https://www.psychologytoday.com/us/blog/hide-and-seek/201205/hell-yes-the-7-best-reasons-swearing
    Rage Clicks
    

    View Slide

28. Measuring Rage Clicks
    Rage Clicks are measured in JavaScript by looking
    for more than 2 clicks within the same 10 pixels in a
    short period of time.
    onMouse*, event.timestamp, and an array
    https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html
    

    View Slide

29. Rage Clicks depend on Latency of Page Usability
    ● Interacting with a page before onload
    OR interactive is the most common
    cause of Rage Clicks
    ● Some Rage Clicks happen after the
    page is usable, possibly due to
    JavaScript errors or CPU intensive
    tasks; they could also be false
    positives.
    ● In over 30% of cases, a page is
    interactive after onload fires, and in
    15% of cases, users try interacting
    between onload and interactive.
    Data collected and analysed with boomerang and Akamai mPulse
    

    View Slide

30. Data collected and analysed with boomerang and Akamai mPulse
    Rage Clicking is fairly consistent if first Interaction is
    before the page becomes Interactive
    Steady drop off if first Interaction is after the
    page becomes Interactive
    Rage Clicks are most likely if first Interaction is just
    before onload, possibly because DOM Ready event
    handlers are hogging CPU.
    (This is true even if the page becomes Interactive
    before onload)
    

    View Slide

31. Data collected and analysed with boomerang and Akamai mPulse
    Desktop vs Mobile
    

    View Slide

32. Rage Clicks as a Function of First Interaction & Visually REady
    Data collected and analysed with boomerang and Akamai mPulse
    

    View Slide

33. People who are angry are more likely to use the mouse in
    a jerky and sudden, but surprisingly slow fashion.
    People who feel frustrated, confused or sad are less
    precise in their mouse movements and move it at
    different speeds.
    https://www.telegraph.co.uk/technology/news/12050481/Websites-could-read-emotions-by-seeing-how-fast-you-move-your-mouse.html
    https://conversionxl.com/blog/user-frustration/
    https://www.academia.edu/3085041/Patterns_of_cursor_movement_for_different_devices
    Cursor Thrashing/Wild Mouse
    

    View Slide

34. Measuring Cursor Thrashing & Wild Mouse
    Cursor Thrashing is measured in JavaScript by
    looking at mouse movements over time, both in
    absolute terms and as a fraction of screen size.
    onMouseMove, setTimeout, event.clientX,
    event.clientY, ...
    https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html
    

    View Slide

35. We’re still working on correlating this with
    performance and business metrics
    
    

    View Slide

36. Optimizing user centric Metrics
    ● Remove render blocking scripts from the HEAD
    ● Inline critical CSS
    ● Reduce work done in JavaScript
    ● Move non-critical JavaScript to after onload
    ● Split up long tasks
    ● If you can, move expensive JavaScript over to Web Workers
    

    View Slide

37. Summary
    ● Measure what affects your
    user’s emotional state.
    ● Reduce the amount of work
    you do in onload and
    readyState handlers.
    ● Measure Responsiveness and
    Smoothness of your UI.
    Pacific Islander Navigation Map, Museum of Fine Arts, Boston
    https://www.flickr.com/photos/bluesmoon/1266590108/
    

    View Slide

38. ● Addy Osmani on Usability
    ● Reliably Measuring Responsiveness in the Wild
    ● Last Painted Hero
    ● Boomerang Continuity Plugin
    ● User Centric Performance Metrics
    ● First Input Delay
    ● Measuring Continuity
    ● Understanding Frustrating User Experiences
    ● The 7 Best Reasons for Swearing
    ● Reading Emotions by Measuring Mouse Movements
    ● Your users are Frustrated
    ● Patterns of Cursor Movement for Different Devices
    ● Tracking CPU with the Long Tasks API
    References
    ● Paint Timing API
    ● Navigation Timing
    ● Resource Timing
    ● User Timing
    ● MutationObserver
    ● PerformanceObserver
    ● msFirstPaint
    ● readystatechange
    ● requestAnimationFrame
    ● setInterval / requestIdleCallback
    ● Lighthouse
    ● Long Tasks
    

    View Slide

39. Thank You!
    

    View Slide