$30 off During Our Annual Pro Sale. View Details »

Metrics that Matter

Metrics that Matter

Whenever we speak about measuring web performance and user experience, we typically refer to static events in the browsing experience. Modern websites, however, are far from static, and user interactions with these websites have continuous aspects that cannot be represented by distinct events.

* How do you measure user frustration?
* How do you tell how fast your users expect your site to be?
* How do you tell if users were frustrated or annoyed before they got to your site or if your site caused that frustration?
* What are TTVR, TTI, TTFI, FID, and which of them is the most important?
* How do you tell if your page locks up during user interaction?

In this talk, we’ll look at some new user experience metrics. We’ll find out how to measure page responsiveness, smoothness, jank, and usability. We’ll learn about things like Rage Clicks, Missed Clicks, Dead Clicks, and Cursor Thrashing. We’ll also look at real user data that we’ve collected showing how these aspects of the page affect user behaviour.

Philip Tellis

May 22, 2019
Tweet

More Decks by Philip Tellis

Other Decks in Technology

Transcript

  1. Metrics that Matter
    Measuring the effects of website performance on users

    View Slide

  2. Philip Tellis
    Principal RUM Distiller @ Akamai
    Author of the OpenSource boomerang RUM
    library
    twitter:@bluesmoon ⦿ github:@bluesmoon
    speakerdeck:@bluesmoon

    View Slide

  3. Abbé Jean-Antoine Nollet
    1700 - 1770
    French Clergyman & Budding
    Electrician
    Invented one of the first Electroscopes
    (we now call them beacon collectors)
    L'Abbé Jean Antoine Nollet — Maurice Quentin de La Tour
    Alte Pinakothek, Munich, Germany Public Domain

    View Slide

  4. In 1746, he conducted the first ever RUM Experiment
    He shot an electric current
    through 200 monks, and
    checked how quickly they
    jumped; thereby measuring
    the latency of an electric
    signal with…
    Real Users!

    View Slide

  5. Fortunately, our methods have gotten
    a lot less intrusive

    View Slide

  6. What should we
    measure?
    What are the metrics that matter to users?

    View Slide

  7. Loading a Webpage...
    https://addyosmani.com/blog/usability/

    View Slide

  8. These moments are related and trigger
    different expectations with the user

    View Slide

  9. Is it happening?
    What visual cues does the user
    have that this page might actually
    show up?
    That initial period of a page
    load when the screen goes
    blank causes anxiety & doubt
    for the user…
    ● Did I click correctly?
    ● Is it loading?
    ● Has is already billed my
    credit card?

    View Slide

  10. First Paint
    Framework Ready
    Hero Images
    DOM Content Loaded
    Visually Ready
    https://www.slideshare.net/nicjansma/reliably-measuring-responsiveness

    View Slide

  11. When does the first thing show up on screen?
    ● Register a PerformanceObserver before the event happens
    ● OR use performance.getEntriesByType("paint") after the event
    https://developer.mozilla.org/en-US/docs/Web/API/PerformancePaintTiming
    Currently only available in Chrome and Chrome based browsers

    View Slide

  12. What about other browsers?
    Internet Explorer & Pre-Chrome Edge
    NavigationTiming on IE includes a
    proprietary msFirstPaint property.
    DOM Events
    We can look at when domInteractive fires,
    using NavigationTiming or by listening
    for readystatechange.
    Hero Images/Elements
    We could use a MutationObserver to
    determine when a particular image or
    element shows up in the DOM, or use
    ResourceTiming to check for it later.
    UserTiming
    Page or framework authors can use the
    UserTiming API to specify when
    important items are available.

    View Slide

  13. Last Painted Hero
    max(h1, (biggest_img || bg_img))
    ● Developed by Steve Souders
    ● Named by Paul Irish
    It’s a synthetic measure of when
    the last of the hero elements was
    drawn on screen.
    https://twitter.com/vazac/status/979147191896784897
    https://speedcurve.com/blog/last-painted-hero/

    View Slide

  14. Is it Useful?
    Is this the page I’m looking for?
    This stage…
    ● Removes doubt
    ● Confirms that the user is
    on the right path

    View Slide

  15. Signals to detect usefulness
    ● First Meaningful Paint – Lighthouse
    only at the moment
    ● Hero Element timing is something
    that can be set by page developers
    using User Timing or Resource
    Timing.
    ● Boomerang aggregates all of these
    into a measure called Time to
    Visually Ready or ttvr.
    https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html
    https://developers.google.com/web/fundamentals/performance/user-centric-performance-metrics

    View Slide

  16. Is it Usable?
    If the user tried to interact with
    the page, would they have a good
    experience?
    The user is ready to be
    delighted at this point, they’ve
    come to the right place...
    ● But is the page responsive?
    ● Is there jank?
    ● Are there random lockups?
    ● Is the interface confusing?

    View Slide

  17. Not Interactive Until Here
    User Sees Content
    Time to Interactive
    https://www.slideshare.net/nicjansma/reliably-measuring-responsiveness

    View Slide

  18. Signals to detect usability
    After Visually Ready (ie, user thinks they can use the page), check...
    ● Frame Rate using requestAnimationFrame
    ● Long Tasks using PerformanceObserver or use setInterval on older browsers
    ● Measure latency of responding to any user input (click, keypress, scroll)
    First period of 500ms after VR with no Long Tasks, FPS > 20, and Page
    Busy < 10%
    https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html
    https://developers.google.com/web/updates/2018/05/first-input-delay

    View Slide

  19. Short Tasks Long Tasks
    https://developers.google.com/web/fundamentals/performance/user-centric-performance-metrics

    View Slide

  20. Long Task Duration
    https://www.slideshare.net/nicjansma/reliably-measuring-responsiveness

    View Slide

  21. LongTasks as % of Front End Load Time
    https://www.slideshare.net/nicjansma/reliably-measuring-responsiveness

    View Slide

  22. Check Frame Rate using requestAnimationFrame
    var frames = 0;
    function measureFps() {
    frames++;
    // request a callback before the next frame
    window.requestAnimationFrame(measureFps);
    }
    // start measuring
    window.requestAnimationFrame(measureFps);
    // report on frame rate (FPS) once a second
    setInterval(function() {
    console.log("FPS: " + frames);
    frames = 0;
    }, 1000);
    https://www.slideshare.net/nicjansma/measuring-continuity

    View Slide

  23. Other important continuity metrics
    ● Measure Battery Level using navigator.getBattery(...)
    ● Measure memory usage using
    performance.memory.usedJSHeapSize
    ● Check visibilityState and use IntersectionObserver to stop
    expensive tasks when not visible.
    ● Capture JavaScript errors

    View Slide

  24. Is it delightful?
    When the user does interact, is
    that interaction pleasant?
    ● Were interactions smooth?
    ● Was the interface responsive?
    ● Was the interface consistent?

    View Slide

  25. Signals to detect Delight
    Developer Signals
    ● Long Tasks come up again
    ● As does Frame Rate and
    input latency
    ● Measure these continuously
    while the page is active
    User Signals
    ● Rage Clicks
    ● Cursor Thrashing / Wild
    Mouse
    ● Scrandom

    View Slide

  26. ● Ask the user
    ● Bounce / Conversion Rate /
    LD
    50
    ● Rage Clicks / Cursor Thrashing
    / Wild Mouse
    ● Facial analysis
    ● Wireless Brain Interface
    Methods of measuring Emotion
    (with JavaScript)

    View Slide

  27. Rage Clicks occur when users rapid-fire click
    (or tap) on your site or app.
    Rage clicking is the digital equivalent of
    cursing to release frustration.
    https://www.fullstory.com/resources/guide-to-understanding-frustrating-user-experiences-online/
    https://www.psychologytoday.com/us/blog/hide-and-seek/201205/hell-yes-the-7-best-reasons-swearing
    Rage Clicks

    View Slide

  28. Measuring Rage Clicks
    Rage Clicks are measured in JavaScript by looking
    for more than 2 clicks within the same 10 pixels in a
    short period of time.
    onMouse*, event.timestamp, and an array
    https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html

    View Slide

  29. Rage Clicks depend on Latency of Page Usability
    ● Interacting with a page before onload
    OR interactive is the most common
    cause of Rage Clicks
    ● Some Rage Clicks happen after the
    page is usable, possibly due to
    JavaScript errors or CPU intensive
    tasks; they could also be false
    positives.
    ● In over 30% of cases, a page is
    interactive after onload fires, and in
    15% of cases, users try interacting
    between onload and interactive.
    Data collected and analysed with boomerang and Akamai mPulse

    View Slide

  30. Data collected and analysed with boomerang and Akamai mPulse
    Rage Clicking is fairly consistent if first Interaction is
    before the page becomes Interactive
    Steady drop off if first Interaction is after the
    page becomes Interactive
    Rage Clicks are most likely if first Interaction is just
    before onload, possibly because DOM Ready event
    handlers are hogging CPU.
    (This is true even if the page becomes Interactive
    before onload)

    View Slide

  31. Data collected and analysed with boomerang and Akamai mPulse
    Desktop vs Mobile

    View Slide

  32. Rage Clicks as a Function of First Interaction & Visually REady
    Data collected and analysed with boomerang and Akamai mPulse

    View Slide

  33. People who are angry are more likely to use the mouse in
    a jerky and sudden, but surprisingly slow fashion.
    People who feel frustrated, confused or sad are less
    precise in their mouse movements and move it at
    different speeds.
    https://www.telegraph.co.uk/technology/news/12050481/Websites-could-read-emotions-by-seeing-how-fast-you-move-your-mouse.html
    https://conversionxl.com/blog/user-frustration/
    https://www.academia.edu/3085041/Patterns_of_cursor_movement_for_different_devices
    Cursor Thrashing/Wild Mouse

    View Slide

  34. Measuring Cursor Thrashing & Wild Mouse
    Cursor Thrashing is measured in JavaScript by
    looking at mouse movements over time, both in
    absolute terms and as a fraction of screen size.
    onMouseMove, setTimeout, event.clientX,
    event.clientY, ...
    https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html

    View Slide

  35. We’re still working on correlating this with
    performance and business metrics

    View Slide

  36. Optimizing user centric Metrics
    ● Remove render blocking scripts from the HEAD
    ● Inline critical CSS
    ● Reduce work done in JavaScript
    ● Move non-critical JavaScript to after onload
    ● Split up long tasks
    ● If you can, move expensive JavaScript over to Web Workers

    View Slide

  37. Summary
    ● Measure what affects your
    user’s emotional state.
    ● Reduce the amount of work
    you do in onload and
    readyState handlers.
    ● Measure Responsiveness and
    Smoothness of your UI.
    Pacific Islander Navigation Map, Museum of Fine Arts, Boston
    https://www.flickr.com/photos/bluesmoon/1266590108/

    View Slide

  38. ● Addy Osmani on Usability
    ● Reliably Measuring Responsiveness in the Wild
    ● Last Painted Hero
    ● Boomerang Continuity Plugin
    ● User Centric Performance Metrics
    ● First Input Delay
    ● Measuring Continuity
    ● Understanding Frustrating User Experiences
    ● The 7 Best Reasons for Swearing
    ● Reading Emotions by Measuring Mouse Movements
    ● Your users are Frustrated
    ● Patterns of Cursor Movement for Different Devices
    ● Tracking CPU with the Long Tasks API
    References
    ● Paint Timing API
    ● Navigation Timing
    ● Resource Timing
    ● User Timing
    ● MutationObserver
    ● PerformanceObserver
    ● msFirstPaint
    ● readystatechange
    ● requestAnimationFrame
    ● setInterval / requestIdleCallback
    ● Lighthouse
    ● Long Tasks

    View Slide

  39. Thank You!

    View Slide