Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Metrics that Matter

Metrics that Matter

Whenever we speak about measuring web performance and user experience, we typically refer to static events in the browsing experience. Modern websites, however, are far from static, and user interactions with these websites have continuous aspects that cannot be represented by distinct events.

* How do you measure user frustration?
* How do you tell how fast your users expect your site to be?
* How do you tell if users were frustrated or annoyed before they got to your site or if your site caused that frustration?
* What are TTVR, TTI, TTFI, FID, and which of them is the most important?
* How do you tell if your page locks up during user interaction?

In this talk, we’ll look at some new user experience metrics. We’ll find out how to measure page responsiveness, smoothness, jank, and usability. We’ll learn about things like Rage Clicks, Missed Clicks, Dead Clicks, and Cursor Thrashing. We’ll also look at real user data that we’ve collected showing how these aspects of the page affect user behaviour.


Philip Tellis

May 22, 2019


  1. Metrics that Matter Measuring the effects of website performance on

  2. Philip Tellis Principal RUM Distiller @ Akamai Author of the

    OpenSource boomerang RUM library twitter:@bluesmoon ⦿ github:@bluesmoon speakerdeck:@bluesmoon
  3. Abbé Jean-Antoine Nollet 1700 - 1770 French Clergyman & Budding

    Electrician Invented one of the first Electroscopes (we now call them beacon collectors) L'Abbé Jean Antoine Nollet — Maurice Quentin de La Tour Alte Pinakothek, Munich, Germany Public Domain
  4. In 1746, he conducted the first ever RUM Experiment He

    shot an electric current through 200 monks, and checked how quickly they jumped; thereby measuring the latency of an electric signal with… Real Users!
  5. Fortunately, our methods have gotten a lot less intrusive

  6. What should we measure? What are the metrics that matter

    to users?
  7. Loading a Webpage... https://addyosmani.com/blog/usability/

  8. These moments are related and trigger different expectations with the

  9. Is it happening? What visual cues does the user have

    that this page might actually show up? That initial period of a page load when the screen goes blank causes anxiety & doubt for the user… • Did I click correctly? • Is it loading? • Has is already billed my credit card?
  10. First Paint Framework Ready Hero Images DOM Content Loaded Visually

    Ready https://www.slideshare.net/nicjansma/reliably-measuring-responsiveness
  11. When does the first thing show up on screen? •

    Register a PerformanceObserver before the event happens • OR use performance.getEntriesByType("paint") after the event https://developer.mozilla.org/en-US/docs/Web/API/PerformancePaintTiming Currently only available in Chrome and Chrome based browsers
  12. What about other browsers? Internet Explorer & Pre-Chrome Edge NavigationTiming

    on IE includes a proprietary msFirstPaint property. DOM Events We can look at when domInteractive fires, using NavigationTiming or by listening for readystatechange. Hero Images/Elements We could use a MutationObserver to determine when a particular image or element shows up in the DOM, or use ResourceTiming to check for it later. UserTiming Page or framework authors can use the UserTiming API to specify when important items are available.
  13. Last Painted Hero max(h1, (biggest_img || bg_img)) • Developed by

    Steve Souders • Named by Paul Irish It’s a synthetic measure of when the last of the hero elements was drawn on screen. https://twitter.com/vazac/status/979147191896784897 https://speedcurve.com/blog/last-painted-hero/
  14. Is it Useful? Is this the page I’m looking for?

    This stage… • Removes doubt • Confirms that the user is on the right path
  15. Signals to detect usefulness • First Meaningful Paint – Lighthouse

    only at the moment • Hero Element timing is something that can be set by page developers using User Timing or Resource Timing. • Boomerang aggregates all of these into a measure called Time to Visually Ready or ttvr. https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html https://developers.google.com/web/fundamentals/performance/user-centric-performance-metrics
  16. Is it Usable? If the user tried to interact with

    the page, would they have a good experience? The user is ready to be delighted at this point, they’ve come to the right place... • But is the page responsive? • Is there jank? • Are there random lockups? • Is the interface confusing?
  17. Not Interactive Until Here User Sees Content Time to Interactive

  18. Signals to detect usability After Visually Ready (ie, user thinks

    they can use the page), check... • Frame Rate using requestAnimationFrame • Long Tasks using PerformanceObserver or use setInterval on older browsers • Measure latency of responding to any user input (click, keypress, scroll) First period of 500ms after VR with no Long Tasks, FPS > 20, and Page Busy < 10% https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html https://developers.google.com/web/updates/2018/05/first-input-delay
  19. Short Tasks Long Tasks https://developers.google.com/web/fundamentals/performance/user-centric-performance-metrics

  20. Long Task Duration https://www.slideshare.net/nicjansma/reliably-measuring-responsiveness

  21. LongTasks as % of Front End Load Time https://www.slideshare.net/nicjansma/reliably-measuring-responsiveness

  22. Check Frame Rate using requestAnimationFrame var frames = 0; function

    measureFps() { frames++; // request a callback before the next frame window.requestAnimationFrame(measureFps); } // start measuring window.requestAnimationFrame(measureFps); // report on frame rate (FPS) once a second setInterval(function() { console.log("FPS: " + frames); frames = 0; }, 1000); https://www.slideshare.net/nicjansma/measuring-continuity
  23. Other important continuity metrics • Measure Battery Level using navigator.getBattery(...)

    • Measure memory usage using performance.memory.usedJSHeapSize • Check visibilityState and use IntersectionObserver to stop expensive tasks when not visible. • Capture JavaScript errors
  24. Is it delightful? When the user does interact, is that

    interaction pleasant? • Were interactions smooth? • Was the interface responsive? • Was the interface consistent?
  25. Signals to detect Delight Developer Signals • Long Tasks come

    up again • As does Frame Rate and input latency • Measure these continuously while the page is active User Signals • Rage Clicks • Cursor Thrashing / Wild Mouse • Scrandom
  26. • Ask the user • Bounce / Conversion Rate /

    LD 50 • Rage Clicks / Cursor Thrashing / Wild Mouse • Facial analysis • Wireless Brain Interface Methods of measuring Emotion (with JavaScript)
  27. Rage Clicks occur when users rapid-fire click (or tap) on

    your site or app. Rage clicking is the digital equivalent of cursing to release frustration. https://www.fullstory.com/resources/guide-to-understanding-frustrating-user-experiences-online/ https://www.psychologytoday.com/us/blog/hide-and-seek/201205/hell-yes-the-7-best-reasons-swearing Rage Clicks
  28. Measuring Rage Clicks Rage Clicks are measured in JavaScript by

    looking for more than 2 clicks within the same 10 pixels in a short period of time. onMouse*, event.timestamp, and an array https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html
  29. Rage Clicks depend on Latency of Page Usability • Interacting

    with a page before onload OR interactive is the most common cause of Rage Clicks • Some Rage Clicks happen after the page is usable, possibly due to JavaScript errors or CPU intensive tasks; they could also be false positives. • In over 30% of cases, a page is interactive after onload fires, and in 15% of cases, users try interacting between onload and interactive. Data collected and analysed with boomerang and Akamai mPulse
  30. Data collected and analysed with boomerang and Akamai mPulse Rage

    Clicking is fairly consistent if first Interaction is before the page becomes Interactive Steady drop off if first Interaction is after the page becomes Interactive Rage Clicks are most likely if first Interaction is just before onload, possibly because DOM Ready event handlers are hogging CPU. (This is true even if the page becomes Interactive before onload)
  31. Data collected and analysed with boomerang and Akamai mPulse Desktop

    vs Mobile
  32. Rage Clicks as a Function of First Interaction & Visually

    REady Data collected and analysed with boomerang and Akamai mPulse
  33. People who are angry are more likely to use the

    mouse in a jerky and sudden, but surprisingly slow fashion. People who feel frustrated, confused or sad are less precise in their mouse movements and move it at different speeds. https://www.telegraph.co.uk/technology/news/12050481/Websites-could-read-emotions-by-seeing-how-fast-you-move-your-mouse.html https://conversionxl.com/blog/user-frustration/ https://www.academia.edu/3085041/Patterns_of_cursor_movement_for_different_devices Cursor Thrashing/Wild Mouse
  34. Measuring Cursor Thrashing & Wild Mouse Cursor Thrashing is measured

    in JavaScript by looking at mouse movements over time, both in absolute terms and as a fraction of screen size. onMouseMove, setTimeout, event.clientX, event.clientY, ... https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html
  35. We’re still working on correlating this with performance and business

  36. Optimizing user centric Metrics • Remove render blocking scripts from

    the HEAD • Inline critical CSS • Reduce work done in JavaScript • Move non-critical JavaScript to after onload • Split up long tasks • If you can, move expensive JavaScript over to Web Workers
  37. Summary • Measure what affects your user’s emotional state. •

    Reduce the amount of work you do in onload and readyState handlers. • Measure Responsiveness and Smoothness of your UI. Pacific Islander Navigation Map, Museum of Fine Arts, Boston https://www.flickr.com/photos/bluesmoon/1266590108/
  38. • Addy Osmani on Usability • Reliably Measuring Responsiveness in

    the Wild • Last Painted Hero • Boomerang Continuity Plugin • User Centric Performance Metrics • First Input Delay • Measuring Continuity • Understanding Frustrating User Experiences • The 7 Best Reasons for Swearing • Reading Emotions by Measuring Mouse Movements • Your users are Frustrated • Patterns of Cursor Movement for Different Devices • Tracking CPU with the Long Tasks API References • Paint Timing API • Navigation Timing • Resource Timing • User Timing • MutationObserver • PerformanceObserver • msFirstPaint • readystatechange • requestAnimationFrame • setInterval / requestIdleCallback • Lighthouse • Long Tasks
  39. Thank You!