UX & Performance Metrics that Matter

Philip Tellis Principal RUM Distiller @ Akamai Author of the OpenSource boomerang RUM library twitter:@bluesmoon * github:@bluesmoon

The Art & Science of Real User Measurement

What should we measure?

How should we measure it?

A user’s emotional response to latency and tolerance of errors appears to correlate with their emotional state. Measuring Emotion... https://speakerdeck.com/bluesmoon/sciencing-data

Methods of measuring Emotion ● Ask the user ● Bounce / Conversion Rate / LD 50 ● Rage Clicks / Cursor Thrashing ● Facial analysis ● Wireless Brain Interface

Rage Clicks occur when users rapid-fire click (or tap) on your site or app. Rage clicking is the digital equivalent of cursing to release frustration. https://www.fullstory.com/resources/guide-to-understanding-frustrating-user-experiences-online/ https://www.psychologytoday.com/us/blog/hide-and-seek/201205/hell-yes-the-7-best-reasons-swearing Rage Clicks

People who are angry are more likely to use the mouse in a jerky and sudden, but surprisingly slow fashion. People who feel frustrated, confused or sad are less precise in their mouse movements and move it at different speeds. https://www.telegraph.co.uk/technology/news/12050481/Websites-could-read-emotions-by-seeing-how-fast-you-move-your-mouse.html Cursor Thrashing

https://webgazer.cs.brown.edu/ Facial Analysis Mind Reading https://www.emotiv.com/ We do NOT do either of these

Ask the User We do NOT do this either ● Selection Bias ● Hawthorne Effect ● Intrusive action (boomerang is passive)

Rage Clicks are measured in JavaScript by looking for more than 2 clicks within the same 10 pixels in a short period of time. onMouse*, event.timestamp, and an array The code is in boomerang’s Continuity Plugin. Measuring Rage Clicks https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html

Rage Click likelihood depends on Latency of Page Usability ● Interacting with a page before onload OR interactive is the most common cause of Rage Clicks ● Some Rage Clicks happen after the page is usable, possibly due to JavaScript errors or CPU intensive tasks; they could also be false positives. ● In over 30% of cases, a page is interactive after onload fires, and in 15% of cases, users try interacting between onload and interactive. Data collected and analysed with boomerang and Akamai mPulse

Data collected and analysed with boomerang and Akamai mPulse Rage Clicking is fairly consistent if first Interaction is before the page becomes Interactive Steady drop off if first Interaction is after the page becomes Interactive Rage Clicks are most likely if first Interaction is just before onload, possibly because DOM Ready event handlers are hogging CPU. (This is true even if the page becomes Interactive before onload)

Data collected and analysed with boomerang and Akamai mPulse

● We found that the majority of users who Rage Click attempt to interact with a page between 1.25 and 1.5x the visually ready time. ● We don’t know why, but users expect a page to be interactive at 1.3x the time when it is visual. ● So, make sure your page is interactive and loaded before 1.3x the visually ready time. What is the optimum time to reduce Rage Clicking? Data collected and analysed with boomerang and Akamai mPulse Ratio of First Interaction to Visually Ready Rage Clicks

● There is a clear linear correlation ● The slope varies between 1.2 and 1.5 Rage Clicks by First Interaction & Visually Ready Data collected and analysed with boomerang and Akamai mPulse

Cursor Thrashing is measured in JavaScript by looking at mouse movements over time, both in absolute terms and as a fraction of screen size. onMouseMove, setTimeout, event.clientX, event.clientY, ... The code is in boomerang’s Continuity Plugin. Measuring Cursor Thrashing https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html

Rage Clicks are ● A user’s response to lack of responsiveness on your site. ● Indicative of design or performance issues with your site. Key differences between Rage Clicks & Cursor Thrashing... Cursor Thrashing is ● Indicative of the user’s emotional state before they got to your site ● A signal you can use to change how you present your site.

Definitions of our Metrics that Matter https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html

Timers ● Visually Ready: When did the user feel like they could interact with the site? ● Interactive: After the page was Visually Ready, when was the first time the user could have interacted with the site, and had a good experience? ● First Interaction: When was the first time the user tried to interact with the site? ● First Input Delay: For the first interaction on the page, how responsive was it?

● Delayed Interactions: How often was the user's interaction delayed more than 50ms? ● Rage Clicks: Did the user repeatedly click on the same element/region? ● Mouse Movements: How did the user move the mouse around the screen? Interaction Metrics

Page Performance Metrics ● Frame Rate data: FPS during page load, minimum FPS, number of long frames ● Long Task data: Number of Long Tasks, how much time they took, attribution to what caused them ● Page Busy: Measurement of the page's “busyness”

How do we measure what matters? ● Use boomerang! ● Read the docs of boomerang’s Continuity Plugin ● Watch the talk by Nic Jansma and Philip Tellis at Velocity . . .

● First Paint, First Contentful Paint, DOMContentLoaded ● Hero Images (eg: class="hero-image") ● Framework Ready Event Pick the last event from the above, that is when the user thinks the page is visually ready Visually Ready

● Measure Frame Rate using requestAnimationFrame. ● Measure Long Tasks using PerformanceObserver (if available). ● Measure Page Busy using setInterval (if Long Tasks not available). ● Measure Interaction Latency First period of 500ms after Visually Ready with No Long Tasks, FPS > 20, and Page Busy < 10% Time to Interactive

● Instrument Mouse, Keyboard, and Scroll Events in passive mode ● Measure latency between event.timestamp, and event handler being called performance.now(). Delayed Interactions

● Use requestAnimationFrame ● Measure time between consecutive calls to rAF ● 16ms for the best experience This measurement is expensive on the CPU and cause excessive battery usage when the page is idle, so best to use it only during the page load process. Frame Rate

var observer = new PerformanceObserver(function(list) { var perfEntries = list.getEntries(); for (var i = 0; i < perfEntries.length; i++) { // Process long task notifications: } }); // register observer for long task notifications observer.observe({entryTypes: ["longtask"]}); Long Tasks: Use PerformanceObserver

Summary ● Measure what affects your user’s emotional state. ● Reduce the amount of work you do in onload and readyState handlers. ● Measure Responsiveness and Smoothness of your UI. Pacific Islander Navigation Map, Museum of Fine Arts, Boston https://www.flickr.com/photos/bluesmoon/1266590108/

Thank You Check out boomerang on github for much more detail

References ● Boomerang’s Continuity Plugin https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html ● Fullstory’s guide to User Frustration https://www.fullstory.com/resources/guide-to-understanding-frustrating-user-experiences-online/ ● Reading emotion through measuring mouse movements https://www.telegraph.co.uk/technology/news/12050481/Websites-could-read-emotions-by-seeing-how... ● Inferring Emotion through HCI Devices https://www.researchgate.net/publication/283422711_How_Is_Your_User_Feeling_Inferring_Emotion... ● Correlation between emotional state and browsing behaviour https://speakerdeck.com/bluesmoon/sciencing-data ● Scroll behaviour across the web http://blog.chartbeat.com/2013/08/12/scroll-behavior-across-the-web/ ● Debouncing & Throttling Events https://css-tricks.com/debouncing-throttling-explained-examples/ ● Measuring Web Page Jank https://webperf.ninja/2015/jank-meter/ ● Scroll Performance with Passive Event Listeners https://developers.google.com/web/updates/2016/06/passive-event-listeners ● What Every Browser knows about You http://webkay.robinlinus.com/ ● An Audit of Boomerang’s Performance https://nicj.net/an-audit-of-boomerangs-performance/ ● The Long Tasks API https://w3c.github.io/longtasks/ ● The Paint Timing API https://www.w3.org/TR/paint-timing/ ● Resource Timing Level 2 https://www.w3.org/TR/resource-timing-2/ ● Intersection Observer API https://developer.mozilla.org/en-US/docs/Web/API/Intersection_Observer_API ● requestIdleCallback https://developers.google.com/web/updates/2015/08/using-requestidlecallback ● requestAnimationFrame https://developer.mozilla.org/en-US/docs/Web/API/window/requestAnimationFrame ● Last Painted Hero https://speedcurve.com/blog/last-painted-hero/ ● The Runtime Performance Checklist https://calendar.perfplanet.com/2013/the-runtime-performance-checklist/