Metrics that Matter Measuring the effects of website performance on users

Philip Tellis Principal RUM Distiller @ Akamai Author of the OpenSource boomerang RUM library twitter:@bluesmoon ⦿ github:@bluesmoon speakerdeck:@bluesmoon

Abbé Jean-Antoine Nollet 1700 - 1770 French Clergyman & Budding Electrician Invented one of the first Electroscopes (we now call them beacon collectors) L'Abbé Jean Antoine Nollet — Maurice Quentin de La Tour Alte Pinakothek, Munich, Germany Public Domain

In 1746, he conducted the first ever RUM Experiment He shot an electric current through 200 monks, and checked how quickly they jumped; thereby measuring the latency of an electric signal with… Real Users!

Fortunately, our methods have gotten a lot less intrusive

What should we measure? What are the metrics that matter to users?

Loading a Webpage... https://addyosmani.com/blog/usability/

These moments are related and trigger different expectations with the user

Is it happening? What visual cues does the user have that this page might actually show up? That initial period of a page load when the screen goes blank causes anxiety & doubt for the user… ● Did I click correctly? ● Is it loading? ● Has is already billed my credit card?

First Paint Framework Ready Hero Images DOM Content Loaded Visually Ready https://www.slideshare.net/nicjansma/reliably-measuring-responsiveness

When does the first thing show up on screen? ● Register a PerformanceObserver before the event happens ● OR use performance.getEntriesByType("paint") after the event https://developer.mozilla.org/en-US/docs/Web/API/PerformancePaintTiming Currently only available in Chrome and Chrome based browsers

What about other browsers? Internet Explorer & Pre-Chrome Edge NavigationTiming on IE includes a proprietary msFirstPaint property. DOM Events We can look at when domInteractive fires, using NavigationTiming or by listening for readystatechange. Hero Images/Elements We could use a MutationObserver to determine when a particular image or element shows up in the DOM, or use ResourceTiming to check for it later. UserTiming Page or framework authors can use the UserTiming API to specify when important items are available.

Last Painted Hero max(h1, (biggest_img || bg_img)) ● Developed by Steve Souders ● Named by Paul Irish It’s a synthetic measure of when the last of the hero elements was drawn on screen. https://twitter.com/vazac/status/979147191896784897 https://speedcurve.com/blog/last-painted-hero/

Is it Useful? Is this the page I’m looking for? This stage… ● Removes doubt ● Confirms that the user is on the right path

Signals to detect usefulness ● First Meaningful Paint – Lighthouse only at the moment ● Hero Element timing is something that can be set by page developers using User Timing or Resource Timing. ● Boomerang aggregates all of these into a measure called Time to Visually Ready or ttvr. https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html https://developers.google.com/web/fundamentals/performance/user-centric-performance-metrics

Is it Usable? If the user tried to interact with the page, would they have a good experience? The user is ready to be delighted at this point, they’ve come to the right place... ● But is the page responsive? ● Is there jank? ● Are there random lockups? ● Is the interface confusing?

Not Interactive Until Here User Sees Content Time to Interactive https://www.slideshare.net/nicjansma/reliably-measuring-responsiveness

Signals to detect usability After Visually Ready (ie, user thinks they can use the page), check... ● Frame Rate using requestAnimationFrame ● Long Tasks using PerformanceObserver or use setInterval on older browsers ● Measure latency of responding to any user input (click, keypress, scroll) First period of 500ms after VR with no Long Tasks, FPS > 20, and Page Busy < 10% https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html https://developers.google.com/web/updates/2018/05/first-input-delay

Short Tasks Long Tasks https://developers.google.com/web/fundamentals/performance/user-centric-performance-metrics

Long Task Duration https://www.slideshare.net/nicjansma/reliably-measuring-responsiveness

LongTasks as % of Front End Load Time https://www.slideshare.net/nicjansma/reliably-measuring-responsiveness

Check Frame Rate using requestAnimationFrame var frames = 0; function measureFps() { frames++; // request a callback before the next frame window.requestAnimationFrame(measureFps); } // start measuring window.requestAnimationFrame(measureFps); // report on frame rate (FPS) once a second setInterval(function() { console.log("FPS: " + frames); frames = 0; }, 1000); https://www.slideshare.net/nicjansma/measuring-continuity

Other important continuity metrics ● Measure Battery Level using navigator.getBattery(...) ● Measure memory usage using performance.memory.usedJSHeapSize ● Check visibilityState and use IntersectionObserver to stop expensive tasks when not visible. ● Capture JavaScript errors

Is it delightful? When the user does interact, is that interaction pleasant? ● Were interactions smooth? ● Was the interface responsive? ● Was the interface consistent?

Signals to detect Delight Developer Signals ● Long Tasks come up again ● As does Frame Rate and input latency ● Measure these continuously while the page is active User Signals ● Rage Clicks ● Cursor Thrashing / Wild Mouse ● Scrandom

● Ask the user ● Bounce / Conversion Rate / LD 50 ● Rage Clicks / Cursor Thrashing / Wild Mouse ● Facial analysis ● Wireless Brain Interface Methods of measuring Emotion (with JavaScript)

Rage Clicks occur when users rapid-fire click (or tap) on your site or app. Rage clicking is the digital equivalent of cursing to release frustration. https://www.fullstory.com/resources/guide-to-understanding-frustrating-user-experiences-online/ https://www.psychologytoday.com/us/blog/hide-and-seek/201205/hell-yes-the-7-best-reasons-swearing Rage Clicks

Measuring Rage Clicks Rage Clicks are measured in JavaScript by looking for more than 2 clicks within the same 10 pixels in a short period of time. onMouse*, event.timestamp, and an array https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html

Rage Clicks depend on Latency of Page Usability ● Interacting with a page before onload OR interactive is the most common cause of Rage Clicks ● Some Rage Clicks happen after the page is usable, possibly due to JavaScript errors or CPU intensive tasks; they could also be false positives. ● In over 30% of cases, a page is interactive after onload fires, and in 15% of cases, users try interacting between onload and interactive. Data collected and analysed with boomerang and Akamai mPulse

Data collected and analysed with boomerang and Akamai mPulse Rage Clicking is fairly consistent if first Interaction is before the page becomes Interactive Steady drop off if first Interaction is after the page becomes Interactive Rage Clicks are most likely if first Interaction is just before onload, possibly because DOM Ready event handlers are hogging CPU. (This is true even if the page becomes Interactive before onload)

Data collected and analysed with boomerang and Akamai mPulse Desktop vs Mobile

Rage Clicks as a Function of First Interaction & Visually REady Data collected and analysed with boomerang and Akamai mPulse

People who are angry are more likely to use the mouse in a jerky and sudden, but surprisingly slow fashion. People who feel frustrated, confused or sad are less precise in their mouse movements and move it at different speeds. https://www.telegraph.co.uk/technology/news/12050481/Websites-could-read-emotions-by-seeing-how-fast-you-move-your-mouse.html https://conversionxl.com/blog/user-frustration/ https://www.academia.edu/3085041/Patterns_of_cursor_movement_for_different_devices Cursor Thrashing/Wild Mouse

Measuring Cursor Thrashing & Wild Mouse Cursor Thrashing is measured in JavaScript by looking at mouse movements over time, both in absolute terms and as a fraction of screen size. onMouseMove, setTimeout, event.clientX, event.clientY, ... https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html

We’re still working on correlating this with performance and business metrics

Optimizing user centric Metrics ● Remove render blocking scripts from the HEAD ● Inline critical CSS ● Reduce work done in JavaScript ● Move non-critical JavaScript to after onload ● Split up long tasks ● If you can, move expensive JavaScript over to Web Workers

Summary ● Measure what affects your user’s emotional state. ● Reduce the amount of work you do in onload and readyState handlers. ● Measure Responsiveness and Smoothness of your UI. Pacific Islander Navigation Map, Museum of Fine Arts, Boston https://www.flickr.com/photos/bluesmoon/1266590108/

● Addy Osmani on Usability ● Reliably Measuring Responsiveness in the Wild ● Last Painted Hero ● Boomerang Continuity Plugin ● User Centric Performance Metrics ● First Input Delay ● Measuring Continuity ● Understanding Frustrating User Experiences ● The 7 Best Reasons for Swearing ● Reading Emotions by Measuring Mouse Movements ● Your users are Frustrated ● Patterns of Cursor Movement for Different Devices ● Tracking CPU with the Long Tasks API References ● Paint Timing API ● Navigation Timing ● Resource Timing ● User Timing ● MutationObserver ● PerformanceObserver ● msFirstPaint ● readystatechange ● requestAnimationFrame ● setInterval / requestIdleCallback ● Lighthouse ● Long Tasks

Thank You!