How I Learned to Stop Worrying and Love UX Metrics [PerfNow 2018]

This talk at the 2018 performance.now() conference (Amsterdam) walks through a brief history of UX and web performance research, highlighting landmark studies that helped connect the dots between performance and user experience. I also demystify the current state of performance metrics and help you understand what you need to focus on for your site and your users.

Transcript

1. @tameverts #PerfNow How I learned to stop worrying and love UX metrics
2. @tameverts WPOstats.com
3. What is the best UX metric? How fast should it be? How can we stay on track?
5. https://www.cnet.com/news/appliance-science-the-well-done-physics-chemistry-of-the-toaster/
9. TTFB DNS TCP Start render DOM loading DOM ready Page load Fully loaded TTI Resource timing Number of requests Bytes in Speed Index PageSpeed Navigation timing DOM elements DOM size Visually complete TTFP TTFMP
10. “The real thing we are after is to create a user experience that people love and they feel is fast… and so we might be front-end engineers, we might be dev, we might be ops, but what we really are is perception brokers.” Steve Souders
11. How do we measure perception… at scale?
12. Is it happening? Is it useful? Is it usable? Is it delightful? https://developers.google.com/web/fundamentals/ performance/user-centric-performance-metrics
14. TTFB DNS TCP TTI FCP FMP FID OMG WTF
15. ❑ Correlates to what users actually see in the browser ❑ Is easy to use and accessible right out of the box ❑ Recognizes that not all pixels and page elements are equal ❑ Allows us to customize what we measure on specific pages The best UX metric… #PerfNow @tameverts
17. A brief history of performance metrics
18. Load Time the time from the start of the initial navigation until the beginning of the window load event
19. BBC loses 10% of users for every additional second it takes their site to load Ancestry.com saw a 7% conversion rate increase after improving load time by 64% FT.com reduced desktop load times to 1.5s and mobile to 2.1s and increased user engagement by 30% @tameverts #PerfNow
20. Start Render the time from the start of the initial navigation until the first non-white content is painted
21. start render repeat visits
22. wow!
23. @tameverts #PerfNow
24. Speed Index average time at which visible parts of the page are in the viewport
25. When good metrics go bad…
30. First Paint First Contentful Paint First Meaningful Paint First CPU Idle First Interactive
31. First Paint (FP) Pixels first start to render
32. First Contentful Paint (FCP) Text and graphics start to render… BUT often catches non-meaningful paints (e.g. headers, nav bars) https://tinyurl.com/paint-metrics
33. First Meaningful Paint (FMP) The paint after which the biggest ATF layout change has happened and web fonts have loaded https://tinyurl.com/paint-metrics
35. https://speedcurve.com/blog/an-analysis-of-chromiums-paint-timing-metrics/
36. “The rendering pipeline is very complex, and the timestamp should be the latest timestamp the browser is able to note in the pipeline (best effort). Typically the time at which the frame is submitted to the OS for display is recommended for this API.” https://w3c.github.io/paint-timing/#sec-terminology
37. Analysis of 40 top Alexa-ranked sites 95% of FP events occur before Start Render 85% of FCP events occur before Start Render 50% of FMP events occur before Start Render https://speedcurve.com/blog/ an-analysis-of-chromiums-paint-timing-metrics/
38. ❑ Correlates to what users actually see in the browser ❑ Is easy to use and accessible right out of the box ❑ Recognizes that not all pixels and page elements are equal ❑ Allows us to customize what we measure on specific pages The best UX metric…
39. First CPU Idle (formerly known as First Interactive) Page is minimally interactive Most visible UI elements are interactive Responds to user input reasonably quickly
40. Time to Interactive (TTI) (formerly known as Time to Consistently Interactive) Displays useful content Event handlers are registered for most visible elements Page responds to user interaction within 50ms
42. ❑ Correlates to what users actually see in the browser ❑ Is easy to use and accessible right out of the box ❑ Recognizes that not all pixels and page elements are equal ❑ Allows us to customize what we measure on specific pages The best UX metric…
43. Okay… so then what?
44. Hero Rendering Times H1 Largest Image Largest Background Image First Painted Hero Last Painted Hero https://speedcurve.com/blog/web-performance-monitoring-hero-times/
48. One-size-fits-most (not all) Dependent on how page is built Don’t work for image carousels and popups Hero times gotchas
49. So which metric is best?
50. http://lab.speedcurve.com/rendering/picker.php
51. And the winner is…
52. It depends! (You probably saw that coming.)
53. Deliver any content? Start Render Deliver significant amount of content? Speed Index, FMP Deliver critical content? Hero Rendering Times https://speedcurve.com/blog/rendering-metrics/
54. ❑ Correlates to what users actually see in the browser ❑ Is easy to use and accessible right out of the box ❑ Recognizes that not all pixels and page elements are equal ❑ Allows us to customize what we measure on specific pages The best UX metric…
55. Custom metrics Measure performance with high-precision timestamps Available in both synthetic and RUM (yay!) https://www.w3.org/TR/user-timing/ https://speedcurve.com/blog/user-timing-and-custom-metrics/
56. how long does it take to display the main product image on my site?
57. Time to First Tweet The time from clicking the link to viewing the first tweet on each page’s timeline Pinner Wait Time (PWT) The time from initiating an action (e.g., tapping a pin) until the action is complete (pin close-up view is loaded) Time to Interact (TTI) @tameverts #PerfNow
58. @tameverts #PerfNow
59. ❑ Correlates to what users actually see in the browser ❑ Is easy to use and accessible right out of the box ❑ Recognizes that not all pixels and page elements are equal ❑ Allows us to customize what we measure on specific pages The best UX metric…
61. What else can we track?
62. First Click First Scroll First Key First Input Delay (only available with real user monitoring)
65. Great! We have metrics. Now… what to do with them?
66. Performance budgets FTW!
67. Thresholds YOU create for metrics that are meaningful for YOUR site https://addyosmani.com/blog/performance-budgets/ Milestone timings (e.g. start render) Quantity-based (e.g. image weight) Rules-based (e.g. Lighthouse scores)
68. A good performance budget should show you… What your budget is When you go out of bounds How long you’re out of bounds When you’re back within budget
70. Recommended UX perf budgets* Metric How to track it Budget Start render Synthetic & RUM 2s First Painted Hero Synthetic 2s Speed Index Synthetic 4s First Interactive Synthetic 4s First Input Delay RUM 10ms @tameverts *See super important stuff on the next slide
71. Super important! Look at your own data Monitor your competitors No sandbagging allowed Take a step-by-step approach if necessary Use synthetic and RUM (numbers may will vary)
72. Pro tips Create budgets for your popular and regularly changing pages Review violations early and always Compare before and after releases Update budgets accordingly https://www.zillow.com/engineering/bigger-faster-more-engaging-budget/
73. Takeaways
74. 1. There is no unicorn metric 2. Use your eyes 3. Don’t just look at averages and medians 4. Validate your metrics by correlating them with business and engagement KPIs 5. Set targets (e.g., performance budgets) 6. Make yourself accountable (e.g., alerts) 7. Don’t be afraid to investigate new metrics! @tameverts #PerfNow
75. thanks! @tameverts speedcurve.com/blog