How to be a Performance Detective - Performance.now() Conference 2022

When starting on a web performance journey, many companies’ first step is to start monitoring performance metrics over time. But the second step is often much harder - now that you have all of this data, what do you do when you see a regression? How do you sift through the clues to connect a change in a graph to the code that engineers ship to production? How do you become a performance detective?

In this session, Katie will crack open her casefiles to share some real-world examples where performance metrics changed for the worse, and walk through how to find the culprit. You’ll learn what clues to look for to understand a regression in your Core Web Vitals, how different performance metrics influence each other, and when to use data gathered from real users vs. synthetic tests in the lab.

Transcript

1. DETECTIVE DETECTIVE H O W T O B E A E P o FRM Nc R e a Katie Sylor-Miller Frontend Architect, Etsy Fairy Gitmother, OhShitGit.com @ksylor Detective Pikachu © 2019 Warner Bros., Pokémon © 2022 Pokémon. © 1995–2022 Nintendo/Creatures Inc./GAME FREAK inc.
2. WHY DID THIS PERFORMANCE METRIC GET WORSE?
3. HARD TRUTH #1 EVERY ADDITIONAL BYTE WILL SLOW METRICS DOWN
4. HARD TRUTH #2 IT DEPENDS
5. HARD TRUTH #3 THERE'S NO ONE RIGHT WAY TO MEASURE PERFORMANCE
6. PERFORMANCE METRIC TYPES Icons By Andreuvv - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=90073970 @ksylor
7. PERFORMANCE METRIC TYPES WHAT HOW WHERE Primary Diagnostic Timeline-based Independent Field/Real User Lab/Synthetic @ksylor
8. PERFORMANCE METRIC TYPES HOW ARE THEY MEASURED? Timeline-based Independent vs @ksylor
9. PERFORMANCE METRIC TYPES - TIMELINE-BASED VS. INDEPENDENT TIMELINE-BASED METRICS Start a timer at 0, log a timeline of events as the page loads Navigation Start TTFB FCP LCP DCL TTI Page Load 0ms 200ms 300ms 400ms 500ms 600ms 700ms @ksylor
10. PERFORMANCE METRIC TYPES - TIMELINE-BASED VS. INDEPENDENT TIMELINE-BASED METRICS Start a timer at 0, log a timeline of events as the page loads Early slowdowns impact later metrics TTFB FCP TTI Page Load 0ms 200ms 300ms 400ms 500ms 600ms 700ms TTFB FCP TTI Page Load 0ms 200ms 300ms 400ms 500ms 600ms 700ms 800m @ksylor LCP DCL LCP DCL Navigation Start Navigation Start
11. PERFORMANCE METRIC TYPES - TIMELINE-BASED VS. INDEPENDENT INDEPENDENT METRICS Events that are measured independent of the loading timeline 0ms 200ms 300ms 400ms 500ms 600ms 700ms 150ms FID / INP / TBT @ksylor
12. PERFORMANCE METRIC TYPES - TIMELINE-BASED VS. INDEPENDENT INDEPENDENT METRICS Events that are measured independent of the loading timeline 150ms FID / INP / TBT 0ms 200ms 300ms 400ms 500ms 600ms 700ms 0.05 Layout Shift 0.125 Layout Shift 0ms 200ms 300ms 400ms 500ms 600ms 700ms 0.175 CLS @ksylor
13. PERFORMANCE METRIC TYPES WHERE ARE THEY MEASURED? Field/Real User Lab/Synthetic vs @ksylor
14. PERFORMANCE METRIC TYPES - FIELD/REAL USER VS. LAB/SYNTHETIC THE METRIC VENN DIAGRAM Field/Real User Lab/Synthetic Icons By Andreuvv - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=90073970 @ksylor FID TBT TTFB TTI Speed Index FCP LCP CLS DCL INP Page Load CPU Usage Start Render Resource Timings User Timings
15. PERFORMANCE METRIC TYPES - FIELD/REAL USER VS. LAB/SYNTHETIC REAL USER MONITORING (RUM) ✦ Collect data from Real User sessions via browser-exposed APIs ✦ Available via: ✦ Third-party monitoring services ✦ Public datasets Chrome User Experience (CrUX), Akamai RumArchive ✦ Open source JS libraries ✦ or Do-It-Yourself @ksylor
16. PERFORMANCE METRIC TYPES - FIELD/REAL USER VS. LAB/SYNTHETIC REAL USER MONITORING (RUM) ✦ Cool stuff: ✦ Captures the full story of your real users' experiences ✦ Interaction-based metrics like FID and INP are only available in RUM @ksylor
17. PERFORMANCE METRIC TYPES - FIELD/REAL USER VS. LAB/SYNTHETIC REAL USER MONITORING (RUM) ✦ Cool stuff: ✦ Captures the full story of your real users' experiences ✦ Interaction-based metrics like FID and INP are only available in RUM ✦ Meh stuff: ✦ Browser support lacking for some metrics ✦ Every data point is a tradeoff ✦ Good for trends, not in-depth analysis @ksylor
18. PERFORMANCE METRIC TYPES - FIELD/REAL USER VS. LAB/SYNTHETIC RUM POPULATIONS ✦ Performance data is variable and follows a log-normal distribution ✦ Cumulative Distribution Function (CDF) ✦ How do you compare a continuum of populations? Industry standard of p75 @ksylor
19. PERFORMANCE METRIC TYPES - FIELD/REAL USER VS. LAB/SYNTHETIC SYNTHETIC ✦ Collect data from a page load in controlled conditions ✦ Available via: ✦ Webpagetest.org or Lighthouse ✦ In the browser or programmatically ✦ Continuous monitoring or in-the-moment @ksylor
20. PERFORMANCE METRIC TYPES - FIELD/REAL USER VS. LAB/SYNTHETIC SYNTHETIC ✦ Cool stuff: ✦ Apples-to-apples comparison for diagnosis ✦ In-depth metrics not easily available from RUM ✦ Available in pre-prod @ksylor
21. PERFORMANCE METRIC TYPES - FIELD/REAL USER VS. LAB/SYNTHETIC SYNTHETIC ✦ Cool stuff: ✦ Apples-to-apples comparison for diagnosis ✦ In-depth metrics not easily available from RUM ✦ Available in pre-prod ✦ Meh stuff: ✦ Not every metric is available, but there are correlated metrics (e.g. TBT or TTI instead of FID or INP) ✦ Doesn't capture interactions @ksylor
22. PERFORMANCE METRIC TYPES WHAT DO THEY MEASURE? Primary Diagnostic vs @ksylor
23. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC PRIMARY METRICS - WHAT IS MOST IMPORTANT TO USERS? Icons By Andreuvv - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=90073970 @ksylor LCP Largest Contentful Paint FID First Input Delay CLS Cumulative Layout Shift
24. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC PRIMARY METRICS - WHAT IS MOST IMPORTANT TO YOUR USERS? LCP Largest Contentful Paint FID First Input Delay CLS Cumulative Layout Shift FCP First Contentful Paint ??? Your Important Metric TTFB Time to First Byte @ksylor Icons By Andreuvv - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=90073970
25. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC DIAGNOSTIC METRICS - WHAT CAUSED A CHANGE? Navigation Timing Resource Timing LCP/CLS Attribution Synthetic-only Metrics Long Task Timing Custom Timings Icons By Andreuvv - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=90073970 @ksylor
26. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC DIAGNOSTIC METRICS - NAVIGATION TIMING ✦ E.g. the old school metrics ✦ Before Fetch ✦ Connection Time ✦ Time to First Byte ✦ DOM Content Loaded ✦ Page Load Image via MDN https://developer.mozilla.org/en-US/docs/Learn/Performance/Measuring_performance @ksylor
27. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC DIAGNOSTIC METRICS - RESOURCE TIMING IN SYNTHETIC ✦ What resources loaded? ✦ In what order? ✦ How big were they? @ksylor
28. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC DIAGNOSTIC METRICS - RESOURCE TIMING IN RUM Track the count and sizes of all resources: @ksylor ✦ HTML ✦ CSS ✦ JS ✦ Images ✦ Videos ✦ Fonts ✦ XHR requests
29. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC DIAGNOSTIC METRICS - LCP/CLS ATTRIBUTION IN SYNTHETIC @ksylor ✦ Which elements shifted? ✦ What was the LCP element?
30. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC DIAGNOSTIC METRICS - LCP/CLS ATTRIBUTION IN RUM @ksylor
31. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC DIAGNOSTIC METRICS - LONG TASKS IN SYNTHETIC ✦ How many Long Tasks? ✦ When did they start? @ksylor
32. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC DIAGNOSTIC METRICS - LONG TASKS IN RUM ✦ Log count and duration ✦ Sum of duration = proxy for TBT ✦ Attribution is per container so useful only for 1st vs. 3rd party tracking but not much else @ksylor
33. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC DIAGNOSTIC METRICS - CUSTOM TIMINGS ✦ Make your own timings! ✦ Webpagetest will report your custom timings in the UI ✦ Full cross-browser support @ksylor
34. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC DIAGNOSTIC METRICS - SYNTHETIC-ONLY METRICS ✦ JS Execution details ✦ Total blocking time (TBT) ✦ Time to Interactive (TTI) ✦ CPU usage ✦ Processing breakdown (script parse/eval/render/layout/paint etc.) @ksylor
35. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC DIAGNOSTIC METRICS - SYNTHETIC-ONLY METRICS ✦ JS Execution details ✦ Total blocking time (TBT) ✦ Time to Interactive (TTI) ✦ CPU usage ✦ Processing breakdown (script parse/eval/render/layout/paint etc.) ✦ Rendering details ✦ Filmstrip screenshots ✦ Start Render ✦ Speed Index ✦ Visually Complete @ksylor
36. TIME TO CATCH SOME PERF METRICS! @ksylor
37. HTTPS://REPLICATE.COM/LAMBDAL/TEXT-TO-POKEMON @ksylor
38. Emcee and tall person Tim Kadlec TRAINER Emcee and running enthusiast Henri Helvetica TRAINER HTTPS://REPLICATE.COM/LAMBDAL/TEXT-TO-POKEMON @ksylor
39. Timeline-based Primary - Core Web Vitals RUM and Synthetic Largest Contentful Paint LCP CASE FILES @ksylor
40. CASE FILES: LARGEST CONTENTFUL PAINT (LCP) WHO KILLED LCP? @ksylor
41. CASE FILES: LARGEST CONTENTFUL PAINT (LCP) BEFORE (OFF) AFTER (ON) Aha! The design of the likely LCP element changed! @ksylor
42. CASE FILES: LARGEST CONTENTFUL PAINT (LCP) DIAGNOSING CHANGES TO LCP 1. Did the LCP element change? @ksylor
43. TRAINER WEBPAGETEST TRAINER CARD Use to run synthetic tests on a real browser and device Webpagetest.org @ksylor
44. CASE FILES: LARGEST CONTENTFUL PAINT (LCP) GOTCHA! This is not the LCP element you are looking for @ksylor
45. TAKE 2 The LCP Element did not change CASE FILES: LARGEST CONTENTFUL PAINT (LCP) @ksylor
46. DIAGNOSING CHANGES TO LCP 1. Did the LCP element change? NO 2. Did some characteristic of the LCP element change? - If image or video, did the asset size get bigger? - (If text, did the font or text change?) CASE FILES: LARGEST CONTENTFUL PAINT (LCP) @ksylor
47. BEFORE (OFF) AFTER (ON) CASE FILES: LARGEST CONTENTFUL PAINT (LCP) Smaller Image size @ksylor
48. BEFORE (OFF) AFTER (ON) CASE FILES: LARGEST CONTENTFUL PAINT (LCP) Smaller Image size @ksylor Hint to the real culprit!
49. DIAGNOSING CHANGES TO LCP 1. Did the LCP element change? NO 2. Did some characteristic of the LCP element change? NO - If image or video, did the asset size get bigger? - (If text, did the font or text change?) 3. Did metrics before LCP in the timeline change? - e.g. TTFB, FCP, DCL? CASE FILES: LARGEST CONTENTFUL PAINT (LCP) @ksylor
50. RUM TRAINER CARD Collect data from real user sessions Real User Monitoring TRAINER @ksylor
51. Timeline-based Primary (not CWV) RUM and Synthetic Time to First Byte TTFB SUSPECT #1 @ksylor
52. Timeline-based Primary (not CWV) RUM and Synthetic Time to First Byte TTFB CASE FILES @ksylor
53. Timeline-based Primary (not CWV) RUM and Synthetic First Contentful Paint FCP SUSPECT #2 @ksylor
54. Timeline-based Primary (not CWV) RUM and Synthetic First Contentful Paint FCP CASE FILES @ksylor
55. DOM Content Loaded DCL SUSPECT #3 Timeline-based Diagnostic RUM and Synthetic @ksylor
56. DOM Content Loaded DCL CASE FILES Timeline-based Diagnostic RUM and Synthetic @ksylor
57. Page Load PAGE LOAD SUSPECT #4 Timeline-based Diagnostic RUM and Synthetic @ksylor
58. Page Load PAGE LOAD CASE FILES Timeline-based Diagnostic RUM and Synthetic @ksylor
59. DIAGNOSING CHANGES TO LCP 1. Did the LCP element change? NO 2. Did some characteristic of the LCP element change? NO - If image or video, did the asset size change? - (If text, did the font or text change?) 3. Did metrics before LCP in the timeline change? NO - e.g. TTFB, FCP, DCL? 4. Did the loading order change? CASE FILES: LARGEST CONTENTFUL PAINT (LCP) @ksylor
60. BEFORE (OFF) AFTER (ON) LCP Element #12 CASE FILES: LARGEST CONTENTFUL PAINT (LCP) @ksylor
61. BEFORE (OFF) AFTER (ON) LCP Element #12 LCP Element #13 A wild CSS file appears! CASE FILES: LARGEST CONTENTFUL PAINT (LCP) @ksylor
62. DIAGNOSING CHANGES TO LCP 1. Did the LCP element change? NO 2. Did some characteristic of the LCP element change? NO - If image or video, did the asset size change? - (If text, did the font or text change?) 3. Did metrics before LCP in the timeline change? NO - e.g. TTFB, FCP, DCL? 4. Did the loading order change? YES! 5. Did render-blocking assets grow in size or number? CASE FILES: LARGEST CONTENTFUL PAINT (LCP) @ksylor
63. CHECK YOUR REQUESTS CASE FILES: LARGEST CONTENTFUL PAINT (LCP) +1 request Double the bytes :( @ksylor
64. DIAGNOSING CHANGES TO LCP 1. Did the LCP element change? NO 2. Did some characteristic of the LCP element change? NO - If image or video, did the asset size change? - (If text, did the font or text change?) 3. Did metrics before LCP in the timeline change? NO - e.g. TTFB, FCP, DCL? 4. Did the loading order change? YES! 5. Did render-blocking assets grow in size or number? YES! CASE FILES: LARGEST CONTENTFUL PAINT (LCP) @ksylor
65. Independent Primary - Core Web Vitals Cumulative Layout Shift CLS CASE FILES RUM and Synthetic @ksylor
66. CASE FILES: CUMULATIVE LAYOUT SHIFT (CLS) WHO KILLED CLS? @ksylor
67. BEFORE (OFF) AFTER (ON) Aha! A new module above the fold CASE FILES: CUMULATIVE LAYOUT SHIFT (CLS) @ksylor
68. DIAGNOSING CLS 1. What elements are shifting? A. Font loading FOUT/FOIT? B. Images without dimensions? C. Lazy-loaded content without a correct height placeholder? D. Injected banners, ads or iframes? CASE FILES: CUMULATIVE LAYOUT SHIFT (CLS) @ksylor
69. LIGHTHOUSE TRAINER CARD Use to do a quick in-browser synthetic test Lighthouse TRAINER @ksylor
70. BEFORE (OFF) AFTER (ON) CASE FILES: CUMULATIVE LAYOUT SHIFT (CLS) @ksylor
71. BEFORE (OFF) AFTER (ON) CASE FILES: CUMULATIVE LAYOUT SHIFT (CLS) @ksylor
72. ZOOMING IN ON AFTER (ON) CASE FILES: CUMULATIVE LAYOUT SHIFT (CLS) Everything under the new module @ksylor
73. WHAT HAPPENED? CASE FILES: CUMULATIVE LAYOUT SHIFT (CLS) @ksylor
74. DIAGNOSING CLS 1. What elements are shifting? A. Font loading FOUT/FOIT? NO B. Images without dimensions? NO C. Lazy-loaded content without a correct height placeholder? YES! D. Injected banners, ads or iframes? NO CASE FILES: CUMULATIVE LAYOUT SHIFT (CLS) @ksylor
75. Independent Primary - Core Web Vitals RUM First Input Delay FID CASE FILES @ksylor
76. CASE FILES: FIRST INPUT DELAY (FID) WHO KILLED FID? @ksylor
77. AFTER (ON) This new button loads CASE FILES: FIRST INPUT DELAY (FID) @ksylor
78. DIAGNOSING CHANGES TO FID 1. Is there more Javascript execution, e.g. Long Tasks? CASE FILES: FIRST INPUT DELAY (FID) @ksylor
79. Long Tasks LONG TASKS SUSPECT #1 Timeline-based Diagnostic RUM and Synthetic @ksylor
80. MORE LONG TASKS MORE PROBLEMS CASE FILES: FIRST INPUT DELAY (FID) @ksylor
81. DIAGNOSING CHANGES TO FID 1. Is there more Javascript execution, e.g. Long Tasks? YES! 2. Is there more Javascript downloading? - Check both first- and third-party CASE FILES: FIRST INPUT DELAY (FID) @ksylor
82. Resource Timing RESOURCE TIMING SUSPECT #2 Timeline-based Diagnostic RUM and Synthetic @ksylor
83. JS RESOURCES CASE FILES: FIRST INPUT DELAY (FID) @ksylor
84. DIAGNOSING CHANGES TO FID 1. Is there more Javascript execution, e.g. Long Tasks? YES! 2. Is there more Javascript downloading? NO? I think? - Check both first- and third-party 3. Did TBT or TTI get worse? CASE FILES: FIRST INPUT DELAY (FID) @ksylor
85. Independent Primary - Core Web Vitals? Total Blocking Time TBT CASE FILES Synthetic @ksylor
86. Timeline-based Primary Time to Interactive TTI CASE FILES Synthetic @ksylor
87. TBT OR TTI? ✦ ~30 Synthetic tests were all inconclusive :( CASE FILES: FIRST INPUT DELAY (FID) @ksylor
88. DIAGNOSING CHANGES TO FID 1. Is there more Javascript execution, e.g. Long Tasks? YES! 2. Is there more Javascript downloading? NO? I think? - Check both first- and third-party 3. Did TBT or TTI get worse? NO 4. Did user behavior change? IDK CASE FILES: FIRST INPUT DELAY (FID) @ksylor
89. DIAGNOSING CHANGES TO FID 1. Is there more Javascript execution, e.g. Long Tasks? YES! 2. Is there more Javascript downloading? NO? I think? - Check both first- and third-party 3. Did TBT or TTI get worse? NO 4. Did user behavior change? IDK 5. Can we get FID attribution please Yoav FFS CASE FILES: FIRST INPUT DELAY (FID) @ksylor
90. Independent Primary - Core Web Vitals? Interaction to Next Paint INP CASE FILES RUM @ksylor ?????
91. WHAT DID WE LEARN? IT ALL STARTS WITH MONITORING @ksylor
92. WHAT DID WE LEARN? PERF DETECTING ISN'T MAGIC askastaffengineer.com @ksylor
93. WHAT DID WE LEARN? IT DEPENDS askastaffengineer.com @ksylor
94. ! U Y k o T @ksylor hAn