$30 off During Our Annual Pro Sale. View Details »

How to be a Performance Detective - Performance.now() Conference 2022

How to be a Performance Detective - Performance.now() Conference 2022

When starting on a web performance journey, many companies’ first step is to start monitoring performance metrics over time. But the second step is often much harder - now that you have all of this data, what do you do when you see a regression? How do you sift through the clues to connect a change in a graph to the code that engineers ship to production? How do you become a performance detective?

In this session, Katie will crack open her casefiles to share some real-world examples where performance metrics changed for the worse, and walk through how to find the culprit. You’ll learn what clues to look for to understand a regression in your Core Web Vitals, how different performance metrics influence each other, and when to use data gathered from real users vs. synthetic tests in the lab.

Katie Sylor-Miller

October 31, 2022
Tweet

Other Decks in Programming

Transcript

  1. DETECTIVE DETECTIVE H O W T O B E A

    E P o FRM Nc R e a Katie Sylor-Miller Frontend Architect, Etsy Fairy Gitmother, OhShitGit.com @ksylor Detective Pikachu © 2019 Warner Bros., Pokémon © 2022 Pokémon. © 1995–2022 Nintendo/Creatures Inc./GAME FREAK inc.
  2. WHY DID THIS PERFORMANCE METRIC GET WORSE?

  3. HARD TRUTH #1 EVERY ADDITIONAL BYTE WILL SLOW METRICS DOWN

  4. HARD TRUTH #2 IT DEPENDS

  5. HARD TRUTH #3 THERE'S NO ONE RIGHT WAY TO MEASURE

    PERFORMANCE
  6. PERFORMANCE METRIC TYPES Icons By Andreuvv - Own work, CC

    BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=90073970 @ksylor
  7. PERFORMANCE METRIC TYPES WHAT HOW WHERE Primary Diagnostic Timeline-based Independent

    Field/Real User Lab/Synthetic @ksylor
  8. PERFORMANCE METRIC TYPES HOW ARE THEY MEASURED? Timeline-based Independent vs

    @ksylor
  9. PERFORMANCE METRIC TYPES - TIMELINE-BASED VS. INDEPENDENT TIMELINE-BASED METRICS Start

    a timer at 0, log a timeline of events as the page loads Navigation Start TTFB FCP LCP DCL TTI Page Load 0ms 200ms 300ms 400ms 500ms 600ms 700ms @ksylor
  10. PERFORMANCE METRIC TYPES - TIMELINE-BASED VS. INDEPENDENT TIMELINE-BASED METRICS Start

    a timer at 0, log a timeline of events as the page loads Early slowdowns impact later metrics TTFB FCP TTI Page Load 0ms 200ms 300ms 400ms 500ms 600ms 700ms TTFB FCP TTI Page Load 0ms 200ms 300ms 400ms 500ms 600ms 700ms 800m @ksylor LCP DCL LCP DCL Navigation Start Navigation Start
  11. PERFORMANCE METRIC TYPES - TIMELINE-BASED VS. INDEPENDENT INDEPENDENT METRICS Events

    that are measured independent of the loading timeline 0ms 200ms 300ms 400ms 500ms 600ms 700ms 150ms FID / INP / TBT @ksylor
  12. PERFORMANCE METRIC TYPES - TIMELINE-BASED VS. INDEPENDENT INDEPENDENT METRICS Events

    that are measured independent of the loading timeline 150ms FID / INP / TBT 0ms 200ms 300ms 400ms 500ms 600ms 700ms 0.05 Layout Shift 0.125 Layout Shift 0ms 200ms 300ms 400ms 500ms 600ms 700ms 0.175 CLS @ksylor
  13. PERFORMANCE METRIC TYPES WHERE ARE THEY MEASURED? Field/Real User Lab/Synthetic

    vs @ksylor
  14. PERFORMANCE METRIC TYPES - FIELD/REAL USER VS. LAB/SYNTHETIC THE METRIC

    VENN DIAGRAM Field/Real User Lab/Synthetic Icons By Andreuvv - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=90073970 @ksylor FID TBT TTFB TTI Speed Index FCP LCP CLS DCL INP Page Load CPU Usage Start Render Resource Timings User Timings
  15. PERFORMANCE METRIC TYPES - FIELD/REAL USER VS. LAB/SYNTHETIC REAL USER

    MONITORING (RUM) ✦ Collect data from Real User sessions via browser-exposed APIs ✦ Available via: ✦ Third-party monitoring services ✦ Public datasets Chrome User Experience (CrUX), Akamai RumArchive ✦ Open source JS libraries ✦ or Do-It-Yourself @ksylor
  16. PERFORMANCE METRIC TYPES - FIELD/REAL USER VS. LAB/SYNTHETIC REAL USER

    MONITORING (RUM) ✦ Cool stuff: ✦ Captures the full story of your real users' experiences ✦ Interaction-based metrics like FID and INP are only available in RUM @ksylor
  17. PERFORMANCE METRIC TYPES - FIELD/REAL USER VS. LAB/SYNTHETIC REAL USER

    MONITORING (RUM) ✦ Cool stuff: ✦ Captures the full story of your real users' experiences ✦ Interaction-based metrics like FID and INP are only available in RUM ✦ Meh stuff: ✦ Browser support lacking for some metrics ✦ Every data point is a tradeoff ✦ Good for trends, not in-depth analysis @ksylor
  18. PERFORMANCE METRIC TYPES - FIELD/REAL USER VS. LAB/SYNTHETIC RUM POPULATIONS

    ✦ Performance data is variable and follows a log-normal distribution ✦ Cumulative Distribution Function (CDF) ✦ How do you compare a continuum of populations? Industry standard of p75 @ksylor
  19. PERFORMANCE METRIC TYPES - FIELD/REAL USER VS. LAB/SYNTHETIC SYNTHETIC ✦

    Collect data from a page load in controlled conditions ✦ Available via: ✦ Webpagetest.org or Lighthouse ✦ In the browser or programmatically ✦ Continuous monitoring or in-the-moment @ksylor
  20. PERFORMANCE METRIC TYPES - FIELD/REAL USER VS. LAB/SYNTHETIC SYNTHETIC ✦

    Cool stuff: ✦ Apples-to-apples comparison for diagnosis ✦ In-depth metrics not easily available from RUM ✦ Available in pre-prod @ksylor
  21. PERFORMANCE METRIC TYPES - FIELD/REAL USER VS. LAB/SYNTHETIC SYNTHETIC ✦

    Cool stuff: ✦ Apples-to-apples comparison for diagnosis ✦ In-depth metrics not easily available from RUM ✦ Available in pre-prod ✦ Meh stuff: ✦ Not every metric is available, but there are correlated metrics (e.g. TBT or TTI instead of FID or INP) ✦ Doesn't capture interactions @ksylor
  22. PERFORMANCE METRIC TYPES WHAT DO THEY MEASURE? Primary Diagnostic vs

    @ksylor
  23. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC PRIMARY METRICS -

    WHAT IS MOST IMPORTANT TO USERS? Icons By Andreuvv - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=90073970 @ksylor LCP Largest Contentful Paint FID First Input Delay CLS Cumulative Layout Shift
  24. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC PRIMARY METRICS -

    WHAT IS MOST IMPORTANT TO YOUR USERS? LCP Largest Contentful Paint FID First Input Delay CLS Cumulative Layout Shift FCP First Contentful Paint ??? Your Important Metric TTFB Time to First Byte @ksylor Icons By Andreuvv - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=90073970
  25. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC DIAGNOSTIC METRICS -

    WHAT CAUSED A CHANGE? Navigation Timing Resource Timing LCP/CLS Attribution Synthetic-only Metrics Long Task Timing Custom Timings Icons By Andreuvv - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=90073970 @ksylor
  26. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC DIAGNOSTIC METRICS -

    NAVIGATION TIMING ✦ E.g. the old school metrics ✦ Before Fetch ✦ Connection Time ✦ Time to First Byte ✦ DOM Content Loaded ✦ Page Load Image via MDN https://developer.mozilla.org/en-US/docs/Learn/Performance/Measuring_performance @ksylor
  27. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC DIAGNOSTIC METRICS -

    RESOURCE TIMING IN SYNTHETIC ✦ What resources loaded? ✦ In what order? ✦ How big were they? @ksylor
  28. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC DIAGNOSTIC METRICS -

    RESOURCE TIMING IN RUM Track the count and sizes of all resources: @ksylor ✦ HTML ✦ CSS ✦ JS ✦ Images ✦ Videos ✦ Fonts ✦ XHR requests
  29. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC DIAGNOSTIC METRICS -

    LCP/CLS ATTRIBUTION IN SYNTHETIC @ksylor ✦ Which elements shifted? ✦ What was the LCP element?
  30. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC DIAGNOSTIC METRICS -

    LCP/CLS ATTRIBUTION IN RUM @ksylor
  31. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC DIAGNOSTIC METRICS -

    LONG TASKS IN SYNTHETIC ✦ How many Long Tasks? ✦ When did they start? @ksylor
  32. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC DIAGNOSTIC METRICS -

    LONG TASKS IN RUM ✦ Log count and duration ✦ Sum of duration = proxy for TBT ✦ Attribution is per container so useful only for 1st vs. 3rd party tracking but not much else @ksylor
  33. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC DIAGNOSTIC METRICS -

    CUSTOM TIMINGS ✦ Make your own timings! ✦ Webpagetest will report your custom timings in the UI ✦ Full cross-browser support @ksylor
  34. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC DIAGNOSTIC METRICS -

    SYNTHETIC-ONLY METRICS ✦ JS Execution details ✦ Total blocking time (TBT) ✦ Time to Interactive (TTI) ✦ CPU usage ✦ Processing breakdown (script parse/eval/render/layout/paint etc.) @ksylor
  35. PERFORMANCE METRIC TYPES - PRIMARY VS. DIAGNOSTIC DIAGNOSTIC METRICS -

    SYNTHETIC-ONLY METRICS ✦ JS Execution details ✦ Total blocking time (TBT) ✦ Time to Interactive (TTI) ✦ CPU usage ✦ Processing breakdown (script parse/eval/render/layout/paint etc.) ✦ Rendering details ✦ Filmstrip screenshots ✦ Start Render ✦ Speed Index ✦ Visually Complete @ksylor
  36. TIME TO CATCH SOME PERF METRICS! @ksylor

  37. HTTPS://REPLICATE.COM/LAMBDAL/TEXT-TO-POKEMON @ksylor

  38. Emcee and tall person Tim Kadlec TRAINER Emcee and running

    enthusiast Henri Helvetica TRAINER HTTPS://REPLICATE.COM/LAMBDAL/TEXT-TO-POKEMON @ksylor
  39. Timeline-based Primary - Core Web Vitals RUM and Synthetic Largest

    Contentful Paint LCP CASE FILES @ksylor
  40. CASE FILES: LARGEST CONTENTFUL PAINT (LCP) WHO KILLED LCP? @ksylor

  41. CASE FILES: LARGEST CONTENTFUL PAINT (LCP) BEFORE (OFF) AFTER (ON)

    Aha! The design of the likely LCP element changed! @ksylor
  42. CASE FILES: LARGEST CONTENTFUL PAINT (LCP) DIAGNOSING CHANGES TO LCP

    1. Did the LCP element change? @ksylor
  43. TRAINER WEBPAGETEST TRAINER CARD Use to run synthetic tests on

    a real browser and device Webpagetest.org @ksylor
  44. CASE FILES: LARGEST CONTENTFUL PAINT (LCP) GOTCHA! This is not

    the LCP element you are looking for @ksylor
  45. TAKE 2 The LCP Element did not change CASE FILES:

    LARGEST CONTENTFUL PAINT (LCP) @ksylor
  46. DIAGNOSING CHANGES TO LCP 1. Did the LCP element change?

    NO 2. Did some characteristic of the LCP element change? - If image or video, did the asset size get bigger? - (If text, did the font or text change?) CASE FILES: LARGEST CONTENTFUL PAINT (LCP) @ksylor
  47. BEFORE (OFF) AFTER (ON) CASE FILES: LARGEST CONTENTFUL PAINT (LCP)

    Smaller Image size @ksylor
  48. BEFORE (OFF) AFTER (ON) CASE FILES: LARGEST CONTENTFUL PAINT (LCP)

    Smaller Image size @ksylor Hint to the real culprit!
  49. DIAGNOSING CHANGES TO LCP 1. Did the LCP element change?

    NO 2. Did some characteristic of the LCP element change? NO - If image or video, did the asset size get bigger? - (If text, did the font or text change?) 3. Did metrics before LCP in the timeline change? - e.g. TTFB, FCP, DCL? CASE FILES: LARGEST CONTENTFUL PAINT (LCP) @ksylor
  50. RUM TRAINER CARD Collect data from real user sessions Real

    User Monitoring TRAINER @ksylor
  51. Timeline-based Primary (not CWV) RUM and Synthetic Time to First

    Byte TTFB SUSPECT #1 @ksylor
  52. Timeline-based Primary (not CWV) RUM and Synthetic Time to First

    Byte TTFB CASE FILES @ksylor
  53. Timeline-based Primary (not CWV) RUM and Synthetic First Contentful Paint

    FCP SUSPECT #2 @ksylor
  54. Timeline-based Primary (not CWV) RUM and Synthetic First Contentful Paint

    FCP CASE FILES @ksylor
  55. DOM Content Loaded DCL SUSPECT #3 Timeline-based Diagnostic RUM and

    Synthetic @ksylor
  56. DOM Content Loaded DCL CASE FILES Timeline-based Diagnostic RUM and

    Synthetic @ksylor
  57. Page Load PAGE LOAD SUSPECT #4 Timeline-based Diagnostic RUM and

    Synthetic @ksylor
  58. Page Load PAGE LOAD CASE FILES Timeline-based Diagnostic RUM and

    Synthetic @ksylor
  59. DIAGNOSING CHANGES TO LCP 1. Did the LCP element change?

    NO 2. Did some characteristic of the LCP element change? NO - If image or video, did the asset size change? - (If text, did the font or text change?) 3. Did metrics before LCP in the timeline change? NO - e.g. TTFB, FCP, DCL? 4. Did the loading order change? CASE FILES: LARGEST CONTENTFUL PAINT (LCP) @ksylor
  60. BEFORE (OFF) AFTER (ON) LCP Element #12 CASE FILES: LARGEST

    CONTENTFUL PAINT (LCP) @ksylor
  61. BEFORE (OFF) AFTER (ON) LCP Element #12 LCP Element #13

    A wild CSS file appears! CASE FILES: LARGEST CONTENTFUL PAINT (LCP) @ksylor
  62. DIAGNOSING CHANGES TO LCP 1. Did the LCP element change?

    NO 2. Did some characteristic of the LCP element change? NO - If image or video, did the asset size change? - (If text, did the font or text change?) 3. Did metrics before LCP in the timeline change? NO - e.g. TTFB, FCP, DCL? 4. Did the loading order change? YES! 5. Did render-blocking assets grow in size or number? CASE FILES: LARGEST CONTENTFUL PAINT (LCP) @ksylor
  63. CHECK YOUR REQUESTS CASE FILES: LARGEST CONTENTFUL PAINT (LCP) +1

    request Double the bytes :( @ksylor
  64. DIAGNOSING CHANGES TO LCP 1. Did the LCP element change?

    NO 2. Did some characteristic of the LCP element change? NO - If image or video, did the asset size change? - (If text, did the font or text change?) 3. Did metrics before LCP in the timeline change? NO - e.g. TTFB, FCP, DCL? 4. Did the loading order change? YES! 5. Did render-blocking assets grow in size or number? YES! CASE FILES: LARGEST CONTENTFUL PAINT (LCP) @ksylor
  65. Independent Primary - Core Web Vitals Cumulative Layout Shift CLS

    CASE FILES RUM and Synthetic @ksylor
  66. CASE FILES: CUMULATIVE LAYOUT SHIFT (CLS) WHO KILLED CLS? @ksylor

  67. BEFORE (OFF) AFTER (ON) Aha! A new module above the

    fold CASE FILES: CUMULATIVE LAYOUT SHIFT (CLS) @ksylor
  68. DIAGNOSING CLS 1. What elements are shifting? A. Font loading

    FOUT/FOIT? B. Images without dimensions? C. Lazy-loaded content without a correct height placeholder? D. Injected banners, ads or iframes? CASE FILES: CUMULATIVE LAYOUT SHIFT (CLS) @ksylor
  69. LIGHTHOUSE TRAINER CARD Use to do a quick in-browser synthetic

    test Lighthouse TRAINER @ksylor
  70. BEFORE (OFF) AFTER (ON) CASE FILES: CUMULATIVE LAYOUT SHIFT (CLS)

    @ksylor
  71. BEFORE (OFF) AFTER (ON) CASE FILES: CUMULATIVE LAYOUT SHIFT (CLS)

    @ksylor
  72. ZOOMING IN ON AFTER (ON) CASE FILES: CUMULATIVE LAYOUT SHIFT

    (CLS) Everything under the new module @ksylor
  73. WHAT HAPPENED? CASE FILES: CUMULATIVE LAYOUT SHIFT (CLS) @ksylor

  74. DIAGNOSING CLS 1. What elements are shifting? A. Font loading

    FOUT/FOIT? NO B. Images without dimensions? NO C. Lazy-loaded content without a correct height placeholder? YES! D. Injected banners, ads or iframes? NO CASE FILES: CUMULATIVE LAYOUT SHIFT (CLS) @ksylor
  75. Independent Primary - Core Web Vitals RUM First Input Delay

    FID CASE FILES @ksylor
  76. CASE FILES: FIRST INPUT DELAY (FID) WHO KILLED FID? @ksylor

  77. AFTER (ON) This new button loads CASE FILES: FIRST INPUT

    DELAY (FID) @ksylor
  78. DIAGNOSING CHANGES TO FID 1. Is there more Javascript execution,

    e.g. Long Tasks? CASE FILES: FIRST INPUT DELAY (FID) @ksylor
  79. Long Tasks LONG TASKS SUSPECT #1 Timeline-based Diagnostic RUM and

    Synthetic @ksylor
  80. MORE LONG TASKS MORE PROBLEMS CASE FILES: FIRST INPUT DELAY

    (FID) @ksylor
  81. DIAGNOSING CHANGES TO FID 1. Is there more Javascript execution,

    e.g. Long Tasks? YES! 2. Is there more Javascript downloading? - Check both first- and third-party CASE FILES: FIRST INPUT DELAY (FID) @ksylor
  82. Resource Timing RESOURCE TIMING SUSPECT #2 Timeline-based Diagnostic RUM and

    Synthetic @ksylor
  83. JS RESOURCES CASE FILES: FIRST INPUT DELAY (FID) @ksylor

  84. DIAGNOSING CHANGES TO FID 1. Is there more Javascript execution,

    e.g. Long Tasks? YES! 2. Is there more Javascript downloading? NO? I think? - Check both first- and third-party 3. Did TBT or TTI get worse? CASE FILES: FIRST INPUT DELAY (FID) @ksylor
  85. Independent Primary - Core Web Vitals? Total Blocking Time TBT

    CASE FILES Synthetic @ksylor
  86. Timeline-based Primary Time to Interactive TTI CASE FILES Synthetic @ksylor

  87. TBT OR TTI? ✦ ~30 Synthetic tests were all inconclusive

    :( CASE FILES: FIRST INPUT DELAY (FID) @ksylor
  88. DIAGNOSING CHANGES TO FID 1. Is there more Javascript execution,

    e.g. Long Tasks? YES! 2. Is there more Javascript downloading? NO? I think? - Check both first- and third-party 3. Did TBT or TTI get worse? NO 4. Did user behavior change? IDK CASE FILES: FIRST INPUT DELAY (FID) @ksylor
  89. DIAGNOSING CHANGES TO FID 1. Is there more Javascript execution,

    e.g. Long Tasks? YES! 2. Is there more Javascript downloading? NO? I think? - Check both first- and third-party 3. Did TBT or TTI get worse? NO 4. Did user behavior change? IDK 5. Can we get FID attribution please Yoav FFS CASE FILES: FIRST INPUT DELAY (FID) @ksylor
  90. Independent Primary - Core Web Vitals? Interaction to Next Paint

    INP CASE FILES RUM @ksylor ?????
  91. WHAT DID WE LEARN? IT ALL STARTS WITH MONITORING @ksylor

  92. WHAT DID WE LEARN? PERF DETECTING ISN'T MAGIC askastaffengineer.com @ksylor

  93. WHAT DID WE LEARN? IT DEPENDS askastaffengineer.com @ksylor

  94. ! U Y k o T @ksylor hAn