Measuring your measuring

Most marketing measurement quietly lies to us. Discover how your analytics, testing tools, and trackers distort reality by slowing sites, losing users, and biasing the very data they claim to capture. Through real-world examples and hard numbers, we'll expose survivorship bias, hidden bounces, and false confidence in A/B testing, then reframe performance and measurement around what users actually experience.

Transcript

1. Jono Alderson Independent technical SEO consultant MEASURING YOUR MEASURING
2. Jono Alderson Independent technical SEO consultant medium.com/@addyosmani/shopping-for-speed-on-ebay-com-6229711d7573
3. Jono Alderson Independent technical SEO consultant (2020, Deloitte)
4. Jono Alderson Independent technical SEO consultant wpostats.com
5. Jono Alderson Independent technical SEO consultant Image source: LINK
6. Jono Alderson Independent technical SEO consultant Image source: LINK
7. Jono Alderson Independent technical SEO consultant This isn’t news.
8. Jono Alderson Independent technical SEO consultant But maybe this is..
9. Jono Alderson Independent technical SEO consultant
10. Jono Alderson Independent technical SEO consultant
11. Jono Alderson Independent technical SEO consultant A B
12. Jono Alderson Independent technical SEO consultant Image source: LINK 53% of mobile visitors will bounce if a page takes longer than 3 seconds to load (2016, Google)
13. Jono Alderson Independent technical SEO consultant Image source: LINK 0s 1s 2s 3s 4s 5s 6s Optimistic Few, well-optimized scripts (2022, NordVPN)
14. Jono Alderson Independent technical SEO consultant Image source: LINK Optimistic Few, well-optimized scripts Average Some overhead, and tag management bloat 0s 1s 2s 3s 4s 5s 6s (2022, NordVPN) (2025, CrUX) 48 tracking pixels (2022, NordVPN)
15. Jono Alderson Independent technical SEO consultant Image source: LINK 0s 1s 2s 3s 4s 5s 6s Optimistic Few, well-optimized scripts Average Some overhead, and tag management bloat Poor Unmanaged JavaScript hellscape Half of your mobile visitors already left (2022, NordVPN) (2025, CrUX) (2025, most of the crap I see) 48 tracking pixels (2022, NordVPN)
16. Jono Alderson Independent technical SEO consultant Image source: LINK
17. Jono Alderson Independent technical SEO consultant Image source: LINK How big a deal is this?
18. Jono Alderson Independent technical SEO consultant Most-used tools Intercom Optimizely HotJar (reported vs actual) VWO GTM (empty) Facebook Pixel Convert Hubspot Adobe Target GA4 I did some research across the most-used* tools & trackers… * (2022, NordVPN)
19. Jono Alderson Independent technical SEO consultant Tool Size (KB) Intercom 739 Optimizely 133 HotJar (reported vs actual) 130 400 VWO 100 GTM (empty) 80 Facebook Pixel 95 Convert 70 Hubspot 45 Adobe Target 30 GA4 17 How big are these files*? * Varies with usage
20. Jono Alderson Independent technical SEO consultant Tool Size (KB) Broadband (ms) Intercom 739 3,015 Optimizely 133 543 HotJar (reported vs actual) 130 400 530 1,632 VWO 100 408 GTM (empty) 80 326 Facebook Pixel 95 388 Convert 70 286 Hubspot 45 184 Adobe Target 30 122 GA4 17 69 How long do they (maybe*) take to load? * Varies by device * Some of these load in parallel (but some don’t)
21. Jono Alderson Independent technical SEO consultant Tool Size (KB) Broadband (ms) 4G (ms) Intercom 739 3,015 3,547 Optimizely 133 543 638 HotJar (reported vs actual) 130 400 530 1,632 624 1,920 VWO 100 408 480 GTM (empty) 80 326 384 Facebook Pixel 95 388 456 Convert 70 286 336 Hubspot 45 184 216 Adobe Target 30 122 144 GA4 17 69 82
22. Jono Alderson Independent technical SEO consultant Tool Size (KB) Broadband (ms) 4G (ms) 3G (ms) Intercom 739 3,015 3,547 6,651 Optimizely 133 543 638 1,197 HotJar (reported vs actual) 130 400 530 1,632 624 1,920 1,170 3,600 VWO 100 408 480 900 GTM (empty) 80 326 384 720 Facebook Pixel 95 388 456 855 Convert 70 286 336 630 Hubspot 45 184 216 405 Adobe Target 30 122 144 270 GA4 17 69 82 153
23. Jono Alderson Independent technical SEO consultant Tool Size (KB) Broadband (ms) 4G (ms) 3G (ms) Intercom 739 3,015 3,547 6,651 Optimizely 133 543 638 1,197 HotJar (reported vs actual) 130 400 530 1,632 624 1,920 1,170 3,600 VWO 100 408 480 900 GTM (empty) 80 326 384 720 Facebook Pixel 95 388 456 855 Convert 70 286 336 630 Hubspot 45 184 216 405 Adobe Target 30 122 144 270 GA4 17 69 82 153
24. Jono Alderson Independent technical SEO consultant Tool Size (KB) Broadband (ms) 4G (ms) 3G (ms) Intercom 739 3,015 3,547 6,651 Optimizely 133 543 638 1,197 HotJar (reported vs actual) 130 400 530 1,632 624 1,920 1,170 3,600 VWO 100 408 480 900 GTM (empty) 80 326 384 720 Facebook Pixel 95 388 456 855 Convert 70 286 336 630 Hubspot 45 184 216 405 Adobe Target 30 122 144 270 GA4 17 69 82 153
25. Jono Alderson Independent technical SEO consultant Tool Size (KB) Broadband (ms) 4G (ms) 3G (ms) Intercom 739 3,015 3,547 6,651 Optimizely 133 543 638 1,197 HotJar (reported vs actual) 130 400 530 1,632 624 1,920 1,170 3,600 VWO 100 408 480 900 GTM (empty) 80 326 384 720 Facebook Pixel 95 388 456 855 Convert 70 286 336 630 Hubspot 45 184 216 405 Adobe Target 30 122 144 270 GA4 17 69 82 153
26. Jono Alderson Independent technical SEO consultant Tool Size (KB) Broadband (ms) 4G (ms) 3G (ms) Intercom 739 3,015 3,547 6,651 Optimizely 133 543 638 1,197 HotJar (reported vs actual) 130 400 530 1,632 624 1,920 1,170 3,600 VWO 100 408 480 900 GTM (empty) 80 326 384 720 Facebook Pixel 95 388 456 855 Convert 70 286 336 630 Hubspot 45 184 216 405 Adobe Target 30 122 144 270 GA4 17 69 82 153
27. Jono Alderson Independent technical SEO consultant Tool Size (KB) Broadband (ms) 4G (ms) 3G (ms) Intercom 739 3,015 3,547 6,651 Optimizely 133 543 638 1,197 HotJar (reported vs actual) 130 400 530 1,632 624 1,920 1,170 3,600 VWO 100 408 480 900 GTM (empty) 80 326 384 720 Facebook Pixel 95 388 456 855 Convert 70 286 336 630 Hubspot 45 184 216 405 Adobe Target 30 122 144 270 GA4 17 69 82 153
28. Jono Alderson Independent technical SEO consultant Tool Size (KB) Broadband (ms) 4G (ms) 3G (ms) Intercom 739 3,015 3,547 6,651 Optimizely 133 543 638 1,197 HotJar (reported vs actual) 130 400 530 1,632 624 1,920 1,170 3,600 VWO 100 408 480 900 GTM (empty) 80 326 384 720 Facebook Pixel 95 388 456 855 Convert 70 286 336 630 Hubspot 45 184 216 405 Adobe Target 30 122 144 270 GA4 17 69 82 153
29. Jono Alderson Independent technical SEO consultant • 1.1mb of data transfer • 6-7 seconds (middleweight device, middleweight network) • Just with a basic setup... A typical suite of tracking tools might add:
30. Jono Alderson Independent technical SEO consultant Image source: LINK What’s that mean, in the real world?
31. Jono Alderson Independent technical SEO consultant • Huge website & brand • Famously sophisticated • Thousands of A/B tests
32. Jono Alderson Independent technical SEO consultant • 3mb page • 1.4mb of JavaScript • 1.5s load time
33. Jono Alderson Independent technical SEO consultant A B • So let’s run an A/B test • 100,000 visits • 2% conversion rate
34. Jono Alderson Independent technical SEO consultant A • But we know… • 25% untracked bounces • Only 75,000 tracked visits B
35. Jono Alderson Independent technical SEO consultant A B • Test shows a 5% uplift • But against the total 100,000 visits… • The error margin means it could have been 5% worse
36. Jono Alderson Independent technical SEO consultant
37. Jono Alderson Independent technical SEO consultant Image source: LINK So what do we do?
38. Jono Alderson Independent technical SEO consultant Image source: LINK web.dev/chrome-ux-report-data-studio-dashboard/
39. Jono Alderson Independent technical SEO consultant Image source: LINK web.dev/chrome-ux-report-data-studio-dashboard/ + Competitor data!
40. Jono Alderson Independent technical SEO consultant Image source: LINK pagespeed.web.dev
41. Jono Alderson Independent technical SEO consultant Image source: LINK pagespeed.web.dev
42. Jono Alderson Independent technical SEO consultant performancebudget.io
43. Jono Alderson Independent technical SEO consultant websitecarbon.com
44. Jono Alderson Independent technical SEO consultant
45. Jono Alderson Independent technical SEO consultant partytown.builder.io
46. Jono Alderson Independent technical SEO consultant Image source: LINK Use Avanish’s 90/10 rule
47. Jono Alderson Independent technical SEO consultant GOOD LUCK