What can possibly go wrong with measurements? - Tomer Sharon

Many product teams focus on detailed metrics, such as conversion rates, and not enough on high-level metrics, such as retention and satisfaction. This can lead to executing the wrong plans and failing to understand the overall user experience. Tomer will share examples of how high-level metrics can be used to understand the user experience while discussing Google's HEART framework, which can be used to identify and measure these metrics. Tomer believes that understanding high-level metrics is essential for creating a great user experience and that teams can avoid embarrassing results and failures by doing so with the right focus and mindset.

UX Y'all 2023

Transcript

1. SLIDES: bit.ly/uxy-tomer CHEAT SHEET bit.ly/uxy-tomer-cheat
2. TORTURE THE DATA ENOUGH AND IT WILL CONFESS TO ANYTHING. – Ronald Coase, Economist and author
3. How satisfied are you with UX y’all 2023?
4. 😀😐😕
5. ⭐⭐⭐⭐⭐
6. 👍👎
7. 🌞
8. ⛅
9. ⛄
10. 🥶
11. 🦠
12. 📅
13. ❌
14. 🔬
15. SHORT vs LONG CLICKS
17. HOW TO MEASURE THE SUCCESS OF A ROCK CONCERT?
20. Experience metrics are quantitative scores for important and actionable phenomena related to using products or services. @tsharon
21. 2 3 4 1 Idea Launch Build Learn
22. 2 3 4 1 Idea Launch Build Learn TACTICAL
23. 2 4 1 Idea Launch Build Learn 3 STRATEGIC
24. 2 4 1 Idea Launch Build Learn 3 METRICS
25. Tactical Strategic Metrics before ALL. THE. TIME. after @tsharon
26. I am directly involved in UX measurements for my company
27. WHAT IS THE #1 THING THAT CAN GO WRONG WITH UX MEASUREMENTS?
28. ChatGPT 1. Incomplete data 2. Biased samples 3. Subjectivity 4. Context dependency 5. Lack of long-term insights 6. Limited metrics 7. Misinterpretation of metrics 8. Overemphasis on metrics 9. Ignoring qualitative data 10. Sampling size and timing 11. Ignoring accessibility 12. Lack of actionable insights
29. Bard 1. Choosing the wrong metrics 2. Collecting too little or too much data 3. Using biased data 4. Misinterpreting the data 5. Not using the data to make changes 6. Using the wrong tools 7. Not understanding the limitations of the data 8. Not communicating the data effectively
30. NITPICKING Giving too much attention to details that are not important @tsharon
31. HOW ARE WE DOING? @tsharon
33. https://library.gv.com/change-aversion-why-users-hate-what-you-launched-and-what-to-do-about-it-2fb94ce65766
34. Happiness 87%↑ Engagement 76%↓ Adoption 69%↓ Retention 62%↑ Week of June 5-11, 2023
35. May 2023 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
36. PORTFOLIO Product 1 Product 2 Product 3 Product 4
37. HIGH-LEVEL UX MEASUREMENT PRINCIPLES @tsharon
38. HIGH-LEVEL TEAM LEVEL
40. OVERALL SPECIFIC
42. OVERALL How satisfied are you with the BMW entertainment app? SPECIFIC How satisfied are you with the morning weather notification?
43. ↑82%
44. Product a Product b Product c Credit+thanks: Bob Thomas, Liberty Mutual
45. BEHAVIOR OPINION
47. CORE METRICS
48. Happiness Engagement Adoption Retention Task success
49. Satisfaction score 7-day active Adoption rate Retention rate
51. 100 April 1 100
52. 94 April 30 94 6
53. 94% retention rate 94 6
54. 6% churn rate 94 6
55. 12 new users April 1-30 94 12 6
56. 12% adoption rate 94 12 6
57. 100-6+12=106 106
58. IN JUST A MOMENT…
59. THE CASE FOR HIGH-LEVEL METRICS
60. UNDERSTAND THE USER EXPERIENCE AT A GLANCE
64. SET OKRs
66. O: Reach product/market fit KR1: 95% customer satisfaction KR2: 80% retention KR3: 65% 7-day active
67. UNDERSTAND THE EFFECT OF CHANGE
68. https://library.gv.com/change-aversion-why-users-hate-what-you-launched-and-what-to-do-about-it-2fb94ce65766
69. I get it, Tomer. @tsharon I’m lost, I need measurement therapy.
70. DATA IS LIKE GARBAGE. YOU’D BETTER KNOW WHAT YOU ARE GOING TO DO WITH IT BEFORE YOU COLLECT IT. – Mark Twain
71. 🙏 🌈 🦄
72. SLIDES: bit.ly/uxy-tomer CHEAT SHEET bit.ly/uxy-tomer-cheat
73. Q&A
74. SHAMELESS PLUG 1 The first-ever UX streaming service with exclusive, high-quality content
75. SHAMELESS PLUG 2 leanresearch.co/services