What can possibly go wrong with measurements? - Tomer Sharon

Slide 1

Slide 1 text

SLIDES: bit.ly/uxy-tomer CHEAT SHEET bit.ly/uxy-tomer-cheat

Slide 2

Slide 2 text

TORTURE THE DATA ENOUGH AND IT WILL CONFESS TO ANYTHING. – Ronald Coase, Economist and author

Slide 3

Slide 3 text

How satisﬁed are you with UX y’all 2023?

Slide 4

Slide 4 text

😀😐😕

Slide 5

Slide 5 text

⭐⭐⭐⭐⭐

Slide 6

Slide 6 text

👍👎

Slide 7

Slide 7 text

🌞

Slide 8

Slide 8 text

⛅

Slide 9

Slide 9 text

⛄

Slide 10

Slide 10 text

🥶

Slide 11

Slide 11 text

🦠

Slide 12

Slide 12 text

📅

Slide 13

Slide 13 text

❌

Slide 14

Slide 14 text

🔬

Slide 15

Slide 15 text

SHORT vs LONG CLICKS

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

HOW TO MEASURE THE SUCCESS OF A ROCK CONCERT?

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

Experience metrics are quantitative scores for important and actionable phenomena related to using products or services. @tsharon

Slide 21

Slide 21 text

2 3 4 1 Idea Launch Build Learn

Slide 22

Slide 22 text

2 3 4 1 Idea Launch Build Learn TACTICAL

Slide 23

Slide 23 text

2 4 1 Idea Launch Build Learn 3 STRATEGIC

Slide 24

Slide 24 text

2 4 1 Idea Launch Build Learn 3 METRICS

Slide 25

Slide 25 text

Tactical Strategic Metrics before ALL. THE. TIME. after @tsharon

Slide 26

Slide 26 text

I am directly involved in UX measurements for my company

Slide 27

Slide 27 text

WHAT IS THE #1 THING THAT CAN GO WRONG WITH UX MEASUREMENTS?

Slide 28

Slide 28 text

ChatGPT 1. Incomplete data 2. Biased samples 3. Subjectivity 4. Context dependency 5. Lack of long-term insights 6. Limited metrics 7. Misinterpretation of metrics 8. Overemphasis on metrics 9. Ignoring qualitative data 10. Sampling size and timing 11. Ignoring accessibility 12. Lack of actionable insights

Slide 29

Slide 29 text

Bard 1. Choosing the wrong metrics 2. Collecting too little or too much data 3. Using biased data 4. Misinterpreting the data 5. Not using the data to make changes 6. Using the wrong tools 7. Not understanding the limitations of the data 8. Not communicating the data effectively

Slide 30

Slide 30 text

NITPICKING Giving too much attention to details that are not important @tsharon

Slide 31

Slide 31 text

HOW ARE WE DOING? @tsharon

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

https://library.gv.com/change-aversion-why-users-hate-what-you-launched-and-what-to-do-about-it-2fb94ce65766

Slide 34

Slide 34 text

Happiness 87%↑ Engagement 76%↓ Adoption 69%↓ Retention 62%↑ Week of June 5-11, 2023

Slide 35

Slide 35 text

May 2023 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Slide 36

Slide 36 text

PORTFOLIO Product 1 Product 2 Product 3 Product 4

Slide 37

Slide 37 text

HIGH-LEVEL UX MEASUREMENT PRINCIPLES @tsharon

Slide 38

Slide 38 text

HIGH-LEVEL TEAM LEVEL

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

OVERALL SPECIFIC

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

OVERALL How satisfied are you with the BMW entertainment app? SPECIFIC How satisfied are you with the morning weather notification?

Slide 43

Slide 43 text

↑82%

Slide 44

Slide 44 text

Product a Product b Product c Credit+thanks: Bob Thomas, Liberty Mutual

Slide 45

Slide 45 text

BEHAVIOR OPINION

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

CORE METRICS

Slide 48

Slide 48 text

Happiness Engagement Adoption Retention Task success

Slide 49

Slide 49 text

Satisfaction score 7-day active Adoption rate Retention rate

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

100 April 1 100

Slide 52

Slide 52 text

94 April 30 94 6

Slide 53

Slide 53 text

94% retention rate 94 6

Slide 54

Slide 54 text

6% churn rate 94 6

Slide 55

Slide 55 text

12 new users April 1-30 94 12 6

Slide 56

Slide 56 text

12% adoption rate 94 12 6

Slide 57

Slide 57 text

100-6+12=106 106

Slide 58

Slide 58 text

IN JUST A MOMENT…

Slide 59

Slide 59 text

THE CASE FOR HIGH-LEVEL METRICS

Slide 60

Slide 60 text

UNDERSTAND THE USER EXPERIENCE AT A GLANCE

Slide 61

Slide 61 text

No content

Slide 62

Slide 62 text

No content

Slide 63

Slide 63 text

No content

Slide 64

Slide 64 text

SET OKRs

Slide 65

Slide 65 text

No content

Slide 66

Slide 66 text

O: Reach product/market ﬁt KR1: 95% customer satisfaction KR2: 80% retention KR3: 65% 7-day active

Slide 67

Slide 67 text

UNDERSTAND THE EFFECT OF CHANGE

Slide 68

Slide 68 text

https://library.gv.com/change-aversion-why-users-hate-what-you-launched-and-what-to-do-about-it-2fb94ce65766

Slide 69

Slide 69 text

I get it, Tomer. @tsharon I’m lost, I need measurement therapy.

Slide 70

Slide 70 text

DATA IS LIKE GARBAGE. YOU’D BETTER KNOW WHAT YOU ARE GOING TO DO WITH IT BEFORE YOU COLLECT IT. – Mark Twain

Slide 71

Slide 71 text

🙏 🌈 🦄

Slide 72

Slide 72 text

SLIDES: bit.ly/uxy-tomer CHEAT SHEET bit.ly/uxy-tomer-cheat

Slide 73

Slide 73 text

Q&A

Slide 74

Slide 74 text

SHAMELESS PLUG 1 The ﬁrst-ever UX streaming service with exclusive, high-quality content

Slide 75

Slide 75 text

SHAMELESS PLUG 2 leanresearch.co/services