Grace Ng - Designing Effective Experiments for Product Decisions

Search Love San Diego 2014

Transcript

1. Designing Effective Experiments for Product Decisions Grace Ng Co-founder, Javelin.com creators of QuickMVP @uxceo
2. "Business has only two functions - marketing and innovation." - Peter Drucker
3. Build
4. MEASURE METRICS LEARN PIVOT/PERSEVERE BUILD EXPERIMENT Hypothe size Test Analyze @uxceo LEARN MEASURE BUILD
5. Experiments Start here. Brainstorm with stickies, pull it over to the right to start your experiment. Customer Problem Solution Result & Decision Learning Riskiest Assumption Method & Success Criterion 1 2 3 4 5 To form a Customer/Problem Hypothesis: I believe my customer has a problem achieving this goal. To form your Assumptions: 0UVYKLYMVY̸hypothesis to be true, assumption needs to be true. Determine how you will test it: The least expensive way to test my assumption is... >OVPZ`V\YJ\Z[VTLY&)LHZZWLJPÄJHZWVZZPISL           Time Limit: 5 Min What is the problem? Phrase it from your customer’s perspective. Time Limit: 5 Min +LÄUL[OLZVS\[PVUVUS`HM[LY`V\OH]L]HSPKH[LKHWYVISLT^VY[OZVS]PUNTime Limit: 5 Min List the assumptions that must hold true, for your hypothesis to be true. Time Limit: 10 Min Need help? Use these sentences to help construct your experiment. To form a Problem/Solution Hypothesis: I believe this solution will result in X\HU[PÄHISLV\[JVTL. To identify your Riskiest Assumption: The assumption with the least amount of data, and core to the viability of my hypothesis is... Determine what success looks like: I will run experiment with # of customers and expect a strong signal from # of customers. Project Name: Team Leader Name: GET OUT OF THE BUILDING! © 2014 Javelin. You are free to use it and earn money with it as an entrepreneur, consultant, or executive, as long as you are not a software company (the latter need to license it from us). Download Experiment Board and watch case studies at www.javelin.com Experiment Board Javelin @uxceo grace@QuickMVP.com @uxceo http://bit.ly/xB0D
6. Step 1 Define Your Hypothesis @uxceo
7. Who is your customer? Must be specific to be easily testable! @uxceo
8. @uxceo grace@QuickMVP.com What is his/her problem? Must be specific to the customer @uxceo
9. Problem INCORRECT: “People are not buying enough cameras” is not a problem someone has CORRECT: “The outdoor adventurer hates the bulkiness” “The amateur photographer doesn’t think his skills are good enough to invest in a DSLR ” Different customers, different problems @uxceo grace@QuickMVP.com @uxceo
10. Problem Hypothesis I believe [my customer] has a problem [achieving this goal] @uxceo grace@QuickMVP.com @uxceo
11. Solution Do not define until the problem is validated Why? Because every problem has multiple solutions @uxceo grace@QuickMVP.com @uxceo
12. Discover @uxceo
13. What is the solution? What will solve your customer’s problem? @uxceo
14. Solution Hypothesis @uxceo grace@QuickMVP.com I believe [my solution] will result in [my customer] taking [specific action] @uxceo Solution Hypothesis
15. Plan your experiment Step 2 @uxceo grace@QuickMVP.com @uxceo
16. What must we LEARN? What are the Core Assumptions of our business? Assumption: A behavior, mentality, or action that needs to happen in order for hypothesis to be true. @uxceo grace@QuickMVP.com @uxceo
17. What must we TEST? Prioritize: Which Assumption is the Riskiest? The one that is most unknown, and crucial to the business @uxceo grace@QuickMVP.com @uxceo What must we TEST?
18. Minimum Viable Product Use the least amount of effort to get the most amount of customer feedback @uxceo grace@QuickMVP.com @uxceo
19. What do we BUILD to test? 1. Exploration: test validity of problem 2. Pitch: test value of your solution 3. Concierge: test customer satisfaction or behavior @uxceo grace@QuickMVP.com @uxceo What must we BUILD?
20. How to MEASURE success? Minimum Success Criteria : What’s the weakest outcome you need to see in order to move forward? @uxceo grace@QuickMVP.com @uxceo
21. 1. Cost • Project Cost • Opportunity Cost 2. Existing metrics • Business KPI • Specific user behavior How to MEASURE success? @uxceo grace@QuickMVP.com @uxceo How to MEASURE success?
22. Run your experiment Step 3 @uxceo grace@QuickMVP.com @uxceo
23. @uxceo
24. What did you learn? Step 4 @uxceo grace@QuickMVP.com @uxceo
25. Invalidated? PIVOT! @uxceo
26. Validated? PERSEVERE! @uxceo
27. Keep Testing @uxceo
28. Case Study @uxceo
29. @uxceo Founders UX Help
30. Experiments xperiment. Customer Problem Solution Result & Decision Riskiest Assumption Success Criterion 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the GET OUT OF THE BUILDING! Boot- strapped startup founders Getting UX feedback No reliable source Interview: 6/10
31. Exploration Interviews @uxceo
32. Effective Interviews 1. I’m doing some research around... 2. Have you ever [experienced problem]? 3. Tell me about that time, what happened? 4. What did you do to resolve it? 5. How satisfied were you with the outcome? 6. If unsatisfied, how else have you tried to solve it? @uxceo grace@QuickMVP.com @uxceo Effective Interviews
33. Collect Data • Level of pain/frustration • Frequency of encounter with pain • Existing behavior to solve problem • Time/money spent searching for a solution @uxceo grace@QuickMVP.com @uxceo Collect Data
34. Solution Result & Decision Learning Riskiest Assumption Success Criterion Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the hesis is... like: with # of ct a strong omers. GET OUT OF THE BUILDING! No reliable source Interview: 6/10 9/12 Persevere Friends. Classes. Paid UX designers hourly for help @uxceo
35. Experiments xperiment. Customer Problem Solution Result & Decision Riskiest Assumption Success Criterion 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the GET OUT OF THE BUILDING! Boot- strapped startup founders Getting UX feedback No reliable source Interview: 6/10 Boot- strapped startup founders Getting UX feedback @uxceo
36. Experiments xperiment. Customer Problem Solution Result & Decision Riskiest Assumption Success Criterion 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the GET OUT OF THE BUILDING! Boot- strapped startup founders Getting UX feedback No reliable source Interview: 6/10 Boot- strapped startup founders Getting UX feedback Virtual UX feedback Pay for virtual UX feedback Pitch: 8/10 @uxceo
37. @uxceo grace@QuickMVP.com $10 @uxceo
38. @uxceo grace@QuickMVP.com No Backend @uxceo No Backend @uxceo
39. Solution Result & Decision Learning Riskiest Assumption Success Criterion Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the hesis is... like: with # of ct a strong omers. GET OUT OF THE BUILDING! No reliable source Interview: 6/10 Pay for virtual UX feedback Pitch: 8/10 10/10 Test 2nd side of market Don’t mind virtual at all. Price is cheap 9/12 Persevere Friends. Classes. Paid UX designers for help @uxceo
40. Experiments xperiment. Customer Problem Solution Result & Decision Riskiest Assumption Success Criterion 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the GET OUT OF THE BUILDING! Boot- strapped startup founders Getting UX feedback No reliable source Interview: 6/10 Boot- strapped startup founders Getting UX feedback Virtual UX feedback Pay for virtual UX feedback Pitch: 8/10 UX Designers Getting clients for quick UX consulting @uxceo
41. Boot- strapped startup founders Getting UX feedback No reliable source Interview: 6/10 Boot- strapped startup founders Getting UX feedback Virtual UX feedback Pay for virtual UX feedback Pitch: 8/10 UX Designers Getting clients for quick UX consulting Virtual UX feedback Will consult virtually Pitch: 8/10 Experiments xperiment. Customer Problem Solution Result & Decision Riskiest Assumption Success Criterion 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the GET OUT OF THE BUILDING! Boot- strapped startup founders Getting UX feedback No reliable source Interview: 6/10 Boot- strapped startup founders Getting UX feedback Virtual UX feedback Pay for virtual UX feedback Pitch: 8/10 UX Designers Getting clients for quick UX consulting Virtual UX feedback Will consult virtually Pitch: 8/10 @uxceo
42. Seriously? @uxceo
43. Solution Result & Decision Learning Riskiest Assumption Success Criterion Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the hesis is... like: with # of ct a strong omers. GET OUT OF THE BUILDING! No reliable source Interview: 6/10 Pay for virtual UX feedback Pitch: 8/10 10/10 Test 2nd side of market Don’t mind virtual at all. Price is cheap Will consult virtually Pitch: 8/10 0/10 Pivot! 9/12 Persevere Friends. Classes. Paid UX designers for help @uxceo
44. Solution Result & Decision Learning Riskiest Assumption Success Criterion Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the hesis is... like: with # of ct a strong omers. GET OUT OF THE BUILDING! No reliable source Interview: 6/10 Pay for virtual UX feedback Pitch: 8/10 10/10 Test 2nd side of market Don’t mind virtual at all. Price is cheap Will consult virtually Pitch: 8/10 0/10 Pivot! No! 5 yrs exp. Established 9/12 Persevere Friends. Classes. Paid UX designers for help @uxceo
45. Boot- strapped startup founders Getting UX feedback No reliable source Interview: 6/10 Boot- strapped startup founders Getting UX feedback Virtual UX feedback Pay for virtual UX feedback Pitch: 8/10 UX Designers Getting clients for quick UX consulting Virtual UX feedback Will consult virtually Pitch: 8/10 Recent grad UX designers Experiments xperiment. Customer Problem Solution Result & Decision Riskiest Assumption Success Criterion 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the GET OUT OF THE BUILDING! Boot- strapped startup founders Getting UX feedback No reliable source Interview: 6/10 Boot- strapped startup founders Getting UX feedback Virtual UX feedback Pay for virtual UX feedback Pitch: 8/10 UX Designers Getting clients for quick UX consulting Virtual UX feedback Will consult virtually Pitch: 8/10 Recent grad UX designers Getting clients for quick UX consulting @uxceo
46. Boot- strapped startup founders Getting UX feedback No reliable source Interview: 6/10 Boot- strapped startup founders Getting UX feedback Virtual UX feedback Pay for virtual UX feedback Pitch: 8/10 UX Designers Getting clients for quick UX consulting Virtual UX feedback Will consult virtually Pitch: 8/10 Recent grad UX designers Getting clients for quick UX consulting Virtual UX feedback Give quick online feedback Pitch: 60% Experiments xperiment. Customer Problem Solution Result & Decision Riskiest Assumption Success Criterion 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the GET OUT OF THE BUILDING! Boot- strapped startup founders Getting UX feedback No reliable source Interview: 6/10 Boot- strapped startup founders Getting UX feedback Virtual UX feedback Pay for virtual UX feedback Pitch: 8/10 UX Designers Getting clients for quick UX consulting Virtual UX feedback Will consult virtually Pitch: 8/10 Recent grad UX designers Getting clients for quick UX consulting Virtual UX feedback Give quick online feedback Pitch: 60% @uxceo
47. @uxceo grace@QuickMVP.com @uxceo Pitch @uxceo
48. @uxceo @nycuxpa #ngnycuxpa Google Ads @uxceo
49. No Backend @uxceo
50. @uxceo grace@QuickMVP.com Quick MVP @uxceo grace@QuickMVP.com @uxceo Quick MVP @uxceo
51. 3 Data Points @uxceo @nycuxpa #ngnycuxpa @uxceo 3 Data Points @uxceo
52. Solution Result & Decision Learning Riskiest Assumption Success Criterion Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the hesis is... like: with # of ct a strong omers. GET OUT OF THE BUILDING! No reliable source Interview: 6/10 8/10 Persevere Friends. Classes. 2 paid UX designers for help Pay for virtual UX feedback Pitch: 8/10 10/10 Test 2nd side of market Don’t mind virtual at all. Price is cheap Will consult virtually Pitch: 8/10 0/10 Pivot! No! 5 yrs exp. Established Give quick online feedback Pitch: 60% 37/184 emails Persevere Started referring friends @uxceo
53. Experiments xperiment. Customer Problem Solution Result & Decision Riskiest Assumption Success Criterion 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the GET OUT OF THE BUILDING! Boot- strapped startup founders Getting UX feedback No reliable source Interview: 6/10 Boot- strapped startup founders Getting UX feedback Virtual UX feedback Pay for virtual UX feedback Pitch: 8/10 UX Designers Getting clients for quick UX consulting Virtual UX feedback Will consult virtually Pitch: 8/10 Recent grad UX designers Getting clients for quick UX consulting Virtual UX feedback Give quick online feedback Pitch: 60% Online platform is sufficient Concierge: 60% satisfied Boot- strapped startup founders Getting UX feedback Virtual UX feedback @uxceo
54. Concierge Customer Submits Request UX Designer gets email and submits feedback @uxceo grace@QuickMVP.com Concierge @uxceo
55. Manually Customer Submits Request UX Designer gets email and submits feedback @uxceo grace@QuickMVP.com Manually @uxceo
56. Solution Result & Decision Learning Riskiest Assumption Success Criterion Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the hesis is... like: with # of ct a strong omers. GET OUT OF THE BUILDING! No reliable source Interview: 6/10 8/10 Persevere Friends. Classes. 2 paid UX designers for help Pay for virtual UX feedback Pitch: 8/10 10/10 Test 2nd side of market Don’t mind virtual at all. Price is cheap Will consult virtually Pitch: 8/10 0/10 Pivot! No! 5 yrs exp. Established Give quick online feedback Pitch: 60% 37/184 emails Persevere Started referring friends Online platform is sufficient Concierge: 60% satisfied 20% Iterate Wanted more function- ality @uxceo
57. @uxceo High Pain Low Pain Low Frequency High Frequency Focus
58. @lissijean @uxceo
59. Experiments xperiment. Customer Problem Solution Result & Decision Riskiest Assumption Success Criterion 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the GET OUT OF THE BUILDING! OpenSky Shoppers Think curators not authentic @uxceo
60. Experiments xperiment. Customer Problem Solution Result & Decision Riskiest Assumption Success Criterion 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the GET OUT OF THE BUILDING! CMS Block OpenSky Shoppers Think curators not authentic @uxceo
61. Experiments xperiment. Customer Problem Solution Result & Decision Riskiest Assumption Success Criterion 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the GET OUT OF THE BUILDING! Shoppers will buy @uxceo CMS Block OpenSky Shoppers Think curators not authentic
62. Experiments xperiment. Customer Problem Solution Result & Decision Riskiest Assumption Success Criterion 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the GET OUT OF THE BUILDING! Concierge: Purchase up 20% @uxceo Shoppers will buy CMS Block OpenSky Shoppers Think curators not authentic
63. @lissijean @uxceo
64. Solution Result & Decision Learning Riskiest Assumption Success Criterion Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the hesis is... like: with # of ct a strong omers. GET OUT OF THE BUILDING! @uxceo $0 in Revenue No CT Shoppers will buy from CMS Concierge: Purchase up 20%
65. Solution Result & Decision Learning Riskiest Assumption Success Criterion Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the hesis is... like: with # of ct a strong omers. GET OUT OF THE BUILDING! @uxceo Shoppers don’t looks at the home feed, only emails $0 in Revenue No CT Shoppers will buy from CMS Concierge: Purchase up 20%
66. Experiments xperiment. Customer Problem Solution Result & Decision Riskiest Assumption Success Criterion 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the GET OUT OF THE BUILDING! @uxceo Concierge: Purchase up 20% Shoppers will buy from CMS CMS Block OpenSky Shoppers Think curators not authentic
67. Experiments xperiment. Customer Problem Solution Result & Decision Riskiest Assumption Success Criterion 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the GET OUT OF THE BUILDING! Opensky Shoppers Not seeing curator messages @uxceo Concierge: Purchase up 20% Shoppers will buy from CMS CMS Block OpenSky Shoppers Think curators not authentic
68. Experiments xperiment. Customer Problem Solution Result & Decision Riskiest Assumption Success Criterion 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the GET OUT OF THE BUILDING! Opensky Shoppers Not seeing curator messages @uxceo Concierge: Purchase up 20% Shoppers will buy from CMS CMS Block OpenSky Shoppers Think curators not authentic Email messages
69. Experiments xperiment. Customer Problem Solution Result & Decision Riskiest Assumption Success Criterion 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the GET OUT OF THE BUILDING! Opensky Shoppers Not seeing curator messages @uxceo Concierge: Purchase up 20% Shoppers will buy from CMS CMS Block OpenSky Shoppers Think curators not authentic Email messages Shoppers will buy
70. Experiments xperiment. Customer Problem Solution Result & Decision Riskiest Assumption Success Criterion 1 2 3 4 5 Time Limit: 5 Min Time Limit: 5 Min Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the GET OUT OF THE BUILDING! Opensky Shoppers Not seeing curator messages @uxceo Concierge: Purchase up 20% Shoppers will buy from CMS CMS Block OpenSky Shoppers Think curators not authentic Email messages Shoppers will buy Concierge: Purchase up 20%
71. @uxceo
72. Success! 2x Open Rate 3x Purchases @uxceo
73. Solution Result & Decision Learning Riskiest Assumption Success Criterion Time Limit: 5 Min Time Limit: 10 Min pothesis: n will result TL. mption: the least core to the hesis is... like: with # of ct a strong omers. GET OUT OF THE BUILDING! Shoppers don’t looks at the home feed, only emails $0 in Revenue No CT Shoppers will buy from CMS Concierge: Purchase up 20% Shoppers will buy Concierge: Purchase up 20% 300% on purchases Shoppers like videos, photos, from curator in email @uxceo
74. Guess? @uxceo
75. -$77,000 8 Weeks + 5 People @uxceo
76. +$10,000 2 Weeks + 2 People @uxceo
77. 1. Form your hypothesis 2. Identify your riskiest assumption 3. Decide how you’ll test it 4. Define success 5. Analyze results and try again @uxceo
78. Thanks! Download resources at Javelin.com @uxceo