Customer Driven Development - What, Why & How?

Transcript

1. Customer Driven Development - What, Why & How? Ishan Khanna (@droidchef) Android @ Getaround (Amsterdam)
2. LET’S DEFINE CUSTOMER DRIVEN DEVELOPMENT
3. KEEP THE CUSTOMER AT THE _________ OF ____________.
4. KEEP THE CUSTOMER AT THE CENTER OF ____________.
5. KEEP THE CUSTOMER AT THE CENTER OF EVERYTHING.
6. BY GIVING THEM THE POWER TO SHAPE THE PRODUCT AND DRIVE YOUR BUSINESS FORWARD
7. HOW YOU IMPLEMENT THIS IN PRACTICE?
8. WELCOME A/B TESTS
9. A/B testing means “go[ing] in a direction where your customers are telling you that you’re adding value to their lives.”
10. WHY SHOULD YOU DO THIS?
11. BY UNDERSTANDING YOUR CUSTOMERS, YOU CAN GIVE THEM WHAT THEY ACTUALLY NEED.
12. AND… YOU CAN ALSO DETERMINE, THE DIRECTION FORWARD BECAUSE OF THE DATA YOU COLLECT IN THE PROCESS.
13. WHATEVER THE OUTCOME OF YOUR EXPERIMENT IS, YOU ALWAYS WIN.
14. SIMPLY BECAUSE YOU EITHER EARN MORE OR YOU LEARN MORE.
15. EARNING = METRIC MOVES IN THE EXPECTED DIRECTION
16. LEARNING = METRIC MOVES IN THE UNEXPECTED DIRECTION
17. FOUNDATIONS OF AN A/B TEST
18. • HYPOTHESIS • TARGET AUDIENCE • PRIMARY METRIC • RUNTIME • SECONDARY METRIC(S)
19. WHAT IS A HYPOTHESIS?
20. SIMPLY PUT, IT IS AN EDUCATED GUESS
21. EXAMPLE BY ADDING A TOOLTIP ON A BUTTON WE WILL INCREASE CAR RENTALS.
22. BASE VARIANT It only takes a few mins
23. Changing (element tested) from _____ to _____ will increase/decrease (a conversion metric). SYNTAX
24. FORMULATING A WINNING HYPOTHESIS
25. AIM TO ALTER CUSTOMER BEHAVIOR EITHER POSITIVELY OR NEGATIVELY
26. FOCUS ON DERIVING LEARNINGS FROM THE TEST
27. BACKED BY DATA
28. TEMPLATE FOR WRITING A WINNING HYPOTHESIS
29. “BASED ON _______ WE BELIEVE, BY DOING ______ WE WILL BE ABLE TO _______. WE WILL SEE WHEN _____ MOVES IN ______ DIRECTION.”
30. “Based on our user research we believe that by adding a tooltip, we will be able to inform the users that booking process is blazing fast. We will see this when the net rentals start increasing.”
31. WHAT ARE METRICS?
32. PRIMARY METRIC
33. Which one do you think should be selected? • Net Rentals Per Day • Gross Rentals Per Day • Cancellations Per Day • Clicks on “Next” Button
34. Which one do you think should be selected? • Net Rentals Per Day • Gross Rentals Per Day • Cancellations Per Day • Clicks on “Next” Button
35. Because: • It accounts for cancellations as well. • If it increases, revenue definitely increases.
36. SECONDARY METRICS
37. Which one do you think should be selected? • Gross Rentals Per Day • Cancellations Per Day • Clicks on “Next” Button
38. THEY CAN AID IN UNDERSTANDING THE RESULTS OF AN EXPERIMENT BETTER AND WITH HIGHER CONFIDENCE.
39. CHOOSING THE RIGHT METRICS
40. RUNTIME OF AN EXPERIMENT
41. Statistical power is the probability of a hypothesis test of finding an effect if there is an effect to be found. UNDERSTANDING POWER
44. IMPORTANCE OF CALCULATING RUNTIME BEFORE RUNNING THE EXPERIMENT
45. WHAT IS NOISE?
46. IMAGINE YOU COOK A DISH AND SERVE IT TO 10 PEOPLE, ASKING THEIR FEEDBACK ABOUT THE TASTE.
47. ONLY 6 OF THEM TASTE IT BUT ALL 10 PROVIDE A FEEDBACK.
48. DO YOU THINK THE FEEDBACK OF 4 PEOPLE WHO DIDN’T TASTE YOUR DISH, MATTERS?
49. NO, AND THAT IS WHAT WE CALL NOISE.
50. HOW TO ELIMINATE NOISE?
51. STAGES TO THE RESCUE
52. STAGES ALLOW YOU TO FILTER OUT THE USERS WHOSE BEHAVIOR YOU ARE ACTUALLY INTERESTED IN GAUGING.
53. TRY TO SEGREGATE YOUR TRAFFIC TO A POINT WHERE YOU CAN ELIMINATE MAXIMUM AMOUNT OF NOISE.
54. AVOIDING STATISTICAL GHOSTS
55. A SIGNIFICANT PROBLEM OF PERCEIVING AN INSIGNIFICANT RESULT AS A SIGNIFICANT RESULT
56. EXAMPLE SCENARIO 1 SCENARIO 2 SCENARIO 3 SCENARIO 4 AFTER 200 OBSERVATIONS INSIGNIFICANT INSIGNIFICANT SIGNIFICANT SIGNIFICANT AFTER 500 INSIGNIFICANT SIGNIFICANT INSIGNIFICANT SIGNIFICANT END OF EXPERIMENT INSIGNIFICANT SIGNIFICANT INSIGNIFICANT SIGNIFICANT Suppose you analyze an experiment after 200 and 500 observations. There are four things that could happen: Assuming treatments A and B are the same and the significance level is 5%, then at the end of the experiment, we’ll have a significant result 5% of the time.
57. SCENARIO 1 SCENARIO 2 SCENARIO 3 SCENARIO 4 AFTER 200 OBSERVATIONS INSIGNIFICANT INSIGNIFICANT SIGNIFICANT SIGNIFICANT AFTER 500 INSIGNIFICANT SIGNIFICANT TRIAL STOPPED TRIAL STOPPED END OF EXPERIMENT INSIGNIFICANT SIGNIFICANT SIGNIFICANT SIGNIFICANT But suppose we stop the experiment as soon as there is a significant result. Now look at the four things that could happen:
58. LEARNINGS FIRST ROW EXACTLY THE SAME AS BEFORE. ASSUMING A AND B ARE ACTUALLY THE SAME WE INCREASED THE RATIO OF SIGNIFICANT RELATIVE TO INSIGNIFICANT RESULTS.
59. Therefore, the reported significance level – the “percent of the time the observed difference is due to chance” – will be wrong.
60. MAKE SURE TO PLAN FOLLOW UPS
61. LET’S RECAP THE PROCESS
63. THANK YOU! SLIDES : http://bit.ly/dc-berlin-2019 QUESTIONS? @droidchef