Paul Verbeek-Mast (Booking.com), Bad Evidence in Testing Product Hypothesis, CodeFest 2017

https://2017.codefest.ru/lecture/1181

Criminal investigators have something called ‘bad evidence’ or confirmation bias. When they have a theory about a case, sometimes they tend to avoid evidence that goes against that theory. Unconsciously but also consciously.

This is a big problem we have with testing product hypothesis as well. We tend to ignore the data that go against our theory. And if we do have “bad data” we can’t go around, we test it a bit longer until this disappears.

Why do we do this and how do you deal with this? What other common pitfalls do we have? And is hypothesis testing really worth the time?

Transcript

1. Bad Evidence
2. Paul Verbeek-Mast
3. workingatbooking.com
5. All A/B tests and data shown in this presentation are not based on real experiments. They are made up just for this presentation.
12. How much do you want to create “Bad Evidence”? Насколько вы готовы получить доказательство обратного ?
13. You don’t want to do something if it is going to go against your theory of the case. Вы не хотите делать что-то что повредит вашей теории
14. Rather than trying to get to the truth, what you’re trying to do is build your case, and make it the strongest case possible. Вместо того чтобы докопаться до истины вы пытаетесь защитить свою версию, сделав ее доказательство "пуленепробиваемым".
15. What does verification bias cause you to do? Ignore it and push it to the side. Что вы будете делать со своей предвзятостью ? Просто игнорируйте ее.
17. Base база
18. Base Variant база вариант
19. Base Variant база вариант
20. Base Variant база вариант
21. Base Variant база вариант
22. Base Variant база вариант
23. Base Variant вариант база
24. Base Variant 5234 searches вариант база
25. Base Variant 5234 searches 6252 searches вариант база
26. Base Variant 5234 searches 6252 searches +19.45% вариант база
27. Base Variant 5234 searches 6252 searches +19.45% вариант база
29. Base Variant база вариант
30. Base Variant Making the search box hotpink will result in more searches база вариант
31. Making the search box hotpink will result in more searches
32. Making the search box hotpink will result in more searches 6252 searches +19.45%
33. Making the search box hotpink will result in more searches 6252 searches +19.45% 242 bookings -4.7%
34. Making the search box hotpink will result in more searches
35. Making the search box hotpink will result in more searches ?
36. Because of (why) we believe that changing (what) for (who) will result into (outcome)
37. Why • Based on a gut feeling, I believe (…) • Because I like it better, I believe (…) • Because I saw it on another website, I believe (…) Bad examples Objective and based on data
38. Why • Because of research described in article (…), we believe (…) • After done user research, we believe (…) • Based on a previous experiment doing (…), we believe (…) Objective and based on data Good examples
39. What An accurate, short description of your change • we make it pink • we move it to a different place • we change the title Bad examples
40. What • we make the search box on the homepage pink • we open pictures in the search page in a lightbox when clicking on it Good examples An accurate, short description of your change
41. Who A realistic, accurate description of your target group • everyone • some people • users booking a hotel in Novosibirsk, named Paul, from Amsterdam, with a big beard Bad examples
42. Who A realistic, accurate description of your target group • users visiting the home page • users searching for a property in Novosibirsk • users who are logged in Good examples
43. Outcome measurable, expected changes • users feeling better • the site looking prettier • an increase in loyalty Bad examples
44. Outcome • an increase in earnings • a decrease in returned products • an increase in sign-ups Good examples measurable expected metrics
45. Because of (why) we believe that changing (what) for (who) will result into (outcome)
46. Because of user research we believe that changing (what) for (who) will result into (outcome)
47. Because of user research we believe that changing the background of the search box to pink for (who) will result into (outcome)
48. Because of user research we believe that changing the background of the search box on the homepage pink for users that visit the homepage will result into (outcome)
49. Because of user research we believe that changing the background of the search box on the homepage pink for users that visit the homepage will result into an increase in bookings
50. Because of user research we believe that changing the background of the search box on the homepage pink for users that visit the homepage will result into an increase in bookings
51. Because of user research we believe that changing the background of the search box on the homepage pink for users that visit the homepage will result into an increase in bookings
53. Because of studies done by x and y that show the positive effect of green, we believe that changing our booking buttons to green for users who visit our product page will result into more bookings
54. Because of studies done by x and y that show the positive effect of green, we believe that changing our booking buttons to green for users who visit our product page will result into more bookings Base
55. Because of studies done by x and y that show the positive effect of green, we believe that changing our booking buttons to green for users who visit our product page will result into more bookings Base Variant
56. Base Variant 812 bookings 825 bookings +1.6% Day 1
57. Base Variant 1.924 bookings 1.920 bookings -0.2% Day 2
58. Base Variant 2.714 bookings 2.925 bookings +7.8% Day 3
59. Base Variant 16.623 bookings 16.324 bookings -1.8% Day 20
60. • Number of visitors • How big of a change you want to measure • How confident you want to be, that your test is correct How long should your run your A/B test?
61. You can never be 100% confident that your test is correct
62. The more things you measure, the higher the chance some test metrics are incorrect
63. Variant bookings -1.8%
64. Variant bookings -1.8% price of bookings +4.3%
65. clicks on button hover over button bookings visits on page scrolled to button bookings from IE8 bookings from Malaysia users going to search results logins sign ups clicks on logo time on page returning visitors price of booking number of rooms booked language changes calls to customer service buys with credit card
66. clicks on button hover over button bookings visits on page scrolled to button bookings from IE8 bookings from Malaysia users going to search results logins sign ups clicks on logo time on page returning visitors price of booking number of rooms booked language changes calls to customer service buys with credit card
67. clicks on button hover over button bookings visits on page scrolled to button bookings from IE8 bookings from Malaysia users going to search results logins sign ups clicks on logo time on page returning visitors price of booking number of rooms booked language changes calls to customer service buys with credit card
68. clicks on button hover over button bookings visits on page scrolled to button bookings from IE8 bookings from Malaysia users going to search results logins sign ups clicks on logo time on page returning visitors price of booking number of rooms booked language changes calls to customer service buys with credit card
69. clicks on button hover over button bookings visits on page scrolled to button bookings from IE8 bookings from Malaysia users going to search results logins sign ups clicks on logo time on page returning visitors price of booking number of rooms booked language changes calls to customer service buys with credit card
70. clicks on button hover over button bookings visits on page scrolled to button bookings from IE8 bookings from Malaysia users going to search results logins sign ups clicks on logo time on page returning visitors price of booking number of rooms booked language changes calls to customer service buys with credit card
71. clicks on button hover over button bookings visits on page scrolled to button bookings from IE8 bookings from Malaysia users going to search results logins sign ups clicks on logo time on page returning visitors price of booking number of rooms booked language changes calls to customer service buys with credit card +0.1% -0.2% +2.3% +0.3% +4.7% -3.1% +0.0% +3.5% -1.1% -2.1% +0.3% +2.1% -1.8% -0.3% +0.0% +0.5% +4.3% -0.2%
72. clicks on button hover over button bookings visits on page scrolled to button bookings from IE8 bookings from Malaysia users going to search results logins sign ups clicks on logo time on page returning visitors price of booking number of rooms booked language changes calls to customer service buys with credit card +0.1% -0.2% +2.3% +0.3% +4.7% -3.1% +0.0% +3.5% -1.1% -2.1% +0.3% +2.1% -1.8% -0.3% +0.0% +0.5% +4.3% -0.2%
73. clicks on button hover over button bookings visits on page scrolled to button bookings from IE8 bookings from Malaysia users going to search results logins sign ups clicks on logo time on page returning visitors price of booking number of rooms booked language changes calls to customer service buys with credit card +0.1% -0.2% +2.3% +0.3% +4.7% -3.1% +0.0% +3.5% -1.1% -2.1% +0.3% +2.1% -1.8% -0.3% +0.0% +0.5% +4.3% -0.2%
74. Focus on your defined metrics, but also keep an eye on your health metrics
75. “price is going up, so it must be doing well” “price is going down, so it must be a false negative” vs. Metrics that are not in hypothesis
76. “this new metric is positive, it’s working great!” “this new metric is negative, must be having a bug” vs. Newly implemented metrics
77. “it’s positive after 5 days, let’s put it in production” “it’s negative after 5 days, let’s run it for another few days” vs. Sample size
78. Because of (why) we believe that changing (what) for (who) will result into (outcome)
79. • Number of visitors • How big of a change you want to measure • How confident you want to be, that your test is correct How long should your run your A/B test?
82. Paul Verbeek-Mast
83. @_paulverbeek verbeek.p@gmail.com
84. Questions? Вопросов?