A/B Testing For Product Discovery

Talk given at DUMBODevs (http://www.meetup.com/dumbo-developers) on 3/15/13. Another version will be given at RailsConf 2013 in April.

You probably already know that A/B testing is a low-cost and low-risk way to make more money from your website, but if you've only tested surface-level details like copy and button color, you might not be seeing the huge improvements in conversion rates and revenue you keep hearing about. In this talk, I'll first explore (both conceptually and with code examples from the open source gem we've created) ways developers can build split testing deeply into their products, in order to help guide their roadmap by discovering which features customers really want and how much they're willing to pay for them.

Transcript

1. A/B TESTING FOR PRODUCT DISCOVERY
2. ECONOMICS 101
3. If i was selling dollar bills for 99 cents each, how many would you buy?
4. ALL OF THEM
5. BRICk and mortar
7. WEB BUSINESS
8. scale
9. LICENSE TO PRINT MONEY
10. AGILE DEVELOPMENT RAPID ITERATION CONTINUOUS DEPLOYMENT
11. GOING NOWHERE FAST
12. Make use of your 31337 H4CK1N6 5K1LLZ
13. AN OPTIMIZATION PROBLEM 0 25 50 75 100 2007 2008 2009 2010 Money In Money Out
14. @bryanwoods
17. Dumbo devving since 2010
18. BASIC A/B TESTING
19. An almost real example
25. Skeptical? you might be surprised.
26. 0 25 50 75 100 March April May June July August Converted Viewed
29. When to implement?
30. Not a substitute for finding product/market fit
31. Low risk, low cost, possibly high reward
32. Deeper integration
34. WHAT TO TEST
35. what features do your customers want?
36. What is the simplest version of the feature you can build to gauge interest?
37. what are your customers willing to pay for?
38. which features are free and which require an upgrade?
39. How much will they pay?
40. are your pricing schemes and payment structure flexible enough to be varied?
41. are you undercharging?
42. when (and how often) should you remarket to them?
43. are you over-saturating your users or are you missing possible sales opportunities?
44. Product curiosities
45. test Every new user- facing feature
46. EXAMPLES
47. How can we encourage users to post more dates?
50. should we force them?
53. sufficient data obviates best practices
54. FIND FEEDBACK LOOPS AND EXPLOIT THEM
55. Uploading a photo and posting a date leads to more messages which leads to MORE subscriptions
56. Uploading a photo and posting a date leads to more messages which leads to MORE subscriptions
57. ZOMG A FUNNEL!
58. Your whole business is a conversion funnel
59. results
60. Huge boosts to conversion rates and revenue
61. technical IMPLEMENTATION
67. yours free
69. included batteries
70. user partitioning
71. convenience methods
72. start/end experiments from admin dashboard
73. not (yet) included
74. goals
75. significance
76. visualization
77. ??? Let us know
78. technical debt
79. ...in A brave new world of doubled complexity
80. automated testing is essential
81. all code must be held to the same standard of rigor
82. test in isolation
83. KILL DEAD CODE IMMEDIATELY
84. revertability
86. don’t get emotionally invested
87. MEASURING SUCCESS
88. statistical significance
89. binomial distribution
90. COHORT METRICS
91. Site health dashboard
92. What if there’s a tie?
93. UNEXPECTED BENEFITS
94. less time spent arguing over new features
95. “Mixcloud claims to have increased their conversion rate to signup by 200 - 300% [using Facebook Connect]”... “Although most sites won’t see the same level of improvement I’ve chatted with several other developers...that have quoted a 20%-50% increase in signups.” http://www.forbes.com/sites/quora/2012/03/28/what-are-some-top-strategies-for-conversion-optimization
96. https://www.dailycred.com/blog/6/surprise-people-hate-being-forced-to-use-facebook
97. is it worth building? testing?
98. a defense against conflicting customer feedback
100. ...And more $$$ doesn’t hurt.
101. Recap
102. rapid development is useless if you’re not moving in the right direction
103. tiny improvements over time can become huge in aggregate
104. let data inform your decisions
105. there are tools at your disposal
106. be rigorous and relentless
107. thank you.
108. Questions? https://speakerdeck.com/bryanwoods/b-testing-for- product-discovery @bryanwoods