Split Testing For Product Discovery

Given at Railsconf 2013 in Portland, Oregon.

In this talk, we'll explore split testing as a way to not only increase revenue and conversion through simple, surface-level changes, but also to dig deeper in order to help guide a product's roadmap by discovering which features customers really want and how much they're willing to pay.

Transcript

1. split TESTING FOR PRODUCT DISCOVERY
2. split TESTING FOR PRODUCT DISCOVERY
3. ECONOMICS 101
4. If i was selling dollar bills for 99 cents each, how many would you buy?
5. ALL OF THEM
6. BRICk and mortar
8. WEB BUSINESS
9. scale
10. LICENSE TO PRINT MONEY
11. AGILE DEVELOPMENT RAPID ITERATION CONTINUOUS DEPLOYMENT
12. GOING NOWHERE FAST
13. Make use of your 31337 H4CK1N6 5K1LLZ
14. AN OPTIMIZATION PROBLEM 0 25 50 75 100 2007 2008 2009 2010 Money In Money Out
15. @bryanwoods
19. BASIC A/B TESTING
20. An almost real example
26. Skeptical? you might be surprised.
27. 0 25 50 75 100 March April May June July August Converted Viewed
30. When to implement?
31. Not a substitute for finding product/market fit
32. Low risk, low cost, possibly high reward
33. Deeper integration
35. WHAT TO TEST
36. what features do your customers want?
37. What is the simplest version of the feature you can build to gauge interest?
38. what are your customers willing to pay for?
39. which features are free and which require an upgrade?
40. How much will they pay?
41. are your pricing schemes and payment structure flexible enough to be varied?
42. are you undercharging?
43. when (and how often) should you remarket to them?
44. are you over-saturating your users or are you missing possible sales opportunities?
45. Product curiosities
46. test Every new user-facing feature
47. EXAMPLES
48. How can we get users to receive more messages?
50. How can we encourage users to post more dates?
53. should we force them?
56. sufficient data obviates best practices
57. FIND FEEDBACK LOOPS AND EXPLOIT THEM
58. Uploading a photo and posting a date leads to more messages which leads to MORE subscriptions
59. Uploading a photo and posting a date leads to more messages which leads to MORE subscriptions
60. ZOMG A FUNNEL!
61. Your whole business is a conversion funnel
62. results
63. Huge boosts to conversion rates and revenue
64. technical IMPLEMENTATION
70. yours free
72. included batteries
73. user partitioning
74. convenience methods
75. start/end experiments from admin dashboard
76. not (yet) included
77. goals
78. significance
79. visualization
80. ??? Let us know
81. technical debt
82. ...in A brave new world of doubled complexity
83. automated testing is essential
84. all code must be held to the same standard of rigor
85. test in isolation
86. KILL DEAD CODE IMMEDIATELY
87. revertability
89. don’t get emotionally invested
90. MEASURING SUCCESS
91. statistical significance
92. binomial distribution
93. COHORT METRICS
94. Site health dashboard
95. daily emails
96. What if there’s a tie?
97. UNEXPECTED BENEFITS
98. less time spent arguing over new features
99. “Mixcloud claims to have increased their conversion rate to signup by 200 - 300% [using Facebook Connect]”... “Although most sites won’t see the same level of improvement I’ve chatted with several other developers...that have quoted a 20%-50% increase in signups.” http://www.forbes.com/sites/quora/2012/03/28/what-are-some-top-strategies-for-conversion-optimization
100. https://www.dailycred.com/blog/6/surprise-people-hate-being-forced-to-use-facebook
101. is it worth building? testing?
102. a defense against conflicting customer feedback
103. ...And more $$$ doesn’t hurt.
104. Recap
105. rapid development is useless if you’re not moving in the right direction
106. tiny improvements over time can become huge in aggregate
107. let data inform your decisions
108. there are tools at your disposal
109. be rigorous and relentless
110. thank you.
111. Questions? https://speakerdeck.com/bryanwoods/split-testing-for- product-discovery @bryanwoods