Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Split Testing For Product Discovery

Split Testing For Product Discovery

Given at Railsconf 2013 in Portland, Oregon.

In this talk, we'll explore split testing as a way to not only increase revenue and conversion through simple, surface-level changes, but also to dig deeper in order to help guide a product's roadmap by discovering which features customers really want and how much they're willing to pay.

10610a774494efbb8f8ca5c7a3f33da4?s=128

Bryan Woods

April 20, 2013
Tweet

Transcript

  1. split TESTING FOR PRODUCT DISCOVERY

  2. split TESTING FOR PRODUCT DISCOVERY

  3. ECONOMICS 101

  4. If i was selling dollar bills for 99 cents each,

    how many would you buy?
  5. ALL OF THEM

  6. BRICk and mortar

  7. None
  8. WEB BUSINESS

  9. scale

  10. LICENSE TO PRINT MONEY

  11. AGILE DEVELOPMENT RAPID ITERATION CONTINUOUS DEPLOYMENT

  12. GOING NOWHERE FAST

  13. Make use of your 31337 H4CK1N6 5K1LLZ

  14. AN OPTIMIZATION PROBLEM 0 25 50 75 100 2007 2008

    2009 2010 Money In Money Out
  15. @bryanwoods

  16. None
  17. None
  18. None
  19. BASIC A/B TESTING

  20. An almost real example

  21. None
  22. None
  23. None
  24. None
  25. None
  26. Skeptical? you might be surprised.

  27. 0 25 50 75 100 March April May June July

    August Converted Viewed
  28. None
  29. None
  30. When to implement?

  31. Not a substitute for finding product/market fit

  32. Low risk, low cost, possibly high reward

  33. Deeper integration

  34. None
  35. WHAT TO TEST

  36. what features do your customers want?

  37. What is the simplest version of the feature you can

    build to gauge interest?
  38. what are your customers willing to pay for?

  39. which features are free and which require an upgrade?

  40. How much will they pay?

  41. are your pricing schemes and payment structure flexible enough to

    be varied?
  42. are you undercharging?

  43. when (and how often) should you remarket to them?

  44. are you over-saturating your users or are you missing possible

    sales opportunities?
  45. Product curiosities

  46. test Every new user-facing feature

  47. EXAMPLES

  48. How can we get users to receive more messages?

  49. None
  50. How can we encourage users to post more dates?

  51. None
  52. None
  53. should we force them?

  54. None
  55. None
  56. sufficient data obviates best practices

  57. FIND FEEDBACK LOOPS AND EXPLOIT THEM

  58. Uploading a photo and posting a date leads to more

    messages which leads to MORE subscriptions
  59. Uploading a photo and posting a date leads to more

    messages which leads to MORE subscriptions
  60. ZOMG A FUNNEL!

  61. Your whole business is a conversion funnel

  62. results

  63. Huge boosts to conversion rates and revenue

  64. technical IMPLEMENTATION

  65. None
  66. None
  67. None
  68. None
  69. None
  70. yours free

  71. None
  72. included batteries

  73. user partitioning

  74. convenience methods

  75. start/end experiments from admin dashboard

  76. not (yet) included

  77. goals

  78. significance

  79. visualization

  80. ??? Let us know

  81. technical debt

  82. ...in A brave new world of doubled complexity

  83. automated testing is essential

  84. all code must be held to the same standard of

    rigor
  85. test in isolation

  86. KILL DEAD CODE IMMEDIATELY

  87. revertability

  88. None
  89. don’t get emotionally invested

  90. MEASURING SUCCESS

  91. statistical significance

  92. binomial distribution

  93. COHORT METRICS

  94. Site health dashboard

  95. daily emails

  96. What if there’s a tie?

  97. UNEXPECTED BENEFITS

  98. less time spent arguing over new features

  99. “Mixcloud claims to have increased their conversion rate to signup

    by 200 - 300% [using Facebook Connect]”... “Although most sites won’t see the same level of improvement I’ve chatted with several other developers...that have quoted a 20%-50% increase in signups.” http://www.forbes.com/sites/quora/2012/03/28/what-are-some-top-strategies-for-conversion-optimization
  100. https://www.dailycred.com/blog/6/surprise-people-hate-being-forced-to-use-facebook

  101. is it worth building? testing?

  102. a defense against conflicting customer feedback

  103. ...And more $$$ doesn’t hurt.

  104. Recap

  105. rapid development is useless if you’re not moving in the

    right direction
  106. tiny improvements over time can become huge in aggregate

  107. let data inform your decisions

  108. there are tools at your disposal

  109. be rigorous and relentless

  110. thank you.

  111. Questions? https://speakerdeck.com/bryanwoods/split-testing-for- product-discovery @bryanwoods