Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A/B Testing For Product Discovery

A/B Testing For Product Discovery

Talk given at DUMBODevs (http://www.meetup.com/dumbo-developers) on 3/15/13. Another version will be given at RailsConf 2013 in April.

You probably already know that A/B testing is a low-cost and low-risk way to make more money from your website, but if you've only tested surface-level details like copy and button color, you might not be seeing the huge improvements in conversion rates and revenue you keep hearing about. In this talk, I'll first explore (both conceptually and with code examples from the open source gem we've created) ways developers can build split testing deeply into their products, in order to help guide their roadmap by discovering which features customers really want and how much they're willing to pay for them.

10610a774494efbb8f8ca5c7a3f33da4?s=128

Bryan Woods

March 12, 2013
Tweet

More Decks by Bryan Woods

Other Decks in Technology

Transcript

  1. A/B TESTING FOR PRODUCT DISCOVERY

  2. ECONOMICS 101

  3. If i was selling dollar bills for 99 cents each,

    how many would you buy?
  4. ALL OF THEM

  5. BRICk and mortar

  6. None
  7. WEB BUSINESS

  8. scale

  9. LICENSE TO PRINT MONEY

  10. AGILE DEVELOPMENT RAPID ITERATION CONTINUOUS DEPLOYMENT

  11. GOING NOWHERE FAST

  12. Make use of your 31337 H4CK1N6 5K1LLZ

  13. AN OPTIMIZATION PROBLEM 0 25 50 75 100 2007 2008

    2009 2010 Money In Money Out
  14. @bryanwoods

  15. None
  16. None
  17. Dumbo devving since 2010

  18. BASIC A/B TESTING

  19. An almost real example

  20. None
  21. None
  22. None
  23. None
  24. None
  25. Skeptical? you might be surprised.

  26. 0 25 50 75 100 March April May June July

    August Converted Viewed
  27. None
  28. None
  29. When to implement?

  30. Not a substitute for finding product/market fit

  31. Low risk, low cost, possibly high reward

  32. Deeper integration

  33. None
  34. WHAT TO TEST

  35. what features do your customers want?

  36. What is the simplest version of the feature you can

    build to gauge interest?
  37. what are your customers willing to pay for?

  38. which features are free and which require an upgrade?

  39. How much will they pay?

  40. are your pricing schemes and payment structure flexible enough to

    be varied?
  41. are you undercharging?

  42. when (and how often) should you remarket to them?

  43. are you over-saturating your users or are you missing possible

    sales opportunities?
  44. Product curiosities

  45. test Every new user- facing feature

  46. EXAMPLES

  47. How can we encourage users to post more dates?

  48. None
  49. None
  50. should we force them?

  51. None
  52. None
  53. sufficient data obviates best practices

  54. FIND FEEDBACK LOOPS AND EXPLOIT THEM

  55. Uploading a photo and posting a date leads to more

    messages which leads to MORE subscriptions
  56. Uploading a photo and posting a date leads to more

    messages which leads to MORE subscriptions
  57. ZOMG A FUNNEL!

  58. Your whole business is a conversion funnel

  59. results

  60. Huge boosts to conversion rates and revenue

  61. technical IMPLEMENTATION

  62. None
  63. None
  64. None
  65. None
  66. None
  67. yours free

  68. None
  69. included batteries

  70. user partitioning

  71. convenience methods

  72. start/end experiments from admin dashboard

  73. not (yet) included

  74. goals

  75. significance

  76. visualization

  77. ??? Let us know

  78. technical debt

  79. ...in A brave new world of doubled complexity

  80. automated testing is essential

  81. all code must be held to the same standard of

    rigor
  82. test in isolation

  83. KILL DEAD CODE IMMEDIATELY

  84. revertability

  85. None
  86. don’t get emotionally invested

  87. MEASURING SUCCESS

  88. statistical significance

  89. binomial distribution

  90. COHORT METRICS

  91. Site health dashboard

  92. What if there’s a tie?

  93. UNEXPECTED BENEFITS

  94. less time spent arguing over new features

  95. “Mixcloud claims to have increased their conversion rate to signup

    by 200 - 300% [using Facebook Connect]”... “Although most sites won’t see the same level of improvement I’ve chatted with several other developers...that have quoted a 20%-50% increase in signups.” http://www.forbes.com/sites/quora/2012/03/28/what-are-some-top-strategies-for-conversion-optimization
  96. https://www.dailycred.com/blog/6/surprise-people-hate-being-forced-to-use-facebook

  97. is it worth building? testing?

  98. a defense against conflicting customer feedback

  99. None
  100. ...And more $$$ doesn’t hurt.

  101. Recap

  102. rapid development is useless if you’re not moving in the

    right direction
  103. tiny improvements over time can become huge in aggregate

  104. let data inform your decisions

  105. there are tools at your disposal

  106. be rigorous and relentless

  107. thank you.

  108. Questions? https://speakerdeck.com/bryanwoods/b-testing-for- product-discovery @bryanwoods