Split Testing For Product Discovery

Given at Railsconf 2013 in Portland, Oregon.

In this talk, we'll explore split testing as a way to not only increase revenue and conversion through simple, surface-level changes, but also to dig deeper in order to help guide a product's roadmap by discovering which features customers really want and how much they're willing to pay.

Transcript

1. split TESTING FOR
   PRODUCT
   DISCOVERY
   

   View Slide

2. split TESTING FOR
   PRODUCT
   DISCOVERY
   

   View Slide

3. ECONOMICS 101
   

   View Slide

4. If i was selling dollar bills
   for 99 cents each, how many
   would you buy?
   

   View Slide

5. ALL OF
   THEM
   

   View Slide

6. BRICk and mortar
   

   View Slide

7. View Slide

8. WEB BUSINESS
   

   View Slide

9. scale
   

   View Slide

10. LICENSE TO
    PRINT MONEY
    

    View Slide

11. AGILE DEVELOPMENT
    RAPID ITERATION
    CONTINUOUS
    DEPLOYMENT
    

    View Slide

12. GOING
    NOWHERE
    FAST
    

    View Slide

13. Make use of your
    31337 H4CK1N6
    5K1LLZ
    

    View Slide

14. AN OPTIMIZATION PROBLEM
    0
    25
    50
    75
    100
    2007 2008 2009 2010
    Money In Money Out
    

    View Slide

15. @bryanwoods
    

    View Slide

16. View Slide

17. View Slide

18. View Slide

19. BASIC A/B
    TESTING
    

    View Slide

20. An almost real example
    

    View Slide

21. View Slide

22. View Slide

23. View Slide

24. View Slide

25. View Slide

26. Skeptical?
    you might be
    surprised.
    

    View Slide

27. 0
    25
    50
    75
    100
    March April May June July August
    Converted Viewed
    

    View Slide

28. View Slide

29. View Slide

30. When to implement?
    

    View Slide

31. Not a substitute for finding
    product/market fit
    

    View Slide

32. Low risk, low cost, possibly
    high reward
    

    View Slide

33. Deeper integration
    

    View Slide

34. View Slide

35. WHAT TO TEST
    

    View Slide

36. what features do
    your customers
    want?
    

    View Slide

37. What is the simplest version of
    the feature you can build to
    gauge interest?
    

    View Slide

38. what are your
    customers willing
    to pay for?
    

    View Slide

39. which features are free and
    which require an upgrade?
    

    View Slide

40. How much will they
    pay?
    

    View Slide

41. are your pricing schemes and
    payment structure flexible
    enough to be varied?
    

    View Slide

42. are you undercharging?
    

    View Slide

43. when
    (and how often)
    should you
    remarket to them?
    

    View Slide

44. are you over-saturating your
    users or are you missing
    possible sales opportunities?
    

    View Slide

45. Product curiosities
    

    View Slide

46. test Every new
    user-facing feature
    

    View Slide

47. EXAMPLES
    

    View Slide

48. How can we get users to
    receive more messages?
    

    View Slide

49. View Slide

50. How can we encourage users
    to post more dates?
    

    View Slide

51. View Slide

52. View Slide

53. should we force them?
    

    View Slide

54. View Slide

55. View Slide

56. sufficient data
    obviates best
    practices
    

    View Slide

57. FIND
    FEEDBACK
    LOOPS AND
    EXPLOIT THEM
    

    View Slide

58. Uploading a photo
    and posting a date
    leads to more messages
    which leads to MORE
    subscriptions
    

    View Slide

59. Uploading a photo
    and posting a date
    leads to more messages
    which leads to MORE
    subscriptions
    

    View Slide

60. ZOMG
    A
    FUNNEL!
    

    View Slide

61. Your whole business
    is a conversion
    funnel
    

    View Slide

62. results
    

    View Slide

63. Huge boosts to conversion
    rates and revenue
    

    View Slide

64. technical
    IMPLEMENTATION
    

    View Slide

65. View Slide

66. View Slide

67. View Slide

68. View Slide

69. View Slide

70. yours free
    

    View Slide

71. View Slide

72. included batteries
    

    View Slide

73. user partitioning
    

    View Slide

74. convenience methods
    

    View Slide

75. start/end experiments from
    admin dashboard
    

    View Slide

76. not (yet) included
    

    View Slide

77. goals
    

    View Slide

78. significance
    

    View Slide

79. visualization
    

    View Slide

80. ???
    Let us know
    

    View Slide

81. technical debt
    

    View Slide

82. ...in A brave new world of
    doubled complexity
    

    View Slide

83. automated testing is essential
    

    View Slide

84. all code must be held to the
    same standard of rigor
    

    View Slide

85. test in isolation
    

    View Slide

86. KILL DEAD CODE
    IMMEDIATELY
    

    View Slide

87. revertability
    

    View Slide

88. View Slide

89. don’t get
    emotionally
    invested
    

    View Slide

90. MEASURING SUCCESS
    

    View Slide

91. statistical significance
    

    View Slide

92. binomial distribution
    

    View Slide

93. COHORT METRICS
    

    View Slide

94. Site health dashboard
    

    View Slide

95. daily emails
    

    View Slide

96. What if there’s a tie?
    

    View Slide

97. UNEXPECTED BENEFITS
    

    View Slide

98. less time spent arguing over
    new features
    

    View Slide

99. “Mixcloud claims to have increased
    their conversion rate to signup by
    200 - 300% [using Facebook
    Connect]”...
    “Although most sites won’t see the
    same level of improvement I’ve
    chatted with several other
    developers...that have quoted a
    20%-50% increase in signups.”
    http://www.forbes.com/sites/quora/2012/03/28/what-are-some-top-strategies-for-conversion-optimization
    

    View Slide

100. https://www.dailycred.com/blog/6/surprise-people-hate-being-forced-to-use-facebook
     

     View Slide

101. is it worth building?
     testing?
     

     View Slide

102. a defense against conflicting
     customer feedback
     

     View Slide

103. ...And more $$$ doesn’t hurt.
     

     View Slide

104. Recap
     

     View Slide

105. rapid development is useless if
     you’re not moving in the right
     direction
     

     View Slide

106. tiny improvements over time
     can become huge in aggregate
     

     View Slide

107. let data inform your decisions
     

     View Slide

108. there are tools at your
     disposal
     

     View Slide

109. be rigorous and relentless
     

     View Slide

110. thank you.
     

     View Slide

111. Questions?
     https://speakerdeck.com/bryanwoods/split-testing-for-
     product-discovery
     @bryanwoods
     

     View Slide