Paul Coia - Iterative Experimental Product Development

The team at Redbubble believes in testing so much that they've made it a part of life. Paul Coia talks about what that means and the impact it has on the Redbubble product development approach.

Transcript

1. Iterative Experimental Product Development ! Paul Coia
2. Deploy Experiment Example Lessons
4. 10M designs on 40M products 500,000 artists all over the world. 40M pageviews from 7M Users/month
5. Deploy Experiment Example Lessons
6. 2000 releases 4 years 700 in 2012 Average 3 per day
7. cap production deploy
8. CI Server App Monitoring Error Reporting Business Metrics Monitoring
9. if enrolled_in?(“new”).variant?! // show variant! else! // show normal! end Dark Launching with Feature Toggles
10. Rapid pace of change possible Confidence make lots of changes Obtain user feedback quicker Why we like it
11. Deploy Experiment Example Lessons
12. • Research server logs or web analytics • Surveying users - onsite or email • User interviews and testing • Onsite split tests • …many more options Experimentation
17. Split users at point of variation Isolate experiments Persist the decision across visits Exclude crawlers and ‘old’ users Enrollment
18. Data Collection (Google Analytics)
19. Statistical Analysis (‘R’ + GA API)
20. The mathematics are simple Resist the temptation to shortcut Eyeballed results are o en wrong Respect statistics
21. Model your Product funnel Greater volume earlier in the funnel Higher conversion rate between steps Well defined metrics
22. Homepage Sign up page Signed up
23. 1% Homepage Sign up page Signed up
24. 1% Homepage Sign up page Signed up 20% 5%
25. ‘Landing Page’ Optimizers Client-side changes only Simple experiments
26. Deploy Experiment Example Lessons
27. 22% less likely to Add to Cart 50% less likely to Checkout Mobile Visitors
29. No change to Checkout Completion
31. Improvement in Checkout Completion
33. Increase in Add to Cart rate
35. Further Increase in Add to Cart rate
36. Multiple lightweight experiments A couple of weeks Verified 30% increase in Purchasing
37. Deploy Experiment Example Lessons
38. Deploy as fast as possible
39. Experiment o en - flexible direction
40. Design for incremental product change
41. Whole product team alignment
42. Thank You Paul Coia paul@redbubble.com @pjcoia ! h p://www.redbubble.com/jobs