A/B Testing for Developers

A/B testing doesn’t have as much street cred as SEO or user experience these days, but it’s just as effective a technique for boosting conversion rates and determining how your visitors use your web application. It seems simple: post two pages with the same goal, serve each page to your visitors an equal number of times, and measure the results. But the insight such simple tests can give into your visitors’ intentions is enormous. In this presentation we’ll walk through several examples of A/B testing in action, share some tips and tricks for using it effectively, and explore tools that make A/B testing with Ruby a snap.

Transcript

1. Ab Testing
3. What works? motivation aesthetics flow multimedia design graphics speed testimonials simplicity
4. Testing
5. How to do it
6. Metrics
7. Experiments
8. Iterations hypothesis run test decide PROFIT!!
9. • JavaScript-based • Integrates w/Analytics • Supports multivariate testing https://www.google.com/analytics/siteopt/
10. • Rails plugin • Defunct? http://github.com/paulmars/seven_minute_abs
11. • A/B testing for Rails • Uses Redis • Installs as a gem • Integrates with Analytics http://vanity.labnotes.org
12. • A/B testing for Rails • Installs as a plugin • Uses ActiveRecord http://bingocardcreator.com/abingo
13. • Radiant extension • Integrates w/Vanity • Uses ActiveRecord http://ext.radiantcms.org/extensions/227-metrics
14. Roll your own?
15. Installation & Usage
16. What to test? user count surveys logos button size screencasts button color security badge popups wording form size
17. Permutations • A/B/N • A/B/A • Multivariate • Taguchi methods • Heatmaps
18. Pros • It’s easy • Fast setup • Small tweaks • Bang for buck • Reliability
19. Cons • Time consuming (see next slide) • Over reliance • Dishonest? http://www.codinghorror.com/blog/2010/07/ groundhog-day-or-the-problem-with-ab-testing.html
20. Minimizing Time • A/B over A/B/N or A/B/A • Have lots of traffic
21. A/B testing is like sandpaper. You can use it to smooth out details, but you can't actually create anything with it. Nathan Bowers
22. Use your brain
29. Anecdotes • Client signup button • Useless experiments • Login/session error http://github.com/assaf/vanity/issues#issue/10 http://vanity.labnotes.org/identity.html
30. Go do it.
31. Attributions http://flickr.com aussiegall, d_vdm, euthman, missbossy, ka_tate http://alegion-stock.deviantart.com http://ca.askmen.com Matthew Bass twitter @pelargir http://matthewbass.com