A/B Testing for Developers

A/B Testing for Developers

A/B testing doesn’t have as much street cred as SEO or user experience these days, but it’s just as effective a technique for boosting conversion rates and determining how your visitors use your web application. It seems simple: post two pages with the same goal, serve each page to your visitors an equal number of times, and measure the results. But the insight such simple tests can give into your visitors’ intentions is enormous. In this presentation we’ll walk through several examples of A/B testing in action, share some tips and tricks for using it effectively, and explore tools that make A/B testing with Ruby a snap.


Matthew Bass

October 12, 2010


  1. Ab Testing

  2. None
  3. What works? motivation aesthetics flow multimedia design graphics speed testimonials

  4. Testing

  5. How to do it

  6. Metrics

  7. Experiments

  8. Iterations hypothesis run test decide PROFIT!!

  9. • JavaScript-based • Integrates w/Analytics • Supports multivariate testing https://www.google.com/analytics/siteopt/

  10. • Rails plugin • Defunct? http://github.com/paulmars/seven_minute_abs

  11. • A/B testing for Rails • Uses Redis • Installs

    as a gem • Integrates with Analytics http://vanity.labnotes.org
  12. • A/B testing for Rails • Installs as a plugin

    • Uses ActiveRecord http://bingocardcreator.com/abingo
  13. • Radiant extension • Integrates w/Vanity • Uses ActiveRecord http://ext.radiantcms.org/extensions/227-metrics

  14. Roll your own?

  15. Installation & Usage

  16. What to test? user count surveys logos button size screencasts

    button color security badge popups wording form size
  17. Permutations • A/B/N • A/B/A • Multivariate • Taguchi methods

    • Heatmaps
  18. Pros • It’s easy • Fast setup • Small tweaks

    • Bang for buck • Reliability
  19. Cons • Time consuming (see next slide) • Over reliance

    • Dishonest? http://www.codinghorror.com/blog/2010/07/ groundhog-day-or-the-problem-with-ab-testing.html
  20. Minimizing Time • A/B over A/B/N or A/B/A • Have

    lots of traffic
  21. A/B testing is like sandpaper. You can use it to

    smooth out details, but you can't actually create anything with it. Nathan Bowers
  22. Use your brain

  23. None
  24. None
  25. None
  26. None
  27. None
  28. None
  29. Anecdotes • Client signup button • Useless experiments • Login/session

    error http://github.com/assaf/vanity/issues#issue/10 http://vanity.labnotes.org/identity.html
  30. Go do it.

  31. Attributions http://flickr.com aussiegall, d_vdm, euthman, missbossy, ka_tate http://alegion-stock.deviantart.com http://ca.askmen.com Matthew

    Bass twitter @pelargir http://matthewbass.com