A/B Testing everything with Split

A/B Testing everything with Split

Just because the tests past doesn't mean that your users will actually like the change, you can test your features in front of real users using A/B testing. A/B testing is becoming an important tool on the web to test out new features on a subset of your users to find out if they fulfil their purpose and make your decisions based on data not opinions. Split is a rack based A/B testing framework, it works with Rails, Sinatra and any other rack based web framework. Find out why we needed another framework and how to use in Rails, Sinatra and Radiant and the best practises of A/B testing.

8ddbf811da78bb0daeeb3cacd7cf743f?s=128

Andrew Nesbitt

October 07, 2011
Tweet

Transcript

  1. A/B TESTING SPLIT w

  2. github.com/andrew @ b

  3. • Your code • Your app • Your infrastructure •

    Your tests You probably test..
  4. Do you test your users?

  5. How do you know that you’re making the of your

    features
  6. How do you know if your are succeeding?

  7. If you’re not testing, you don’t know

  8. • Lean • Short feedback loops • Data driven development

    Why bother?
  9. • Signups • Conversions • Usage • Pageviews • Upgrades

    Metrics
  10. $$$

  11. €€€

  12. Are you making the most out of your pages?

  13. “Every time man makes a new experiment he always learns

    more. He cannot learn less.” R r B r F r
  14. How?

  15. A/B Testing

  16. C r +30% +27% +15% +7% http://bit.ly/37tests

  17. • Google web optimizer • Visual Website Optimizer • Optimizely

    Software as a Service
  18. • Abingo • SevenMinuteAbs • Vanity A/B Testing Frameworks

  19. Split github.com/andrew/split

  20. • Simple, modular and hackable • Designed with rack in

    mind • Small and lightweight • Rails, sinatra and padrino support • Sweet dashboard What’s different?
  21. • Redis is awesome • Rack is always there •

    Don’t manage sessions Design decisions
  22. • Swap css stylesheets • Render different templates or partials

    • Set different variables at the controller level • Hack up the page with javascript Different Approaches
  23. Extending Split • Split::Experiment • Split::Alternative • CRUD interface

  24. Example # app/views/home/index.html.erb <% ab_test("signup_button", "orignal.jpg", "awesome.jpg") do |button| %>

    <%= link_to img_tag("/images/#{button}", :alt => "Signup!"), signup_path %> <% end %> # app/controllers/home_controller.rb def signup # some business logic finished("signup_button") redirect_to dashboard_path end
  25. Dashboard

  26. Significance and Confidence http://visualwebsiteoptimizer.com/ab-split-significance-calculator 99.9%

  27. Weighted Averages ab_test('homepage design', 'old' => 9, 'new' => 1)

    90% see ‘old’ homepage 10% see ‘new’ homepage
  28. Overriding alternatives http://nyan.cat/?rainbows=moar http://nyan.cat/?rainbows=less

  29. Split::Analytics github.com/andrew/split-analytics

  30. Tips

  31. What to test?

  32. Test changes b

  33. http://bit.ly/37faces

  34. How long to run you tests for? http://visualwebsiteoptimizer.com/ab-split-test-duration/ ≈ 2

    w
  35. Testing the tests http://rspec.org/testers?gunna=test

  36. Spiders, bots and snake oil # config/initializers/split.rb Split.configure do |config|

    config.robot_regex = /my_custom_robot_regex/ config.ignore_ip_addresses << '81.19.48.130' end # robots.txt User-Agent: * Disallow: /
  37. Don’t cross the streams

  38. Test all the f***’in time

  39. • JSON Api • Javascript library • Support for different

    data stores • Other language support • Hosted data store Future features
  40. • http://github.com/andrew/split • http://abtests.com • http://visualwebsiteoptimizer.com/split-testing-blog • http://forwardtechnology.co.uk Resources

  41. “If you’re not prepared to be wrong, you’ll never come

    up with anything original” S r K R b
  42. Q ?