A/B Testing everything with Split

Just because the tests past doesn't mean that your users will actually like the change, you can test your features in front of real users using A/B testing. A/B testing is becoming an important tool on the web to test out new features on a subset of your users to find out if they fulfil their purpose and make your decisions based on data not opinions. Split is a rack based A/B testing framework, it works with Rails, Sinatra and any other rack based web framework. Find out why we needed another framework and how to use in Rails, Sinatra and Radiant and the best practises of A/B testing.

Transcript

1. A/B TESTING SPLIT w
2. github.com/andrew @ b
3. • Your code • Your app • Your infrastructure • Your tests You probably test..
4. Do you test your users?
5. How do you know that you’re making the of your features
6. How do you know if your are succeeding?
7. If you’re not testing, you don’t know
8. • Lean • Short feedback loops • Data driven development Why bother?
9. • Signups • Conversions • Usage • Pageviews • Upgrades Metrics
10. $$$
11. €€€
12. Are you making the most out of your pages?
13. “Every time man makes a new experiment he always learns more. He cannot learn less.” R r B r F r
14. How?
15. A/B Testing
16. C r +30% +27% +15% +7% http://bit.ly/37tests
17. • Google web optimizer • Visual Website Optimizer • Optimizely Software as a Service
18. • Abingo • SevenMinuteAbs • Vanity A/B Testing Frameworks
19. Split github.com/andrew/split
20. • Simple, modular and hackable • Designed with rack in mind • Small and lightweight • Rails, sinatra and padrino support • Sweet dashboard What’s different?
21. • Redis is awesome • Rack is always there • Don’t manage sessions Design decisions
22. • Swap css stylesheets • Render different templates or partials • Set different variables at the controller level • Hack up the page with javascript Different Approaches
23. Extending Split • Split::Experiment • Split::Alternative • CRUD interface
24. Example # app/views/home/index.html.erb <% ab_test("signup_button", "orignal.jpg", "awesome.jpg") do |button| %> <%= link_to img_tag("/images/#{button}", :alt => "Signup!"), signup_path %> <% end %> # app/controllers/home_controller.rb def signup # some business logic finished("signup_button") redirect_to dashboard_path end
25. Dashboard
26. Significance and Confidence http://visualwebsiteoptimizer.com/ab-split-significance-calculator 99.9%
27. Weighted Averages ab_test('homepage design', 'old' => 9, 'new' => 1) 90% see ‘old’ homepage 10% see ‘new’ homepage
28. Overriding alternatives http://nyan.cat/?rainbows=moar http://nyan.cat/?rainbows=less
29. Split::Analytics github.com/andrew/split-analytics
30. Tips
31. What to test?
32. Test changes b
33. http://bit.ly/37faces
34. How long to run you tests for? http://visualwebsiteoptimizer.com/ab-split-test-duration/ ≈ 2 w
35. Testing the tests http://rspec.org/testers?gunna=test
36. Spiders, bots and snake oil # config/initializers/split.rb Split.configure do |config| config.robot_regex = /my_custom_robot_regex/ config.ignore_ip_addresses << '81.19.48.130' end # robots.txt User-Agent: * Disallow: /
37. Don’t cross the streams
38. Test all the f***’in time
39. • JSON Api • Javascript library • Support for different data stores • Other language support • Hosted data store Future features
40. • http://github.com/andrew/split • http://abtests.com • http://visualwebsiteoptimizer.com/split-testing-blog • http://forwardtechnology.co.uk Resources
41. “If you’re not prepared to be wrong, you’ll never come up with anything original” S r K R b
42. Q ?