A/B Testing everything with Split

Transcript

1. A/B TESTING SPLIT w

2. github.com/andrew @ b

3. • Your code • Your app • Your infrastructure •

   Your tests You probably test..

4. Do you test your users?

5. How do you know that you’re making the of your

   features

6. How do you know if your are succeeding?

7. If you’re not testing, you don’t know

8. • Lean • Short feedback loops • Data driven development

   Why bother?

9. • Signups • Conversions • Usage • Pageviews • Upgrades

   Metrics

10. $$$

11. €€€

12. Are you making the most out of your pages?

13. “Every time man makes a new experiment he always learns

    more. He cannot learn less.” R r B r F r

14. How?

15. A/B Testing

16. C r +30% +27% +15% +7% http://bit.ly/37tests

17. • Google web optimizer • Visual Website Optimizer • Optimizely

    Software as a Service

18. • Abingo • SevenMinuteAbs • Vanity A/B Testing Frameworks

19. Split github.com/andrew/split

20. • Simple, modular and hackable • Designed with rack in

    mind • Small and lightweight • Rails, sinatra and padrino support • Sweet dashboard What’s different?

21. • Redis is awesome • Rack is always there •

    Don’t manage sessions Design decisions

22. • Swap css stylesheets • Render different templates or partials

    • Set different variables at the controller level • Hack up the page with javascript Different Approaches

23. Extending Split • Split::Experiment • Split::Alternative • CRUD interface

24. Example # app/views/home/index.html.erb <% ab_test("signup_button", "orignal.jpg", "awesome.jpg") do |button| %>

    <%= link_to img_tag("/images/#{button}", :alt => "Signup!"), signup_path %> <% end %> # app/controllers/home_controller.rb def signup # some business logic finished("signup_button") redirect_to dashboard_path end

25. Dashboard

26. Significance and Confidence http://visualwebsiteoptimizer.com/ab-split-significance-calculator 99.9%

27. Weighted Averages ab_test('homepage design', 'old' => 9, 'new' => 1)

    90% see ‘old’ homepage 10% see ‘new’ homepage

28. Overriding alternatives http://nyan.cat/?rainbows=moar http://nyan.cat/?rainbows=less

29. Split::Analytics github.com/andrew/split-analytics

30. Tips

31. What to test?

32. Test changes b

33. http://bit.ly/37faces

34. How long to run you tests for? http://visualwebsiteoptimizer.com/ab-split-test-duration/ ≈ 2

    w

35. Testing the tests http://rspec.org/testers?gunna=test

36. Spiders, bots and snake oil # config/initializers/split.rb Split.configure do |config|

    config.robot_regex = /my_custom_robot_regex/ config.ignore_ip_addresses << '81.19.48.130' end # robots.txt User-Agent: * Disallow: /

37. Don’t cross the streams

38. Test all the f***’in time

39. • JSON Api • Javascript library • Support for different

    data stores • Other language support • Hosted data store Future features

40. • http://github.com/andrew/split • http://abtests.com • http://visualwebsiteoptimizer.com/split-testing-blog • http://forwardtechnology.co.uk Resources

41. “If you’re not prepared to be wrong, you’ll never come

    up with anything original” S r K R b

42. Q ?