A/B Testing everything with Split

Just because the tests past doesn't mean that your users will actually like the change, you can test your features in front of real users using A/B testing. A/B testing is becoming an important tool on the web to test out new features on a subset of your users to find out if they fulfil their purpose and make your decisions based on data not opinions. Split is a rack based A/B testing framework, it works with Rails, Sinatra and any other rack based web framework. Find out why we needed another framework and how to use in Rails, Sinatra and Radiant and the best practises of A/B testing.

Transcript

1. A/B TESTING
   SPLIT
   w
   

   View Slide

2. github.com/andrew
   @ b
   

   View Slide

3. • Your code
   • Your app
   • Your infrastructure
   • Your tests
   You probably test..
   

   View Slide

4. Do you test your users?
   

   View Slide

5. How do you know that you’re
   making the of your features
   

   View Slide

6. How do you know if your
   are succeeding?
   

   View Slide

7. If you’re not testing, you don’t know
   

   View Slide

8. • Lean
   • Short feedback loops
   • Data driven development
   Why bother?
   

   View Slide

9. • Signups
   • Conversions
   • Usage
   • Pageviews
   • Upgrades
   Metrics
   

   View Slide

10. $$$
    

    View Slide

11. €€€
    

    View Slide

12. Are you making the most
    out of your pages?
    

    View Slide

13. “Every time man makes a new
    experiment he always learns more.
    He cannot learn less.”
    R r B r F r
    

    View Slide

14. How?
    

    View Slide

15. A/B Testing
    

    View Slide

16. C r
    +30%
    +27%
    +15%
    +7%
    http://bit.ly/37tests
    

    View Slide

17. • Google web optimizer
    • Visual Website Optimizer
    • Optimizely
    Software as a Service
    

    View Slide

18. • Abingo
    • SevenMinuteAbs
    • Vanity
    A/B Testing Frameworks
    

    View Slide

19. Split
    github.com/andrew/split
    

    View Slide

20. • Simple, modular and hackable
    • Designed with rack in mind
    • Small and lightweight
    • Rails, sinatra and padrino support
    • Sweet dashboard
    What’s different?
    

    View Slide

21. • Redis is awesome
    • Rack is always there
    • Don’t manage sessions
    Design decisions
    

    View Slide

22. • Swap css stylesheets
    • Render different templates or partials
    • Set different variables at the controller level
    • Hack up the page with javascript
    Different Approaches
    

    View Slide

23. Extending Split
    • Split::Experiment
    • Split::Alternative
    • CRUD interface
    

    View Slide

24. Example
    # app/views/home/index.html.erb
    <% ab_test("signup_button", "orignal.jpg", "awesome.jpg") do |button| %>
    <%= link_to img_tag("/images/#{button}", :alt => "Signup!"), signup_path %>
    <% end %>
    # app/controllers/home_controller.rb
    def signup
    # some business logic
    finished("signup_button")
    redirect_to dashboard_path
    end
    

    View Slide

25. Dashboard
    

    View Slide

26. Significance and Confidence
    http://visualwebsiteoptimizer.com/ab-split-significance-calculator
    99.9%
    

    View Slide

27. Weighted Averages
    ab_test('homepage design', 'old' => 9, 'new' => 1)
    90% see ‘old’ homepage
    10% see ‘new’ homepage
    

    View Slide

28. Overriding alternatives
    http://nyan.cat/?rainbows=moar
    http://nyan.cat/?rainbows=less
    

    View Slide

29. Split::Analytics
    github.com/andrew/split-analytics
    

    View Slide

30. Tips
    

    View Slide

31. What to test?
    

    View Slide

32. Test changes
    b
    

    View Slide

33. http://bit.ly/37faces
    

    View Slide

34. How long to run you tests for?
    http://visualwebsiteoptimizer.com/ab-split-test-duration/
    ≈ 2 w
    

    View Slide

35. Testing the tests
    http://rspec.org/testers?gunna=test
    

    View Slide

36. Spiders, bots and snake oil
    # config/initializers/split.rb
    Split.configure do |config|
    config.robot_regex = /my_custom_robot_regex/
    config.ignore_ip_addresses << '81.19.48.130'
    end
    # robots.txt
    User-Agent: *
    Disallow: /
    

    View Slide

37. Don’t cross the streams
    

    View Slide

38. Test all the f***’in time
    

    View Slide

39. • JSON Api
    • Javascript library
    • Support for different data stores
    • Other language support
    • Hosted data store
    Future features
    

    View Slide

40. • http://github.com/andrew/split
    • http://abtests.com
    • http://visualwebsiteoptimizer.com/split-testing-blog
    • http://forwardtechnology.co.uk
    Resources
    

    View Slide

41. “If you’re not prepared to be
    wrong, you’ll never come up with
    anything original”
    S r K R b
    

    View Slide

42. Q ?
    

    View Slide