Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A/B Testing everything with Split

A/B Testing everything with Split

Just because the tests past doesn't mean that your users will actually like the change, you can test your features in front of real users using A/B testing. A/B testing is becoming an important tool on the web to test out new features on a subset of your users to find out if they fulfil their purpose and make your decisions based on data not opinions. Split is a rack based A/B testing framework, it works with Rails, Sinatra and any other rack based web framework. Find out why we needed another framework and how to use in Rails, Sinatra and Radiant and the best practises of A/B testing.

Andrew Nesbitt

October 07, 2011
Tweet

More Decks by Andrew Nesbitt

Other Decks in Technology

Transcript

  1. A/B TESTING
    SPLIT
    w

    View full-size slide

  2. github.com/andrew
    @ b

    View full-size slide

  3. • Your code
    • Your app
    • Your infrastructure
    • Your tests
    You probably test..

    View full-size slide

  4. Do you test your users?

    View full-size slide

  5. How do you know that you’re
    making the of your features

    View full-size slide

  6. How do you know if your
    are succeeding?

    View full-size slide

  7. If you’re not testing, you don’t know

    View full-size slide

  8. • Lean
    • Short feedback loops
    • Data driven development
    Why bother?

    View full-size slide

  9. • Signups
    • Conversions
    • Usage
    • Pageviews
    • Upgrades
    Metrics

    View full-size slide

  10. Are you making the most
    out of your pages?

    View full-size slide

  11. “Every time man makes a new
    experiment he always learns more.
    He cannot learn less.”
    R r B r F r

    View full-size slide

  12. C r
    +30%
    +27%
    +15%
    +7%
    http://bit.ly/37tests

    View full-size slide

  13. • Google web optimizer
    • Visual Website Optimizer
    • Optimizely
    Software as a Service

    View full-size slide

  14. • Abingo
    • SevenMinuteAbs
    • Vanity
    A/B Testing Frameworks

    View full-size slide

  15. Split
    github.com/andrew/split

    View full-size slide

  16. • Simple, modular and hackable
    • Designed with rack in mind
    • Small and lightweight
    • Rails, sinatra and padrino support
    • Sweet dashboard
    What’s different?

    View full-size slide

  17. • Redis is awesome
    • Rack is always there
    • Don’t manage sessions
    Design decisions

    View full-size slide

  18. • Swap css stylesheets
    • Render different templates or partials
    • Set different variables at the controller level
    • Hack up the page with javascript
    Different Approaches

    View full-size slide

  19. Extending Split
    • Split::Experiment
    • Split::Alternative
    • CRUD interface

    View full-size slide

  20. Example
    # app/views/home/index.html.erb
    <% ab_test("signup_button", "orignal.jpg", "awesome.jpg") do |button| %>
    <%= link_to img_tag("/images/#{button}", :alt => "Signup!"), signup_path %>
    <% end %>
    # app/controllers/home_controller.rb
    def signup
    # some business logic
    finished("signup_button")
    redirect_to dashboard_path
    end

    View full-size slide

  21. Significance and Confidence
    http://visualwebsiteoptimizer.com/ab-split-significance-calculator
    99.9%

    View full-size slide

  22. Weighted Averages
    ab_test('homepage design', 'old' => 9, 'new' => 1)
    90% see ‘old’ homepage
    10% see ‘new’ homepage

    View full-size slide

  23. Overriding alternatives
    http://nyan.cat/?rainbows=moar
    http://nyan.cat/?rainbows=less

    View full-size slide

  24. Split::Analytics
    github.com/andrew/split-analytics

    View full-size slide

  25. What to test?

    View full-size slide

  26. Test changes
    b

    View full-size slide

  27. http://bit.ly/37faces

    View full-size slide

  28. How long to run you tests for?
    http://visualwebsiteoptimizer.com/ab-split-test-duration/
    ≈ 2 w

    View full-size slide

  29. Testing the tests
    http://rspec.org/testers?gunna=test

    View full-size slide

  30. Spiders, bots and snake oil
    # config/initializers/split.rb
    Split.configure do |config|
    config.robot_regex = /my_custom_robot_regex/
    config.ignore_ip_addresses << '81.19.48.130'
    end
    # robots.txt
    User-Agent: *
    Disallow: /

    View full-size slide

  31. Don’t cross the streams

    View full-size slide

  32. Test all the f***’in time

    View full-size slide

  33. • JSON Api
    • Javascript library
    • Support for different data stores
    • Other language support
    • Hosted data store
    Future features

    View full-size slide

  34. • http://github.com/andrew/split
    • http://abtests.com
    • http://visualwebsiteoptimizer.com/split-testing-blog
    • http://forwardtechnology.co.uk
    Resources

    View full-size slide

  35. “If you’re not prepared to be
    wrong, you’ll never come up with
    anything original”
    S r K R b

    View full-size slide