Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A/B Testing everything with Split

A/B Testing everything with Split

Just because the tests past doesn't mean that your users will actually like the change, you can test your features in front of real users using A/B testing. A/B testing is becoming an important tool on the web to test out new features on a subset of your users to find out if they fulfil their purpose and make your decisions based on data not opinions. Split is a rack based A/B testing framework, it works with Rails, Sinatra and any other rack based web framework. Find out why we needed another framework and how to use in Rails, Sinatra and Radiant and the best practises of A/B testing.

Andrew Nesbitt

October 07, 2011
Tweet

More Decks by Andrew Nesbitt

Other Decks in Technology

Transcript

  1. A/B TESTING
    SPLIT
    w

    View Slide

  2. github.com/andrew
    @ b

    View Slide

  3. • Your code
    • Your app
    • Your infrastructure
    • Your tests
    You probably test..

    View Slide

  4. Do you test your users?

    View Slide

  5. How do you know that you’re
    making the of your features

    View Slide

  6. How do you know if your
    are succeeding?

    View Slide

  7. If you’re not testing, you don’t know

    View Slide

  8. • Lean
    • Short feedback loops
    • Data driven development
    Why bother?

    View Slide

  9. • Signups
    • Conversions
    • Usage
    • Pageviews
    • Upgrades
    Metrics

    View Slide

  10. $$$

    View Slide

  11. €€€

    View Slide

  12. Are you making the most
    out of your pages?

    View Slide

  13. “Every time man makes a new
    experiment he always learns more.
    He cannot learn less.”
    R r B r F r

    View Slide

  14. How?

    View Slide

  15. A/B Testing

    View Slide

  16. C r
    +30%
    +27%
    +15%
    +7%
    http://bit.ly/37tests

    View Slide

  17. • Google web optimizer
    • Visual Website Optimizer
    • Optimizely
    Software as a Service

    View Slide

  18. • Abingo
    • SevenMinuteAbs
    • Vanity
    A/B Testing Frameworks

    View Slide

  19. Split
    github.com/andrew/split

    View Slide

  20. • Simple, modular and hackable
    • Designed with rack in mind
    • Small and lightweight
    • Rails, sinatra and padrino support
    • Sweet dashboard
    What’s different?

    View Slide

  21. • Redis is awesome
    • Rack is always there
    • Don’t manage sessions
    Design decisions

    View Slide

  22. • Swap css stylesheets
    • Render different templates or partials
    • Set different variables at the controller level
    • Hack up the page with javascript
    Different Approaches

    View Slide

  23. Extending Split
    • Split::Experiment
    • Split::Alternative
    • CRUD interface

    View Slide

  24. Example
    # app/views/home/index.html.erb
    <% ab_test("signup_button", "orignal.jpg", "awesome.jpg") do |button| %>
    <%= link_to img_tag("/images/#{button}", :alt => "Signup!"), signup_path %>
    <% end %>
    # app/controllers/home_controller.rb
    def signup
    # some business logic
    finished("signup_button")
    redirect_to dashboard_path
    end

    View Slide

  25. Dashboard

    View Slide

  26. Significance and Confidence
    http://visualwebsiteoptimizer.com/ab-split-significance-calculator
    99.9%

    View Slide

  27. Weighted Averages
    ab_test('homepage design', 'old' => 9, 'new' => 1)
    90% see ‘old’ homepage
    10% see ‘new’ homepage

    View Slide

  28. Overriding alternatives
    http://nyan.cat/?rainbows=moar
    http://nyan.cat/?rainbows=less

    View Slide

  29. Split::Analytics
    github.com/andrew/split-analytics

    View Slide

  30. Tips

    View Slide

  31. What to test?

    View Slide

  32. Test changes
    b

    View Slide

  33. http://bit.ly/37faces

    View Slide

  34. How long to run you tests for?
    http://visualwebsiteoptimizer.com/ab-split-test-duration/
    ≈ 2 w

    View Slide

  35. Testing the tests
    http://rspec.org/testers?gunna=test

    View Slide

  36. Spiders, bots and snake oil
    # config/initializers/split.rb
    Split.configure do |config|
    config.robot_regex = /my_custom_robot_regex/
    config.ignore_ip_addresses << '81.19.48.130'
    end
    # robots.txt
    User-Agent: *
    Disallow: /

    View Slide

  37. Don’t cross the streams

    View Slide

  38. Test all the f***’in time

    View Slide

  39. • JSON Api
    • Javascript library
    • Support for different data stores
    • Other language support
    • Hosted data store
    Future features

    View Slide

  40. • http://github.com/andrew/split
    • http://abtests.com
    • http://visualwebsiteoptimizer.com/split-testing-blog
    • http://forwardtechnology.co.uk
    Resources

    View Slide

  41. “If you’re not prepared to be
    wrong, you’ll never come up with
    anything original”
    S r K R b

    View Slide

  42. Q ?

    View Slide