A/B Testing everything with Split

Slide 1

Slide 1 text

A/B TESTING SPLIT w

Slide 2

Slide 2 text

github.com/andrew @ b

Slide 3

Slide 3 text

• Your code • Your app • Your infrastructure • Your tests You probably test..

Slide 4

Slide 4 text

Do you test your users?

Slide 5

Slide 5 text

How do you know that you’re making the of your features

Slide 6

Slide 6 text

How do you know if your are succeeding?

Slide 7

Slide 7 text

If you’re not testing, you don’t know

Slide 8

Slide 8 text

• Lean • Short feedback loops • Data driven development Why bother?

Slide 9

Slide 9 text

• Signups • Conversions • Usage • Pageviews • Upgrades Metrics

Slide 10

Slide 10 text

$$$

Slide 11

Slide 11 text

€€€

Slide 12

Slide 12 text

Are you making the most out of your pages?

Slide 13

Slide 13 text

“Every time man makes a new experiment he always learns more. He cannot learn less.” R r B r F r

Slide 14

Slide 14 text

How?

Slide 15

Slide 15 text

A/B Testing

Slide 16

Slide 16 text

C r +30% +27% +15% +7% http://bit.ly/37tests

Slide 17

Slide 17 text

• Google web optimizer • Visual Website Optimizer • Optimizely Software as a Service

Slide 18

Slide 18 text

• Abingo • SevenMinuteAbs • Vanity A/B Testing Frameworks

Slide 19

Slide 19 text

Split github.com/andrew/split

Slide 20

Slide 20 text

• Simple, modular and hackable • Designed with rack in mind • Small and lightweight • Rails, sinatra and padrino support • Sweet dashboard What’s different?

Slide 21

Slide 21 text

• Redis is awesome • Rack is always there • Don’t manage sessions Design decisions

Slide 22

Slide 22 text

• Swap css stylesheets • Render different templates or partials • Set different variables at the controller level • Hack up the page with javascript Different Approaches

Slide 23

Slide 23 text

Extending Split • Split::Experiment • Split::Alternative • CRUD interface

Slide 24

Slide 24 text

Example # app/views/home/index.html.erb <% ab_test("signup_button", "orignal.jpg", "awesome.jpg") do |button| %> <%= link_to img_tag("/images/#{button}", :alt => "Signup!"), signup_path %> <% end %> # app/controllers/home_controller.rb def signup # some business logic finished("signup_button") redirect_to dashboard_path end

Slide 25

Slide 25 text

Dashboard

Slide 26

Slide 26 text

Significance and Confidence http://visualwebsiteoptimizer.com/ab-split-significance-calculator 99.9%

Slide 27

Slide 27 text

Weighted Averages ab_test('homepage design', 'old' => 9, 'new' => 1) 90% see ‘old’ homepage 10% see ‘new’ homepage

Slide 28

Slide 28 text

Overriding alternatives http://nyan.cat/?rainbows=moar http://nyan.cat/?rainbows=less

Slide 29

Slide 29 text

Split::Analytics github.com/andrew/split-analytics

Slide 30

Slide 30 text

Tips

Slide 31

Slide 31 text

What to test?

Slide 32

Slide 32 text

Test changes b

Slide 33

Slide 33 text

http://bit.ly/37faces

Slide 34

Slide 34 text

How long to run you tests for? http://visualwebsiteoptimizer.com/ab-split-test-duration/ ≈ 2 w

Slide 35

Slide 35 text

Testing the tests http://rspec.org/testers?gunna=test

Slide 36

Slide 36 text

Spiders, bots and snake oil # config/initializers/split.rb Split.configure do |config| config.robot_regex = /my_custom_robot_regex/ config.ignore_ip_addresses << '81.19.48.130' end # robots.txt User-Agent: * Disallow: /

Slide 37

Slide 37 text

Don’t cross the streams

Slide 38

Slide 38 text

Test all the f***’in time

Slide 39

Slide 39 text

• JSON Api • Javascript library • Support for different data stores • Other language support • Hosted data store Future features

Slide 40

Slide 40 text

• http://github.com/andrew/split • http://abtests.com • http://visualwebsiteoptimizer.com/split-testing-blog • http://forwardtechnology.co.uk Resources

Slide 41

Slide 41 text

“If you’re not prepared to be wrong, you’ll never come up with anything original” S r K R b

Slide 42

Slide 42 text

Q ?