An Introduction to A/B Testing

Barcamp talk given at UX Camp Brighton 2011.

Transcript

1. @"mminor   An Introduction to A/B Testing Photo  credit:  @lucyjspence  and  h7p://www.flickr.com/photos/futureshape/4530140521/  
2. What we’ll be looking at •  What  is  A/B  Tes"ng?   •  What  to  test?   •  What  is  A/B  Tes"ng  good  for?   •  A/B  tes"ng  should  not  replace   •  Do’s  and  Don’ts   •  Tools  
3. Source:  h7p://www.smashingmagazine.com/2010/06/24/the-­‐ul"mate-­‐guide-­‐to-­‐a-­‐b-­‐tes"ng/   What is A/B Testing?
4. What to test? •  Bu7ons:  colours,  sizes,  loca"ons   •  Headlines   •  Product  descrip"ons   •  Form  fields  (adding  and  removing)   •  PPC  landing  pages   •  Email  content   •  Pricing  and  promo"ons   •  Images  
5. What is A/B Testing good for? •  Data-­‐driven  design   •  Ge\ng  stakeholders  on-­‐board   •  Avoiding  personal  preferences  
6. A/B testing should not replace: •  Talking  to  users   •  Usability  tests   •  Acceptance  tests  
7. Do’s and Don’ts •  Test  simultaneously   •  Test  o^en   •  Include  only  new  visitors   •  Show  repeat  visitors  the  same  version   •  Use  results  to  inform  future  development   •  Require  sta"s"cal  significance   DO:
8. •  Test  too  many  ideas  at  once   •  No  clear  idea   •  Don’t  let  your  preconcep"ons  override  results   •  Peek  at  your  results  and  stop  as  soon  as  a   significant  result  has  been  achieved.     See:  Evan  Miller.   Do’s and Don’ts DON’T:
9. The tools Other  tools  are  available:   A/Bingo,  Vanity,  Unbounce,  Performable,  Vertster,  SiteSpect,     Webtrends  Op"mize  and  Omniture’s  Test&Target  
10. The Quiz Show
11. The Corkscrew A
12. The Corkscrew B
13. The Corkscrew A/B
14. A
15. Great American Days A
16. Great American Days B
17. Great American Days A/B
18. B
19. Real Gap
20. Real Gap A/B
21. B
22. CityCliq A
23. CityCliq B
24. CityCliq A/B
25. A
26. Vendio A
27. Vendio B
28. Vendio B
29. A
30. Quick Fire Round •  What  were  the  two  main  tools  men"oned?   •  What  percentage  increase  did  the  winning   RealGap  test  achieve?   •  Name  one  of  the  three  things  A/B  tes"ng   doesn’t  replace.  
31. What we looked at •  What  is  A/B  Tes"ng?   •  What  to  test?   •  What  is  A/B  Tes"ng  good  for?   •  A/B  tes"ng  should  not  replace   •  Do’s  and  Don’ts   •  Tools  
32. References •  First  slide  photograph:  h7p://www.flickr.com/photos/futureshape/4530140521/     •  www.google.com/websiteop"mizer     •  h7p://visualwebsiteop"mizer.com/   •  h7p://www.abtests.com/   •  h7p://whichtestwon.com/     •  h7p://www.smashingmagazine.com/2010/06/24/the-­‐ul"mate-­‐guide-­‐to-­‐a-­‐b-­‐tes"ng/   •  h7p://www.conversion-­‐rate-­‐experts.com/ar"cles/101-­‐google-­‐website-­‐op"mizer-­‐"ps/   •  h7p://www.evanmiller.org/how-­‐not-­‐to-­‐run-­‐an-­‐ab-­‐test.html   •  h7p://elem.com/~b"lly/effec"ve-­‐ab-­‐tes"ng/   •  h7p://remoteresear.ch/