Upgrade to Pro — share decks privately, control downloads, hide ads and more …

An Introduction to A/B Testing

timminor
October 03, 2011
500

An Introduction to A/B Testing

Barcamp talk given at UX Camp Brighton 2011.

timminor

October 03, 2011
Tweet

Transcript

  1. @"mminor   An Introduction to A/B Testing Photo  credit:  @lucyjspence

     and  h7p://www.flickr.com/photos/futureshape/4530140521/  
  2. What we’ll be looking at •  What  is  A/B  Tes"ng?

      •  What  to  test?   •  What  is  A/B  Tes"ng  good  for?   •  A/B  tes"ng  should  not  replace   •  Do’s  and  Don’ts   •  Tools  
  3. What to test? •  Bu7ons:  colours,  sizes,  loca"ons   • 

    Headlines   •  Product  descrip"ons   •  Form  fields  (adding  and  removing)   •  PPC  landing  pages   •  Email  content   •  Pricing  and  promo"ons   •  Images  
  4. What is A/B Testing good for? •  Data-­‐driven  design  

    •  Ge\ng  stakeholders  on-­‐board   •  Avoiding  personal  preferences  
  5. A/B testing should not replace: •  Talking  to  users  

    •  Usability  tests   •  Acceptance  tests  
  6. Do’s and Don’ts •  Test  simultaneously   •  Test  o^en

      •  Include  only  new  visitors   •  Show  repeat  visitors  the  same  version   •  Use  results  to  inform  future  development   •  Require  sta"s"cal  significance   DO:
  7. •  Test  too  many  ideas  at  once   •  No

     clear  idea   •  Don’t  let  your  preconcep"ons  override  results   •  Peek  at  your  results  and  stop  as  soon  as  a   significant  result  has  been  achieved.     See:  Evan  Miller.   Do’s and Don’ts DON’T:
  8. The tools Other  tools  are  available:   A/Bingo,  Vanity,  Unbounce,

     Performable,  Vertster,  SiteSpect,     Webtrends  Op"mize  and  Omniture’s  Test&Target  
  9. A

  10. B

  11. B

  12. A

  13. A

  14. Quick Fire Round •  What  were  the  two  main  tools

     men"oned?   •  What  percentage  increase  did  the  winning   RealGap  test  achieve?   •  Name  one  of  the  three  things  A/B  tes"ng   doesn’t  replace.  
  15. What we looked at •  What  is  A/B  Tes"ng?  

    •  What  to  test?   •  What  is  A/B  Tes"ng  good  for?   •  A/B  tes"ng  should  not  replace   •  Do’s  and  Don’ts   •  Tools  
  16. References •  First  slide  photograph:  h7p://www.flickr.com/photos/futureshape/4530140521/     •  www.google.com/websiteop"mizer

        •  h7p://visualwebsiteop"mizer.com/   •  h7p://www.abtests.com/   •  h7p://whichtestwon.com/     •  h7p://www.smashingmagazine.com/2010/06/24/the-­‐ul"mate-­‐guide-­‐to-­‐a-­‐b-­‐tes"ng/   •  h7p://www.conversion-­‐rate-­‐experts.com/ar"cles/101-­‐google-­‐website-­‐op"mizer-­‐"ps/   •  h7p://www.evanmiller.org/how-­‐not-­‐to-­‐run-­‐an-­‐ab-­‐test.html   •  h7p://elem.com/~b"lly/effec"ve-­‐ab-­‐tes"ng/   •  h7p://remoteresear.ch/