Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A/B Testing to Create Happy Users

245cee81a9c424266e5e401d844ea881?s=47 Lara Hogan
September 23, 2012

A/B Testing to Create Happy Users

We barely know our users. We know what technology they use, where they live, what they look at on our site - but we don't know their personality or decision-making process. However, A/B testing can give you insight into what works best for your audience on your site. It can make your stakeholders happy, it'll make your users happy, and it'll definitely make you happy.

245cee81a9c424266e5e401d844ea881?s=128

Lara Hogan

September 23, 2012
Tweet

Transcript

  1. A/B Testing to Create Happy Users dyn.com/abtest for slides, links

    and extra stu f
  2. @laraswanson

  3. @laraswanson We have no idea what's going on in our

    users' heads.
  4. @laraswanson We rarely check to see if guesses about our

    users are right.
  5. @laraswanson We have some options. eye-tracking field studies surveys/interviews A/B

    testing
  6. @laraswanson We have some options. eye-tracking field studies surveys/interviews A/B

    testing
  7. @laraswanson illustration by Kevin Cornell

  8. @laraswanson illustration by Kevin Cornell Content

  9. @laraswanson illustration by Kevin Cornell Content Design

  10. @laraswanson illustration by Kevin Cornell Content Design Layout

  11. @laraswanson illustration by Kevin Cornell Content Design Layout Action

  12. A/B testing may be what you need.

  13. @laraswanson no user survey could give you this info

  14. @laraswanson subtle, continuous improvements

  15. @laraswanson solve a design disagreement

  16. @laraswanson unique to your site and users

  17. Getting Started

  18. @laraswanson Test screams (not whispers)

  19. @laraswanson Test screams (not whispers) call to action text form

    length subject lines 41 shades of blue landing page design
  20. @laraswanson Test screams (not whispers) call to action text form

    length subject lines 41 shades of blue landing page design
  21. @laraswanson Define your audience

  22. @laraswanson Define your audience undergrad/grad site goals geography desktop/mobile academic

    program
  23. @laraswanson Define success

  24. @laraswanson Define success form conversions pages per visit churn rate

    donations error rate
  25. Guess the winner!

  26. @laraswanson http://whichtestwon.com/archives/7526 A B

  27. @laraswanson http://whichtestwon.com/archives/7526 19% more conversions B

  28. @laraswanson Give it enough time me

  29. @laraswanson Give it enough time Kill a dead test

  30. @laraswanson http://www.usere fect.com/split-test-calculator Visitors Goals A B CALCULATE

  31. @laraswanson http://www.usere fect.com/split-test-calculator Visitors Goals A 200 B 180 CALCULATE

  32. @laraswanson http://www.usere fect.com/split-test-calculator Visitors Goals A 200 26 B 180

    52 CALCULATE
  33. @laraswanson http://www.usere fect.com/split-test-calculator Visitors Goals A 200 26 B 180

    52 99.9% certain B is the winner!
  34. @laraswanson http://www.usere fect.com/split-test-calculator Visitors Goals A 200 26 B 180

    24 CALCULATE
  35. @laraswanson http://www.usere fect.com/split-test-calculator Visitors Goals A 200 26 B 180

    24 no clear winner :(
  36. Technical Stu f

  37. @laraswanson http://www.whichmvt.com/ A/B testing technology

  38. @laraswanson http://www.whichmvt.com/ A/B testing technology initial implementation di ficulty

  39. @laraswanson http://www.whichmvt.com/ A/B testing technology initial implementation di ficulty ease

    of editing by non-techies
  40. @laraswanson http://www.whichmvt.com/ A/B testing technology initial implementation di ficulty ease

    of editing by non-techies page load time
  41. @laraswanson http://www.whichmvt.com/ A/B testing technology initial implementation di ficulty ease

    of editing by non-techies page load time integration with analytics
  42. @laraswanson Audience segmentation academic program mobile/desktop geography

  43. @laraswanson http://www.whichmvt.com/visual-website-optimizer/ Example: Visual Website Optimizer

  44. @laraswanson http://www.whichmvt.com/visual-website-optimizer/ Example: Visual Website Optimizer WYSIWYG view of tests

  45. @laraswanson http://www.whichmvt.com/visual-website-optimizer/ Example: Visual Website Optimizer WYSIWYG view of tests

    integration with GA
  46. @laraswanson http://www.whichmvt.com/visual-website-optimizer/ Example: Visual Website Optimizer WYSIWYG view of tests

    integration with GA calculates significance
  47. @laraswanson http://www.whichmvt.com/visual-website-optimizer/ Example: Visual Website Optimizer WYSIWYG view of tests

    integration with GA calculates significance page load time-optimized
  48. @laraswanson http://www.whichmvt.com/visual-website-optimizer/ Example: Visual Website Optimizer WYSIWYG view of tests

    integration with GA calculates significance page load time-optimized can do multiple tests on one page
  49. @laraswanson http://www.whichmvt.com/visual-website-optimizer/ Example: Visual Website Optimizer WYSIWYG view of tests

    integration with GA calculates significance page load time-optimized can do multiple tests on one page pricing starts at $49/month
  50. Guess the winner!

  51. @laraswanson http://whichtestwon.com/archives/2450 A B

  52. @laraswanson http://whichtestwon.com/archives/2450 73% more conversions A

  53. Google Analytics integration

  54. @laraswanson Version A _gaq.push(['_setCustomVar',1,'Product Comparison','Control',2]); Version B _gaq.push(['_setCustomVar',1,'Product Comparison','Post It',2]);

  55. Version A _gaq.push(['_setCustomVar',1,'Product Comparison','Control',2]); Version B _gaq.push(['_setCustomVar',1,'Product Comparison','Post It',2]); @laraswanson

    custom variable slot custom variable slot
  56. Version A _gaq.push(['_setCustomVar',1,'Product Comparison','Control',2]); Version B _gaq.push(['_setCustomVar',1,'Product Comparison','Post It',2]); @laraswanson

    name of your A/B test name of your A/B test
  57. Version A _gaq.push(['_setCustomVar',1,'Product Comparison','Control',2]); Version B _gaq.push(['_setCustomVar',1,'Product Comparison','Post It',2]); @laraswanson

    which version the user saw which version the user saw
  58. Version A _gaq.push(['_setCustomVar',1,'Product Comparison','Control',2]); Version B _gaq.push(['_setCustomVar',1,'Product Comparison','Post It',2]); @laraswanson

    session-level setting session-level setting
  59. @laraswanson Google Analytics > Audience > Custom > Custom Variables

  60. @laraswanson

  61. @laraswanson

  62. @laraswanson

  63. Guess the winner!

  64. @laraswanson http://whichtestwon.com/archives/8413 A B

  65. @laraswanson http://whichtestwon.com/archives/8413 More PhD candidates submitted request for information B

  66. A/B tests will make you mad.

  67. @laraswanson valuable development time

  68. @laraswanson may not reach significant results

  69. @laraswanson you'll be excited and then maybe disappointed

  70. @laraswanson A/B tests will prove you wrong

  71. A/B tests will make your users happy.

  72. @laraswanson your site will be easier to use

  73. @laraswanson users will accomplish what they came to do

  74. A/B tests will make your stakeholders happy.

  75. @laraswanson better conversion rates

  76. @laraswanson alumni engagement

  77. @laraswanson insight for each audience segment

  78. A/B tests will make you mad happy.

  79. @laraswanson knowing why is better than knowing how

  80. @laraswanson impress your stakeholders

  81. @laraswanson data to say "NO" to bad ideas

  82. Talking to stakeholders

  83. @laraswanson Stakeholder: I heard from a large donor that it's

    hard to donate online.
  84. @laraswanson Decoder ring: Online donations are down, someone complained about

    the online experience, I'm lost when it comes to the web so I need your help.
  85. @laraswanson You: I'm ready to help! We can actually test

    a few di ferent versions of the form to see which gets more donations. Decoder ring: Online donations are down, someone complained about the online experience, I'm lost when it comes to the web so I need your help.
  86. @laraswanson Stakeholder: Really? You can't just fix it right now?

  87. @laraswanson Decoder ring: I don't understand why we'd wait any

    longer.
  88. @laraswanson You: By running a test we can statistically determine

    which version of the form is better. Decoder ring: I don't understand why we'd wait any longer.
  89. @laraswanson You: It avoids putting up a fix now that

    may actually make things worse, because we won't have any data about how users respond to it. Decoder ring: I don't understand why we'd wait any longer.
  90. @laraswanson Stakeholder: I don't understand. How would people see the

    di ferent versions?
  91. @laraswanson Decoder ring: This is sounding very technical. Blurgh. JUST

    FIX IT!
  92. @laraswanson You: Half of visitors will see a new version

    that I think will make our users happier and more likely to donate. Decoder ring: This is sounding very technical. Blurgh. JUST FIX IT!
  93. @laraswanson You: The other half will see the current form.

    I'll measure donations from each version so we can see which performs better. Decoder ring: This is sounding very technical. Blurgh. JUST FIX IT!
  94. @laraswanson Stakeholder: And then what? Then you'll fix it?

  95. @laraswanson Decoder ring: I just want to hear that you're

    going to fix this.
  96. @laraswanson You: You bet. And I can continue to run

    tests on it to improve it more and more over time. Decoder ring: I just want to hear that you're going to fix this.
  97. @laraswanson You: We can make the form better than it's

    ever been and because we'll have data, we'll be confident in its results. Decoder ring: I just want to hear that you're going to fix this.
  98. Guess the winner!

  99. @laraswanson http://whichtestwon.com/archives/14918 A B

  100. @laraswanson http://whichtestwon.com/archives/14918 57% more leads B

  101. @laraswanson http://whichtestwon.com/archives/4781 A B

  102. @laraswanson http://whichtestwon.com/archives/4781 A B

  103. @laraswanson http://whichtestwon.com/archives/4781 A B

  104. @laraswanson http://whichtestwon.com/archives/4781 17% more conversions Wednesday and Thursday a ternoons

    B
  105. @laraswanson http://whichtestwon.com/archives/4781 22% more conversions on weekday and weekend mornings

    A
  106. @laraswanson http://whichtestwon.com/archives/2655 A B

  107. @laraswanson http://whichtestwon.com/archives/2655 86% more "Subscribe" clicks A

  108. @laraswanson http://whichtestwon.com/archives/7453 A B

  109. @laraswanson http://whichtestwon.com/archives/7453 2666% higher CTR B

  110. @laraswanson http://whichtestwon.com/archives/3026 A B

  111. @laraswanson http://whichtestwon.com/archives/3026 25% more form submissions A

  112. How did you do?

  113. @laraswanson https://twitter.com/katiedel/status/240136499704778754 "Only 10-30% (of 1000's) of experiments improved the

    metrics they were designed the improve"
  114. That's why we need to test.

  115. @laraswanson Alternatives to A/B testing

  116. @laraswanson Alternatives to A/B testing Multivariate testing: http://bit.ly/smashing-mvt

  117. @laraswanson Alternatives to A/B testing Multivariate testing: http://bit.ly/smashing-mvt Multi-armed bandit:

    http://bit.ly/multi-armed-bandit
  118. @laraswanson Alternatives to A/B testing Multivariate testing: http://bit.ly/smashing-mvt Multi-armed bandit:

    http://bit.ly/multi-armed-bandit Tons of tools: http://whichtestwon.com/techguide
  119. Thanks! Questions? dyn.com/abtest @laraswanson dyn.com/careers - we're hiring UI devs!