A/B Testing to Create Happy Users

A/B Testing to Create Happy Users dyn.com/abtest for slides, links
and extra stu f

@laraswanson

@laraswanson We have no idea what's going on in our
users' heads.

@laraswanson We rarely check to see if guesses about our
users are right.

@laraswanson We have some options. eye-tracking ﬁeld studies surveys/interviews A/B
testing

@laraswanson illustration by Kevin Cornell

@laraswanson illustration by Kevin Cornell Content

@laraswanson illustration by Kevin Cornell Content Design

@laraswanson illustration by Kevin Cornell Content Design Layout

@laraswanson illustration by Kevin Cornell Content Design Layout Action

A/B testing may be what you need.

@laraswanson no user survey could give you this info

@laraswanson subtle, continuous improvements

@laraswanson solve a design disagreement

@laraswanson unique to your site and users

Getting Started

@laraswanson Test screams (not whispers)

@laraswanson Test screams (not whispers) call to action text form
length subject lines 41 shades of blue landing page design

@laraswanson Deﬁne your audience

@laraswanson Deﬁne your audience undergrad/grad site goals geography desktop/mobile academic
program

@laraswanson Deﬁne success

@laraswanson Deﬁne success form conversions pages per visit churn rate
donations error rate

Guess the winner!

@laraswanson http://whichtestwon.com/archives/7526 A B

@laraswanson http://whichtestwon.com/archives/7526 19% more conversions B

@laraswanson Give it enough time me

@laraswanson Give it enough time Kill a dead test

@laraswanson http://www.usere fect.com/split-test-calculator Visitors Goals A B CALCULATE

@laraswanson http://www.usere fect.com/split-test-calculator Visitors Goals A 200 B 180 CALCULATE

@laraswanson http://www.usere fect.com/split-test-calculator Visitors Goals A 200 26 B 180
52 CALCULATE

52 99.9% certain B is the winner!

24 CALCULATE

24 no clear winner :(

Technical Stu f

@laraswanson http://www.whichmvt.com/ A/B testing technology

@laraswanson http://www.whichmvt.com/ A/B testing technology initial implementation di ﬁculty

@laraswanson http://www.whichmvt.com/ A/B testing technology initial implementation di ﬁculty ease
of editing by non-techies

of editing by non-techies page load time

of editing by non-techies page load time integration with analytics

@laraswanson Audience segmentation academic program mobile/desktop geography

@laraswanson http://www.whichmvt.com/visual-website-optimizer/ Example: Visual Website Optimizer

@laraswanson http://www.whichmvt.com/visual-website-optimizer/ Example: Visual Website Optimizer WYSIWYG view of tests

integration with GA

integration with GA calculates signiﬁcance

integration with GA calculates signiﬁcance page load time-optimized

integration with GA calculates signiﬁcance page load time-optimized can do multiple tests on one page

integration with GA calculates signiﬁcance page load time-optimized can do multiple tests on one page pricing starts at $49/month

Guess the winner!

@laraswanson http://whichtestwon.com/archives/2450 73% more conversions A

Google Analytics integration

@laraswanson Version A _gaq.push(['_setCustomVar',1,'Product Comparison','Control',2]); Version B _gaq.push(['_setCustomVar',1,'Product Comparison','Post It',2]);

Version A _gaq.push(['_setCustomVar',1,'Product Comparison','Control',2]); Version B _gaq.push(['_setCustomVar',1,'Product Comparison','Post It',2]); @laraswanson
custom variable slot custom variable slot

name of your A/B test name of your A/B test

which version the user saw which version the user saw

session-level setting session-level setting

@laraswanson Google Analytics > Audience > Custom > Custom Variables

@laraswanson

Guess the winner!

@laraswanson http://whichtestwon.com/archives/8413 More PhD candidates submitted request for information B

A/B tests will make you mad.

@laraswanson valuable development time

@laraswanson may not reach signiﬁcant results

@laraswanson you'll be excited and then maybe disappointed

@laraswanson A/B tests will prove you wrong

A/B tests will make your users happy.

@laraswanson your site will be easier to use

@laraswanson users will accomplish what they came to do

A/B tests will make your stakeholders happy.

@laraswanson better conversion rates

@laraswanson alumni engagement

@laraswanson insight for each audience segment

A/B tests will make you mad happy.

@laraswanson knowing why is better than knowing how

@laraswanson impress your stakeholders

@laraswanson data to say "NO" to bad ideas

Talking to stakeholders

@laraswanson Stakeholder: I heard from a large donor that it's
hard to donate online.

@laraswanson Decoder ring: Online donations are down, someone complained about
the online experience, I'm lost when it comes to the web so I need your help.

@laraswanson You: I'm ready to help! We can actually test
a few di ferent versions of the form to see which gets more donations. Decoder ring: Online donations are down, someone complained about the online experience, I'm lost when it comes to the web so I need your help.

@laraswanson Stakeholder: Really? You can't just ﬁx it right now?

@laraswanson Decoder ring: I don't understand why we'd wait any
longer.

@laraswanson You: By running a test we can statistically determine
which version of the form is better. Decoder ring: I don't understand why we'd wait any longer.

@laraswanson You: It avoids putting up a ﬁx now that
may actually make things worse, because we won't have any data about how users respond to it. Decoder ring: I don't understand why we'd wait any longer.

@laraswanson Stakeholder: I don't understand. How would people see the
di ferent versions?

@laraswanson Decoder ring: This is sounding very technical. Blurgh. JUST
FIX IT!

@laraswanson You: Half of visitors will see a new version
that I think will make our users happier and more likely to donate. Decoder ring: This is sounding very technical. Blurgh. JUST FIX IT!

@laraswanson You: The other half will see the current form.
I'll measure donations from each version so we can see which performs better. Decoder ring: This is sounding very technical. Blurgh. JUST FIX IT!

@laraswanson Stakeholder: And then what? Then you'll ﬁx it?

@laraswanson Decoder ring: I just want to hear that you're
going to ﬁx this.

@laraswanson You: You bet. And I can continue to run
tests on it to improve it more and more over time. Decoder ring: I just want to hear that you're going to ﬁx this.

@laraswanson You: We can make the form better than it's
ever been and because we'll have data, we'll be conﬁdent in its results. Decoder ring: I just want to hear that you're going to ﬁx this.

Guess the winner!

@laraswanson http://whichtestwon.com/archives/14918 57% more leads B

@laraswanson http://whichtestwon.com/archives/4781 17% more conversions Wednesday and Thursday a ternoons
B

@laraswanson http://whichtestwon.com/archives/4781 22% more conversions on weekday and weekend mornings
A

@laraswanson http://whichtestwon.com/archives/2655 86% more "Subscribe" clicks A

@laraswanson http://whichtestwon.com/archives/7453 2666% higher CTR B

@laraswanson http://whichtestwon.com/archives/3026 25% more form submissions A

How did you do?

@laraswanson https://twitter.com/katiedel/status/240136499704778754 "Only 10-30% (of 1000's) of experiments improved the
metrics they were designed the improve"

That's why we need to test.

@laraswanson Alternatives to A/B testing

@laraswanson Alternatives to A/B testing Multivariate testing: http://bit.ly/smashing-mvt

@laraswanson Alternatives to A/B testing Multivariate testing: http://bit.ly/smashing-mvt Multi-armed bandit:
http://bit.ly/multi-armed-bandit

@laraswanson Alternatives to A/B testing Multivariate testing: http://bit.ly/smashing-mvt Multi-armed bandit:
http://bit.ly/multi-armed-bandit Tons of tools: http://whichtestwon.com/techguide

Thanks! Questions? dyn.com/abtest @laraswanson dyn.com/careers - we're hiring UI devs!

A/B Testing to Create Happy Users

A/B Testing to Create Happy Users

More Decks by Lara Hogan

Other Decks in Design

Featured

Transcript