@"mminor
An Introduction
to A/B Testing
Photo
credit:
@lucyjspence
and
h7p://www.flickr.com/photos/futureshape/4530140521/
Slide 2
Slide 2 text
What we’ll be looking at
• What
is
A/B
Tes"ng?
• What
to
test?
• What
is
A/B
Tes"ng
good
for?
• A/B
tes"ng
should
not
replace
• Do’s
and
Don’ts
• Tools
Slide 3
Slide 3 text
Source:
h7p://www.smashingmagazine.com/2010/06/24/the-‐ul"mate-‐guide-‐to-‐a-‐b-‐tes"ng/
What is A/B Testing?
Slide 4
Slide 4 text
What to test?
• Bu7ons:
colours,
sizes,
loca"ons
• Headlines
• Product
descrip"ons
• Form
fields
(adding
and
removing)
• PPC
landing
pages
• Email
content
• Pricing
and
promo"ons
• Images
Slide 5
Slide 5 text
What is A/B Testing good for?
• Data-‐driven
design
• Ge\ng
stakeholders
on-‐board
• Avoiding
personal
preferences
Slide 6
Slide 6 text
A/B testing should not replace:
• Talking
to
users
• Usability
tests
• Acceptance
tests
Slide 7
Slide 7 text
Do’s and Don’ts
• Test
simultaneously
• Test
o^en
• Include
only
new
visitors
• Show
repeat
visitors
the
same
version
• Use
results
to
inform
future
development
• Require
sta"s"cal
significance
DO:
Slide 8
Slide 8 text
• Test
too
many
ideas
at
once
• No
clear
idea
• Don’t
let
your
preconcep"ons
override
results
• Peek
at
your
results
and
stop
as
soon
as
a
significant
result
has
been
achieved.
See:
Evan
Miller.
Do’s and Don’ts
DON’T:
Slide 9
Slide 9 text
The tools
Other
tools
are
available:
A/Bingo,
Vanity,
Unbounce,
Performable,
Vertster,
SiteSpect,
Webtrends
Op"mize
and
Omniture’s
Test&Target
Slide 10
Slide 10 text
The Quiz Show
Slide 11
Slide 11 text
The Corkscrew
A
Slide 12
Slide 12 text
The Corkscrew
B
Slide 13
Slide 13 text
The Corkscrew
A/B
Slide 14
Slide 14 text
A
Slide 15
Slide 15 text
Great American Days
A
Slide 16
Slide 16 text
Great American Days
B
Slide 17
Slide 17 text
Great American Days
A/B
Slide 18
Slide 18 text
B
Slide 19
Slide 19 text
Real Gap
Slide 20
Slide 20 text
Real Gap
A/B
Slide 21
Slide 21 text
B
Slide 22
Slide 22 text
CityCliq
A
Slide 23
Slide 23 text
CityCliq
B
Slide 24
Slide 24 text
CityCliq
A/B
Slide 25
Slide 25 text
A
Slide 26
Slide 26 text
Vendio
A
Slide 27
Slide 27 text
Vendio
B
Slide 28
Slide 28 text
Vendio
B
Slide 29
Slide 29 text
A
Slide 30
Slide 30 text
Quick Fire Round
• What
were
the
two
main
tools
men"oned?
• What
percentage
increase
did
the
winning
RealGap
test
achieve?
• Name
one
of
the
three
things
A/B
tes"ng
doesn’t
replace.
Slide 31
Slide 31 text
What we looked at
• What
is
A/B
Tes"ng?
• What
to
test?
• What
is
A/B
Tes"ng
good
for?
• A/B
tes"ng
should
not
replace
• Do’s
and
Don’ts
• Tools