A/B
Test
to
Improve
the
User
Experience
by
Lara
Swanson
(Lead
Front
End
Dev
at
Dyn)
Slide 2
Slide 2 text
Background
• Dyn
=
DNS
and
Email
Delivery
– E-‐commerce
site
– ApplicaHon
management
– User
seIngs
– MigraHon
tools
Slide 3
Slide 3 text
What
is
an
A/B
test?
• Analyze
user
interacHon
with
an
element:
– Content
–
Design
– Layout
–
AcHon
• Create
two
versions
of
the
element
to
test
• “Winning”
version
=
happy
users!
Picture
by
Kevin
Cornell
via
A
List
Apart
Slide 4
Slide 4 text
App
landing
page
with
installaHon
screenshots
App
landing
page
with
installaHon
video
A
B
Slide 5
Slide 5 text
Don’t
miss
out
on
becoming
a
VIP
user.
Sign
up
now!
Don’t
be
an
idiot;
become
a
VIP!
A
B
Slide 6
Slide 6 text
“Buy”
buZon
adds
the
product
at
$20
for
1
year
“Buy”
buZon
adds
the
product
at
$35
for
2
years
A
B
Slide 7
Slide 7 text
Running
an
A/B
test
1. Split
users
into
two
segments
2. Randomly
show
users
one
of
the
two
versions
3. Track
what
users
do
on
each
version
Slide 8
Slide 8 text
Conversion
Tracking
Determine
the
“conversion”
to
track:
• Next
user
acHon
(next
screen,
next
click)
• Final
user
goal
(checkout,
sign
up)
Slide 9
Slide 9 text
App
landing
page
with
installaHon
screenshots
App
landing
page
with
installaHon
video
A
B
}measure
%
new
user
completed
installaHons
Slide 10
Slide 10 text
Don’t
miss
out
on
becoming
a
VIP
user.
Sign
up
now!
Don’t
be
an
idiot;
become
a
VIP!
A
B
}measure
%
of
users
who
signed
up
Slide 11
Slide 11 text
“Buy”
buZon
adds
the
product
at
$20
for
1
year
“Buy”
buZon
adds
the
product
at
$35
for
2
years
A
B
}measure
average
order
value
per
completed
checkout
Slide 12
Slide 12 text
Conversion
Tracking
• StaHsHcal
significance
– 95%
confidence
level
to
determine
“winner”
• You’ll
need
enough
users
and
conversions
have
significant
results
My
favorite
calculator:
usereffect.com/split-‐test-‐calculator
Slide 13
Slide 13 text
AddiHonal
Data
Examine
data
for
addiHonal
user
acHons
beyond
the
primary
conversion
rate
Slide 14
Slide 14 text
AddiHonal
Data
Segment
your
user
data
by
various
demographics
Slide 15
Slide 15 text
MulHvariate
Tests
Create
mulHple
versions
and
test
them
all
at
the
same
Hme
(use
adcomparator.com’s
Taguchi-‐Based
calculator)
(source)
Slide 16
Slide 16 text
Best
PracHces
• Don’t
let
tests
go
on
forever
• Test
“screams”
rather
than
“whispers”
• Use
tests
to
solve
UX
design
disagreements
Photo
by
Poppy
Thomas-‐Hill
(source)