@SequinsNsearch | #Measurefest
The Ultimate Guide
to Test & Learns
Giulia Panozzo
NEUROSCIENTIVE
speakerdeck.com/giuliapanozzo
@SequinsNsearch
Slide 2
Slide 2 text
@SequinsNsearch | #Measurefest
Hi Brighton! I’m Giulia
👩💻 In-house Director of Customer
Acquisition
🧠 Freelance Consultant
✨ Figure skater, aerialist and fire
breather
@SequinsNsearch | #Measurefest
Slide 3
Slide 3 text
@SequinsNsearch | #Measurefest
I’ve also done a lot of
test and learns
@SequinsNsearch | #Measurefest
Slide 4
Slide 4 text
@SequinsNsearch | #Measurefest
So I usually get some
questions
@SequinsNsearch #Measurefest
Slide 5
Slide 5 text
@SequinsNsearch | #Measurefest
‘How do I select
test & control
groups?’
Slide 6
Slide 6 text
@SequinsNsearch | #Measurefest
‘What metrics
should I track?’
@SequinsNsearch | #Measurefest
Slide 7
Slide 7 text
@SequinsNsearch | #Measurefest
‘What timeframe
& test design
should I use?’
Slide 8
Slide 8 text
@SequinsNsearch | #Measurefest
‘How should I
analyse & report
data?’
Slide 9
Slide 9 text
@SequinsNsearch | #Measurefest
Let’s dive into it
Slide 10
Slide 10 text
@SequinsNsearch | #Measurefest
Hypothesis
Slide 11
Slide 11 text
@SequinsNsearch | #Measurefest
@SequinsNsearch | #Measurefest
What questions are you
trying to answer?
Slide 12
Slide 12 text
@SequinsNsearch | #Measurefest
“Updating my titles to
include a recency signal will
significantly improve CTRs”
Slide 13
Slide 13 text
@SequinsNsearch | #Measurefest
Slide 14
Slide 14 text
@SequinsNsearch | #Measurefest
Slide 15
Slide 15 text
@SequinsNsearch | #Measurefest
“Updating my titles to
include a recency signal will
significantly improve CTRs”
Independent variable
Dependent variable
Slide 16
Slide 16 text
@SequinsNsearch | #Measurefest
Some other example questions
that might drive your
testing strategy
Slide 17
Slide 17 text
@SequinsNsearch | #Measurefest
‘How can we improve our content strategy to
increase traffic?'
VS
Slide 18
Slide 18 text
@SequinsNsearch | #Measurefest
‘How do we leverage internal linking to improve
rankings?'
Slide 19
Slide 19 text
@SequinsNsearch | #Measurefest
‘What on-page elements do we need to focus
on to improve CTRs?'
Slide 20
Slide 20 text
@SequinsNsearch | #Measurefest
‘What kind of schema mark-up ensures us
the most SERP real estate?'
Slide 21
Slide 21 text
@SequinsNsearch | #Measurefest
SEOClarity Split Testing Ideas | SearchPilot Newsletter
(free)
SEO Case Study Database by Adam Gent (paid)
Need inspiration?
💡
Slide 22
Slide 22 text
@SequinsNsearch | #Measurefest
Test Design
Slide 23
Slide 23 text
@SequinsNsearch | #Measurefest
What are the variables I
am testing?
To decide, ask yourself
Slide 24
Slide 24 text
@SequinsNsearch | #Measurefest
What kind of changes
do I intend to make?
To decide, ask yourself
Slide 25
Slide 25 text
@SequinsNsearch | #Measurefest
What tools do I have
to implement my changes?
To decide, ask yourself
@SequinsNsearch | #Measurefest
📈 Some kinds of on page SEO
(if running at the edge)
A/B & Multivariate Testing
Slide 38
Slide 38 text
@SequinsNsearch | #Measurefest
Some Technical Directions for SEO
Source: Google Search Central Testing Guidelines
🥸 No cloaking
Slide 39
Slide 39 text
@SequinsNsearch | #Measurefest
🔀 302 redirects VS 301
(or JS-based redirects)
Source: Google Search Central Testing Guidelines
Some Technical Directions for SEO
Slide 40
Slide 40 text
@SequinsNsearch | #Measurefest
Source: Google Search Central Testing Guidelines
🏆 Use rel= "canonical" attributes
Some Technical Directions for SEO
Slide 41
Slide 41 text
@SequinsNsearch | #Measurefest
H1
Hero Image
Default
Content
H1
Hero Image
FAQ Content
July August
Pre-post
Slide 42
Slide 42 text
@SequinsNsearch | #Measurefest
Pre-post
Control and Variant are
tested in
2 subsequent periods
Slide 43
Slide 43 text
@SequinsNsearch | #Measurefest
🔂 Single on-page changes
Pre-post
Better suited for
Slide 44
Slide 44 text
@SequinsNsearch | #Measurefest
⁉ Groups that cannot be
matched for other conditions
Pre-post
Slide 45
Slide 45 text
@SequinsNsearch | #Measurefest
🚀 Off page campaigns
Pre-post
Slide 46
Slide 46 text
@SequinsNsearch | #Measurefest
Test & Control Groups
Slide 47
Slide 47 text
@SequinsNsearch | #Measurefest
How do I select my test group?
🚥 Pages with enough traffic
Slide 48
Slide 48 text
@SequinsNsearch | #Measurefest
How do I select my test group?
🚥 Pages with enough traffic
🤪 Not your outliers
Slide 49
Slide 49 text
@SequinsNsearch | #Measurefest
How do I select my test group?
🚥 Pages with enough traffic
🤪 Not your outliers
👯 Pages of the same type (e.g. PLPs)
Slide 50
Slide 50 text
@SequinsNsearch | #Measurefest
All to help reach
statistical significance
Slide 51
Slide 51 text
@SequinsNsearch | #Measurefest
How do I select my control group(s)?
Tldr: keep everything
as close as possible
to the test group
Slide 52
Slide 52 text
@SequinsNsearch | #Measurefest
A few options
(non mutually exclusive)
Slide 53
Slide 53 text
@SequinsNsearch | #Measurefest
Same page types
as the test group
with similar 6-12
months traffic
Ideal for small scale
websites & tests
Slide 54
Slide 54 text
@SequinsNsearch | #Measurefest
Same pages as
test group, but in a
different,
untreated market
Ideal for
international
websites & tests
Slide 55
Slide 55 text
@SequinsNsearch | #Measurefest
Overall website trend
across same dates
Ideal for ecom and
large scale
websites & tests
Slide 56
Slide 56 text
@SequinsNsearch | #Measurefest
Timeframe
Slide 57
Slide 57 text
@SequinsNsearch | #Measurefest
Slide 58
Slide 58 text
@SequinsNsearch | #Measurefest
For SEO tests:
2-4 weeks
(with
intermediate
check-ins)
How long should I let my test run?
Slide 59
Slide 59 text
@SequinsNsearch | #Measurefest
When are my ‘pre’ and ‘post’ periods?
Slide 60
Slide 60 text
@SequinsNsearch | #Measurefest
Launch date
For SEO tests, check when changes have been
crawled rather than implemented
Slide 61
Slide 61 text
@SequinsNsearch | #Measurefest
Analysis
Slide 62
Slide 62 text
@SequinsNsearch | #Measurefest
This is tricky because it
depends on the test design
and the resources you have
Slide 63
Slide 63 text
@SequinsNsearch | #Measurefest
Flying solo? 🧗
Good old delta formulas
1⃣
(Post – Pre)/Pre
(Post/Pre) -1
%
OR
@SequinsNsearch | #Measurefest
Then invest in tools that can make this
easier and faster
Got budget? 💸
And more…
Slide 66
Slide 66 text
@SequinsNsearch | #Measurefest
Report
Slide 67
Slide 67 text
@SequinsNsearch | #Measurefest
Slide 68
Slide 68 text
@SequinsNsearch | #Measurefest
Provide a top-level summary
Hypothesis Groups & Treatment
Launch & Check-In
Dates
Results &
Classification
Slide 69
Slide 69 text
@SequinsNsearch | #Measurefest
Provide a top-level summary
Hypothesis Groups & Treatment
Launch & Check-In
Dates
Results &
Classification
Next Steps
Slide 70
Slide 70 text
@SequinsNsearch | #Measurefest
Tell a story with the help of data visualisation
Launch
Slide 71
Slide 71 text
@SequinsNsearch | #Measurefest
Build live dashboards when you can, in order to
easily spot trends
Slide 72
Slide 72 text
@SequinsNsearch | #Measurefest
Make a note of anything that could have
impacted your test in the same timeframe
Slide 73
Slide 73 text
@SequinsNsearch | #Measurefest
Algo updates
Other releases /
Bugs
4th July, Black
Friday, etc.
Make a note of anything that could have
impacted your test in the same timeframe
Slide 74
Slide 74 text
@SequinsNsearch | #Measurefest
Summing it up
Slide 75
Slide 75 text
@SequinsNsearch | #Measurefest
Hypothesis
Test Design
Slide 76
Slide 76 text
@SequinsNsearch | #Measurefest
Hypothesis
Test Design
Test & Control Groups
Slide 77
Slide 77 text
@SequinsNsearch | #Measurefest
Hypothesis
Test Design
Test & Control Groups
Analysis
Slide 78
Slide 78 text
@SequinsNsearch | #Measurefest
Hypothesis
Test Design
Test & Control Groups
Reporting
Analysis
Slide 79
Slide 79 text
@SequinsNsearch | #Measurefest
Ok, we tested…
but what about the learnings?
Slide 80
Slide 80 text
@SequinsNsearch | #Measurefest
Create your own repository
Slide 81
Slide 81 text
@SequinsNsearch | #Measurefest
More Learnings
More Strategic Testing
Better Customer Insights
Slide 82
Slide 82 text
@SequinsNsearch | #Measurefest
Get my template here