Giulia Panozzo - @SequinsNSearch #brightonseo
However, with Causal Impact…
X
Slide 15
Slide 15 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Causal Impact gives you the
confidence to leverage
statistically significant results and
drive changes at scale
Slide 16
Slide 16 text
Giulia Panozzo - @SequinsNSearch #brightonseo
‘But will this help
bring more traffic
in?’
Giulia Panozzo - @SequinsNSearch #brightonseo
You can use Causal Impact on a
number of domains, not only on
SEO tests!
Slide 19
Slide 19 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Impact of feature changes on
app installs
Slide 20
Slide 20 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Impact of influencer campaigns
Slide 21
Slide 21 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Impact of offline events and
campaigns
Slide 22
Slide 22 text
Giulia Panozzo - @SequinsNSearch #brightonseo
And almost any time series data
Slide 23
Slide 23 text
Giulia Panozzo - @SequinsNSearch #brightonseo
If you run Causal Impact on R
Studio,
it’s free and open-source
Slide 24
Slide 24 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Slide 25
Slide 25 text
Giulia Panozzo - @SequinsNSearch #brightonseo
What is Causal Impact
And how it can help your
strategy
Slide 26
Slide 26 text
Giulia Panozzo - @SequinsNSearch #brightonseo
What is Causal Impact?
Slide 27
Slide 27 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Powerful package to analyse
data and infer the
cumulative impact of a change
in a time series
Slide 28
Slide 28 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Based on BSTS
(Bayesian Structural Time
Series)
statistical model
Slide 29
Slide 29 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Uses past data to predict the
outcome in the absence of the
treatment
(the counterfactual)
Slide 30
Slide 30 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Defines the impact by
measuring the deviation of the
actual VS predicted outcome
Slide 31
Slide 31 text
Giulia Panozzo - @SequinsNSearch #brightonseo
What is Causal Impact?
Slide 32
Slide 32 text
Giulia Panozzo - @SequinsNSearch #brightonseo
How can it help us in marketing?
Slide 33
Slide 33 text
Giulia Panozzo - @SequinsNSearch #brightonseo
How can it help us in marketing?
Example of a clear winner from a title tag change
Clicks: +58% CTR: +38% Position: -15%
Slide 34
Slide 34 text
Giulia Panozzo - @SequinsNSearch #brightonseo
It can validate proposed strategy
changes in case of any doubts
Slide 35
Slide 35 text
Giulia Panozzo - @SequinsNSearch #brightonseo
It’s great to clearly show
stakeholders
the impact of our team’s work
Slide 36
Slide 36 text
Giulia Panozzo - @SequinsNSearch #brightonseo
It can help forecast the
direction of changes at scale and
help
make a case for more resources
Slide 37
Slide 37 text
Giulia Panozzo - @SequinsNSearch #brightonseo
How can it help us in marketing?
Example of a clear loser from a title tag change
Slide 38
Slide 38 text
Giulia Panozzo - @SequinsNSearch #brightonseo
By clearly identifying a winner or
loser, we can understand
what works and doesn’t work
for our audience
Slide 39
Slide 39 text
Giulia Panozzo - @SequinsNSearch #brightonseo
How to run your analysis
with Causal Impact on
R Studio
Slide 40
Slide 40 text
Giulia Panozzo - @SequinsNSearch #brightonseo
1. Download R Studio
Download R first
https://cran.r-project.org/
Download RStudio
https://www.rstudio.com/products/rstudio/download/
Slide 41
Slide 41 text
Giulia Panozzo - @SequinsNSearch #brightonseo
1. Download R Studio
Giulia Panozzo - @SequinsNSearch #brightonseo
2. Prepare the data
Slide 44
Slide 44 text
Giulia Panozzo - @SequinsNSearch #brightonseo
2. Prepare the data
Slide 45
Slide 45 text
Giulia Panozzo - @SequinsNSearch #brightonseo
The first column is always
your test group.
Other columns can be used as
control groups if they are a good
fit
Slide 46
Slide 46 text
Giulia Panozzo - @SequinsNSearch #brightonseo
The pre-period should be at least
twice as long as the post-period,
to allow the model to plot the
actual VS predicted outcome
Slide 47
Slide 47 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Any column with 0 should be
either removed or corrected
Slide 48
Slide 48 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Isolated 0 in data set
Multiple 0s
VS
Slide 49
Slide 49 text
Giulia Panozzo - @SequinsNSearch #brightonseo
3. Run the script!
Slide 50
Slide 50 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Choose file to import
Giulia Panozzo - @SequinsNSearch #brightonseo
Set pre and post
periods
Slide 53
Slide 53 text
Giulia Panozzo - @SequinsNSearch #brightonseo
It’s a winner!
Slide 54
Slide 54 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Slide 55
Slide 55 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Slide 56
Slide 56 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Now give it a go!
Request access to this
script here
Slide 57
Slide 57 text
Giulia Panozzo - @SequinsNSearch #brightonseo
What I’ve learned from (several)
trials and errors…
Slide 58
Slide 58 text
Giulia Panozzo - @SequinsNSearch #brightonseo
The date column should always
be removed
when using this script
Slide 59
Slide 59 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Column titles can error out if
they contain special characters,
spaces, capitalised letters
Slide 60
Slide 60 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Start small, then expand your
datasets with additional controls
and features once you’re
comfortable with the script
Giulia Panozzo - @SequinsNSearch #brightonseo
Create a document to map
internal changes & external events
This will allow you to take into account any
other known factors and isolate the treatment
in the analysis
Slide 67
Slide 67 text
Giulia Panozzo - @SequinsNSearch #brightonseo
2. Mind the Outliers!
Slide 68
Slide 68 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Slide 69
Slide 69 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Outliers can
originate from…
Slide 70
Slide 70 text
Giulia Panozzo - @SequinsNSearch #brightonseo
New product launches
(within the test group)
Slide 71
Slide 71 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Holidays and seasonal events
Slide 72
Slide 72 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Results only from one page
Page 1
Page 2
Page 3
Page 4
Page 5
Giulia Panozzo - @SequinsNSearch #brightonseo
You can spot outliers and rule them out by…
Slide 75
Slide 75 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Increasing the size of the test group
Slide 76
Slide 76 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Adding control groups
Slide 77
Slide 77 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Pre-processing your raw data
Slide 78
Slide 78 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Slide 79
Slide 79 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Slide 80
Slide 80 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Slide 81
Slide 81 text
Giulia Panozzo - @SequinsNSearch #brightonseo
3. Beware of confirmation bias
Slide 82
Slide 82 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Sometimes, your test will be inconclusive, or might
be a loser even when you thought it’d be an easy
winner
Slide 83
Slide 83 text
Giulia Panozzo - @SequinsNSearch #brightonseo
Slide 84
Slide 84 text
Giulia Panozzo - @SequinsNSearch #brightonseo
In that case, you can run the test a little longer,
or repeat the test with bigger groups
Slide 85
Slide 85 text
Giulia Panozzo - @SequinsNSearch #brightonseo
If it’s still inconclusive or a loser, it’s probably best
to revert the change and focus on other tests
Slide 86
Slide 86 text
Giulia Panozzo - @SequinsNSearch #brightonseo
References and useful resources
• How we use causal impact analysis to validate campaign success - Part and Sum
• Measuring No-ID Campaigns with Causal Impact - Remerge & Alicia Horsch
• Causal Impact – Data Skeptic
• R Studio on GitHub
• The Comprehensive R Archive Network
• Causal Impact for App Store Analysis - William Martin
Slide 87
Slide 87 text
#brightonseo
Thank You for
Listening!
Slideshare.Net/GiuliaPanozzo1
@SequinsNSearch
Giulia Panozzo