Tweet
Tweet

Slide 1

Slide 1 text

Causal Impact Analysis for SEO: From theory to practice Giulia Panozzo

Slide 2

Slide 2 text

#SMXMunich Giulia Panozzo | @SequinsNsearch What is Causal Impact?

Slide 3

Slide 3 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Powerful package to analyse data and infer the cumulative impact of a change in a time series

Slide 4

Slide 4 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Based on BSTS (Bayesian Structural Time Series) statistical model

Slide 5

Slide 5 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Uses past data to predict the outcome in the absence of the treatment (the counterfactual)

Slide 6

Slide 6 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Defines the impact by measuring the deviation of the actual VS predicted outcome

Slide 7

Slide 7 text

#SMXMunich Giulia Panozzo | @SequinsNsearch

Slide 8

Slide 8 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Example: Title change ‘Let’s add a price point to the title!’

Slide 9

Slide 9 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Example: Title change ‘Let’s add a price point to the title!’

Slide 10

Slide 10 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Change implemented

Slide 11

Slide 11 text

#SMXMunich Giulia Panozzo | @SequinsNsearch

Slide 12

Slide 12 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Pre- implementation data Post- implementation data

Slide 13

Slide 13 text

#SMXMunich Giulia Panozzo | @SequinsNsearch

Slide 14

Slide 14 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Actual Prediction Deviation

Slide 15

Slide 15 text

#SMXMunich Giulia Panozzo | @SequinsNsearch

Slide 16

Slide 16 text

#SMXMunich Giulia Panozzo | @SequinsNsearch

Slide 17

Slide 17 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Because Testing is Hard!

Slide 18

Slide 18 text

#SMXMunich Giulia Panozzo | @SequinsNsearch

Slide 19

Slide 19 text

#SMXMunich Giulia Panozzo | @SequinsNsearch

Slide 20

Slide 20 text

#SMXMunich Giulia Panozzo | @SequinsNsearch

Slide 21

Slide 21 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Example ‘Let’s add a price point to the title tag!’

Slide 22

Slide 22 text

#SMXMunich Giulia Panozzo | @SequinsNsearch ‘But will this help bring more traffic in?’

Slide 23

Slide 23 text

#SMXMunich Giulia Panozzo | @SequinsNsearch ‘Well…’

Slide 24

Slide 24 text

#SMXMunich Giulia Panozzo | @SequinsNsearch

Slide 25

Slide 25 text

#SMXMunich Giulia Panozzo | @SequinsNsearch However, with Causal Impact… X

Slide 26

Slide 26 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Causal Impact gives you the confidence to leverage statistically significant results and drive changes at scale

Slide 27

Slide 27 text

#SMXMunich Giulia Panozzo | @SequinsNsearch ‘But will this help bring more traffic in?’

Slide 28

Slide 28 text

#SMXMunich Giulia Panozzo | @SequinsNsearch YES (Most likely) NO (Most likely)

Slide 29

Slide 29 text

#SMXMunich Giulia Panozzo | @SequinsNsearch You can use Causal Impact on a number of domains, not only on SEO tests!

Slide 30

Slide 30 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Impact of feature changes on app installs

Slide 31

Slide 31 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Impact of influencer campaigns

Slide 32

Slide 32 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Impact of offline events and campaigns

Slide 33

Slide 33 text

#SMXMunich Giulia Panozzo | @SequinsNsearch

Slide 34

Slide 34 text

#SMXMunich Giulia Panozzo | @SequinsNsearch May 25th: Trailer release July 9th: Barbie premiere

Slide 35

Slide 35 text

#SMXMunich Giulia Panozzo | @SequinsNsearch And almost any time series data

Slide 36

Slide 36 text

#SMXMunich Giulia Panozzo | @SequinsNsearch If you run Causal Impact on R Studio, it’s free and open-source

Slide 37

Slide 37 text

#SMXMunich Giulia Panozzo | @SequinsNsearch

Slide 38

Slide 38 text

#SMXMunich Giulia Panozzo | @SequinsNsearch How can it help us in marketing?

Slide 39

Slide 39 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Example of a clear winner from a title tag change Clicks: +58% CTR: +38% Position: -15%

Slide 40

Slide 40 text

#SMXMunich Giulia Panozzo | @SequinsNsearch It can validate proposed strategy changes in case of any doubts

Slide 41

Slide 41 text

#SMXMunich Giulia Panozzo | @SequinsNsearch

Slide 42

Slide 42 text

#SMXMunich Giulia Panozzo | @SequinsNsearch It’s great to clearly show stakeholders the impact of our team’s work

Slide 43

Slide 43 text

#SMXMunich Giulia Panozzo | @SequinsNsearch It can help forecast the direction of changes at scale and help make a case for more resources

Slide 44

Slide 44 text

#SMXMunich Giulia Panozzo | @SequinsNsearch How can it help us in marketing? Example of a clear loser from a title tag change

Slide 45

Slide 45 text

#SMXMunich Giulia Panozzo | @SequinsNsearch By clearly identifying a winner or loser, we can understand what works and doesn’t work for our audience

Slide 46

Slide 46 text

Demo time: How to run a Causal Impact Analysis

Slide 47

Slide 47 text

#SMXMunich Giulia Panozzo | @SequinsNsearch 1. Download R Studio Download R first https://cran.r-project.org/ Download RStudio https://www.rstudio.com/products/rstudio/download/

Slide 48

Slide 48 text

#SMXMunich Giulia Panozzo | @SequinsNsearch 1. Download R Studio

Slide 49

Slide 49 text

#SMXMunich Giulia Panozzo | @SequinsNsearch 1.1 Install Causal Impact

Slide 50

Slide 50 text

#SMXMunich Giulia Panozzo | @SequinsNsearch 2. Prepare the data

Slide 51

Slide 51 text

#SMXMunich Giulia Panozzo | @SequinsNsearch 2. Prepare the data

Slide 52

Slide 52 text

#SMXMunich Giulia Panozzo | @SequinsNsearch The first column is always your test group. Other columns can be used as control groups if they are a good fit

Slide 53

Slide 53 text

#SMXMunich Giulia Panozzo | @SequinsNsearch The pre-period should be at least twice as long as the post-period, to allow the model to plot the actual VS predicted outcome

Slide 54

Slide 54 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Isolated 0 in data set Multiple 0s VS

Slide 55

Slide 55 text

#SMXMunich Giulia Panozzo | @SequinsNsearch 3. Run the script!

Slide 56

Slide 56 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Choose file to import

Slide 57

Slide 57 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Check preview

Slide 58

Slide 58 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Set pre and post periods

Slide 59

Slide 59 text

#SMXMunich Giulia Panozzo | @SequinsNsearch It’s a winner!

Slide 60

Slide 60 text

#SMXMunich Giulia Panozzo | @SequinsNsearch

Slide 61

Slide 61 text

#SMXMunich Giulia Panozzo | @SequinsNsearch

Slide 62

Slide 62 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Now give it a go! Access this script here

Slide 63

Slide 63 text

#SMXMunich Giulia Panozzo | @SequinsNsearch What I’ve learned from (several) trials and errors…

Slide 64

Slide 64 text

#SMXMunich Giulia Panozzo | @SequinsNsearch The date column should always be removed when using this script

Slide 65

Slide 65 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Column titles can error out if they contain special characters, spaces, capitalised letters

Slide 66

Slide 66 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Start small, then expand your datasets with additional controls and features once you’re comfortable with the script

Slide 67

Slide 67 text

Statistics VS The World

Slide 68

Slide 68 text

#SMXMunich Giulia Panozzo | @SequinsNsearch

Slide 69

Slide 69 text

#SMXMunich Giulia Panozzo | @SequinsNsearch

Slide 70

Slide 70 text

#SMXMunich Giulia Panozzo | @SequinsNsearch 1. External events can impact your data

Slide 71

Slide 71 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Google algo updates

Slide 72

Slide 72 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Tools tracking failures

Slide 73

Slide 73 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Engineering releases

Slide 74

Slide 74 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Create a document to map internal changes & external events Algo updates Other releases / Bugs 4th July, Black Friday, etc.

Slide 75

Slide 75 text

#SMXMunich Giulia Panozzo | @SequinsNsearch 2. Mind the Outliers!

Slide 76

Slide 76 text

#SMXMunich Giulia Panozzo | @SequinsNsearch

Slide 77

Slide 77 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Outliers can originate from…

Slide 78

Slide 78 text

#SMXMunich Giulia Panozzo | @SequinsNsearch New product launches (within the test group)

Slide 79

Slide 79 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Holidays and seasonal events

Slide 80

Slide 80 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Results only from one page Page 1 Page 2 Page 3 Page 4 Page 5

Slide 81

Slide 81 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Tracking bugs

Slide 82

Slide 82 text

#SMXMunich Giulia Panozzo | @SequinsNsearch You can spot outliers and rule them out by…

Slide 83

Slide 83 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Increasing the size of the test group

Slide 84

Slide 84 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Adding control groups

Slide 85

Slide 85 text

#SMXMunich Giulia Panozzo | @SequinsNsearch Pre-processing your raw data

Slide 86

Slide 86 text

#SMXMunich Giulia Panozzo | @SequinsNsearch

Slide 87

Slide 87 text

#SMXMunich Giulia Panozzo | @SequinsNsearch

Slide 88

Slide 88 text

#SMXMunich Giulia Panozzo | @SequinsNsearch

Slide 89

Slide 89 text

#SMXMunich Giulia Panozzo | @SequinsNsearch 3. Beware of confirmation bias

Slide 90

Slide 90 text

#SMXMunich Giulia Panozzo | @SequinsNsearch

Slide 91

Slide 91 text

#SMXMunich Giulia Panozzo | @SequinsNsearch

Slide 92

Slide 92 text

#SMXMunich Giulia Panozzo | @SequinsNsearch In that case, you can run the test a little longer, or repeat the test with bigger groups

Slide 93

Slide 93 text

#SMXMunich Giulia Panozzo | @SequinsNsearch If it’s still inconclusive or a loser, it’s probably best to revert the change and focus on other tests

Slide 94

Slide 94 text

#SMXMunich Giulia Panozzo | @SequinsNsearch References and useful resources • How to Measure the Impact of your SEO Changes with Causal Impact – Giulia Panozzo (step by step guide) • How we use causal impact analysis to validate campaign success - Part and Sum • Measuring No-ID Campaigns with Causal Impact - Remerge & Alicia Horsch • Causal Impact – Data Skeptic • R Studio on GitHub • The Comprehensive R Archive Network • Causal Impact for App Store Analysis - William Martin

Slide 95

Slide 95 text

Thank You! speakerdeck.com/giuliapanozzo @SequinsNSearch Questions? Find me here: Neuroscientive.com