Slide 1

Slide 1 text

#brightonseo Winner or Loser? Analysing Your Test Results with Causal Impact on R Studio Slideshare.Net/GiuliaPanozzo1 @SequinsNSearch Giulia Panozzo

Slide 2

Slide 2 text

Giulia Panozzo - @SequinsNSearch #brightonseo Pre- implementation data Post- implementation data

Slide 3

Slide 3 text

Giulia Panozzo - @SequinsNSearch #brightonseo

Slide 4

Slide 4 text

Giulia Panozzo - @SequinsNSearch #brightonseo Why do we care about Causal Impact?

Slide 5

Slide 5 text

Giulia Panozzo - @SequinsNSearch #brightonseo Because Testing is Hard!

Slide 6

Slide 6 text

Giulia Panozzo - @SequinsNSearch #brightonseo

Slide 7

Slide 7 text

Giulia Panozzo - @SequinsNSearch #brightonseo

Slide 8

Slide 8 text

Giulia Panozzo - @SequinsNSearch #brightonseo

Slide 9

Slide 9 text

Giulia Panozzo - @SequinsNSearch #brightonseo Example ‘Let’s add a price point to the title tag!’

Slide 10

Slide 10 text

Giulia Panozzo - @SequinsNSearch #brightonseo Example ‘Let’s add a price point to the title tag!’

Slide 11

Slide 11 text

Giulia Panozzo - @SequinsNSearch #brightonseo ‘But will this help bring more traffic in?’

Slide 12

Slide 12 text

Giulia Panozzo - @SequinsNSearch #brightonseo ‘Well…’

Slide 13

Slide 13 text

Giulia Panozzo - @SequinsNSearch #brightonseo

Slide 14

Slide 14 text

Giulia Panozzo - @SequinsNSearch #brightonseo However, with Causal Impact… X

Slide 15

Slide 15 text

Giulia Panozzo - @SequinsNSearch #brightonseo Causal Impact gives you the confidence to leverage statistically significant results and drive changes at scale

Slide 16

Slide 16 text

Giulia Panozzo - @SequinsNSearch #brightonseo ‘But will this help bring more traffic in?’

Slide 17

Slide 17 text

Giulia Panozzo - @SequinsNSearch #brightonseo YES (Most likely) NO (Most likely)

Slide 18

Slide 18 text

Giulia Panozzo - @SequinsNSearch #brightonseo You can use Causal Impact on a number of domains, not only on SEO tests!

Slide 19

Slide 19 text

Giulia Panozzo - @SequinsNSearch #brightonseo Impact of feature changes on app installs

Slide 20

Slide 20 text

Giulia Panozzo - @SequinsNSearch #brightonseo Impact of influencer campaigns

Slide 21

Slide 21 text

Giulia Panozzo - @SequinsNSearch #brightonseo Impact of offline events and campaigns

Slide 22

Slide 22 text

Giulia Panozzo - @SequinsNSearch #brightonseo And almost any time series data

Slide 23

Slide 23 text

Giulia Panozzo - @SequinsNSearch #brightonseo If you run Causal Impact on R Studio, it’s free and open-source

Slide 24

Slide 24 text

Giulia Panozzo - @SequinsNSearch #brightonseo

Slide 25

Slide 25 text

Giulia Panozzo - @SequinsNSearch #brightonseo What is Causal Impact And how it can help your strategy

Slide 26

Slide 26 text

Giulia Panozzo - @SequinsNSearch #brightonseo What is Causal Impact?

Slide 27

Slide 27 text

Giulia Panozzo - @SequinsNSearch #brightonseo Powerful package to analyse data and infer the cumulative impact of a change in a time series

Slide 28

Slide 28 text

Giulia Panozzo - @SequinsNSearch #brightonseo Based on BSTS (Bayesian Structural Time Series) statistical model

Slide 29

Slide 29 text

Giulia Panozzo - @SequinsNSearch #brightonseo Uses past data to predict the outcome in the absence of the treatment (the counterfactual)

Slide 30

Slide 30 text

Giulia Panozzo - @SequinsNSearch #brightonseo Defines the impact by measuring the deviation of the actual VS predicted outcome

Slide 31

Slide 31 text

Giulia Panozzo - @SequinsNSearch #brightonseo What is Causal Impact?

Slide 32

Slide 32 text

Giulia Panozzo - @SequinsNSearch #brightonseo How can it help us in marketing?

Slide 33

Slide 33 text

Giulia Panozzo - @SequinsNSearch #brightonseo How can it help us in marketing? Example of a clear winner from a title tag change Clicks: +58% CTR: +38% Position: -15%

Slide 34

Slide 34 text

Giulia Panozzo - @SequinsNSearch #brightonseo It can validate proposed strategy changes in case of any doubts

Slide 35

Slide 35 text

Giulia Panozzo - @SequinsNSearch #brightonseo It’s great to clearly show stakeholders the impact of our team’s work

Slide 36

Slide 36 text

Giulia Panozzo - @SequinsNSearch #brightonseo It can help forecast the direction of changes at scale and help make a case for more resources

Slide 37

Slide 37 text

Giulia Panozzo - @SequinsNSearch #brightonseo How can it help us in marketing? Example of a clear loser from a title tag change

Slide 38

Slide 38 text

Giulia Panozzo - @SequinsNSearch #brightonseo By clearly identifying a winner or loser, we can understand what works and doesn’t work for our audience

Slide 39

Slide 39 text

Giulia Panozzo - @SequinsNSearch #brightonseo How to run your analysis with Causal Impact on R Studio

Slide 40

Slide 40 text

Giulia Panozzo - @SequinsNSearch #brightonseo 1. Download R Studio Download R first https://cran.r-project.org/ Download RStudio https://www.rstudio.com/products/rstudio/download/

Slide 41

Slide 41 text

Giulia Panozzo - @SequinsNSearch #brightonseo 1. Download R Studio

Slide 42

Slide 42 text

Giulia Panozzo - @SequinsNSearch #brightonseo 1.1 Install Causal Impact

Slide 43

Slide 43 text

Giulia Panozzo - @SequinsNSearch #brightonseo 2. Prepare the data

Slide 44

Slide 44 text

Giulia Panozzo - @SequinsNSearch #brightonseo 2. Prepare the data

Slide 45

Slide 45 text

Giulia Panozzo - @SequinsNSearch #brightonseo The first column is always your test group. Other columns can be used as control groups if they are a good fit

Slide 46

Slide 46 text

Giulia Panozzo - @SequinsNSearch #brightonseo The pre-period should be at least twice as long as the post-period, to allow the model to plot the actual VS predicted outcome

Slide 47

Slide 47 text

Giulia Panozzo - @SequinsNSearch #brightonseo Any column with 0 should be either removed or corrected

Slide 48

Slide 48 text

Giulia Panozzo - @SequinsNSearch #brightonseo Isolated 0 in data set Multiple 0s VS

Slide 49

Slide 49 text

Giulia Panozzo - @SequinsNSearch #brightonseo 3. Run the script!

Slide 50

Slide 50 text

Giulia Panozzo - @SequinsNSearch #brightonseo Choose file to import

Slide 51

Slide 51 text

Giulia Panozzo - @SequinsNSearch #brightonseo Check preview

Slide 52

Slide 52 text

Giulia Panozzo - @SequinsNSearch #brightonseo Set pre and post periods

Slide 53

Slide 53 text

Giulia Panozzo - @SequinsNSearch #brightonseo It’s a winner!

Slide 54

Slide 54 text

Giulia Panozzo - @SequinsNSearch #brightonseo

Slide 55

Slide 55 text

Giulia Panozzo - @SequinsNSearch #brightonseo

Slide 56

Slide 56 text

Giulia Panozzo - @SequinsNSearch #brightonseo Now give it a go! Request access to this script here

Slide 57

Slide 57 text

Giulia Panozzo - @SequinsNSearch #brightonseo What I’ve learned from (several) trials and errors…

Slide 58

Slide 58 text

Giulia Panozzo - @SequinsNSearch #brightonseo The date column should always be removed when using this script

Slide 59

Slide 59 text

Giulia Panozzo - @SequinsNSearch #brightonseo Column titles can error out if they contain special characters, spaces, capitalised letters

Slide 60

Slide 60 text

Giulia Panozzo - @SequinsNSearch #brightonseo Start small, then expand your datasets with additional controls and features once you’re comfortable with the script

Slide 61

Slide 61 text

Giulia Panozzo - @SequinsNSearch #brightonseo Statistics VS Everything Else Limitations & Takeaways

Slide 62

Slide 62 text

Giulia Panozzo - @SequinsNSearch #brightonseo 1. External events can impact your data

Slide 63

Slide 63 text

Giulia Panozzo - @SequinsNSearch #brightonseo Google algo updates

Slide 64

Slide 64 text

Giulia Panozzo - @SequinsNSearch #brightonseo Tools tracking failures

Slide 65

Slide 65 text

Giulia Panozzo - @SequinsNSearch #brightonseo Engineering releases

Slide 66

Slide 66 text

Giulia Panozzo - @SequinsNSearch #brightonseo Create a document to map internal changes & external events This will allow you to take into account any other known factors and isolate the treatment in the analysis

Slide 67

Slide 67 text

Giulia Panozzo - @SequinsNSearch #brightonseo 2. Mind the Outliers!

Slide 68

Slide 68 text

Giulia Panozzo - @SequinsNSearch #brightonseo

Slide 69

Slide 69 text

Giulia Panozzo - @SequinsNSearch #brightonseo Outliers can originate from…

Slide 70

Slide 70 text

Giulia Panozzo - @SequinsNSearch #brightonseo New product launches (within the test group)

Slide 71

Slide 71 text

Giulia Panozzo - @SequinsNSearch #brightonseo Holidays and seasonal events

Slide 72

Slide 72 text

Giulia Panozzo - @SequinsNSearch #brightonseo Results only from one page Page 1 Page 2 Page 3 Page 4 Page 5

Slide 73

Slide 73 text

Giulia Panozzo - @SequinsNSearch #brightonseo Tracking bugs

Slide 74

Slide 74 text

Giulia Panozzo - @SequinsNSearch #brightonseo You can spot outliers and rule them out by…

Slide 75

Slide 75 text

Giulia Panozzo - @SequinsNSearch #brightonseo Increasing the size of the test group

Slide 76

Slide 76 text

Giulia Panozzo - @SequinsNSearch #brightonseo Adding control groups

Slide 77

Slide 77 text

Giulia Panozzo - @SequinsNSearch #brightonseo Pre-processing your raw data

Slide 78

Slide 78 text

Giulia Panozzo - @SequinsNSearch #brightonseo

Slide 79

Slide 79 text

Giulia Panozzo - @SequinsNSearch #brightonseo

Slide 80

Slide 80 text

Giulia Panozzo - @SequinsNSearch #brightonseo

Slide 81

Slide 81 text

Giulia Panozzo - @SequinsNSearch #brightonseo 3. Beware of confirmation bias

Slide 82

Slide 82 text

Giulia Panozzo - @SequinsNSearch #brightonseo Sometimes, your test will be inconclusive, or might be a loser even when you thought it’d be an easy winner

Slide 83

Slide 83 text

Giulia Panozzo - @SequinsNSearch #brightonseo

Slide 84

Slide 84 text

Giulia Panozzo - @SequinsNSearch #brightonseo In that case, you can run the test a little longer, or repeat the test with bigger groups

Slide 85

Slide 85 text

Giulia Panozzo - @SequinsNSearch #brightonseo If it’s still inconclusive or a loser, it’s probably best to revert the change and focus on other tests

Slide 86

Slide 86 text

Giulia Panozzo - @SequinsNSearch #brightonseo References and useful resources • How we use causal impact analysis to validate campaign success - Part and Sum • Measuring No-ID Campaigns with Causal Impact - Remerge & Alicia Horsch • Causal Impact – Data Skeptic • R Studio on GitHub • The Comprehensive R Archive Network • Causal Impact for App Store Analysis - William Martin

Slide 87

Slide 87 text

#brightonseo Thank You for Listening! Slideshare.Net/GiuliaPanozzo1 @SequinsNSearch Giulia Panozzo