Winner or Loser? Analysing your test results with Causal Impact on R Studio

Presentation by Giulia Panozzo at Measurefest, a BrightonSEO Fringe Event, in October 2022.

'In this session you will learn how to use Causal Impact Analysis on R Studio, a powerful way to analyse test results and infer the impact of a change on a group of pages. It can be used in any areas where changes in strategy need to be justified by test results first, and it’s an invaluable tool to help your decision-making and clearly show stakeholders the impact of your team’s work.'

Transcript

1. #brightonseo Winner or Loser? Analysing Your Test Results with Causal Impact on R Studio Slideshare.Net/GiuliaPanozzo1 @SequinsNSearch Giulia Panozzo
2. Giulia Panozzo - @SequinsNSearch #brightonseo Pre- implementation data Post- implementation data
3. Giulia Panozzo - @SequinsNSearch #brightonseo
4. Giulia Panozzo - @SequinsNSearch #brightonseo Why do we care about Causal Impact?
5. Giulia Panozzo - @SequinsNSearch #brightonseo Because Testing is Hard!
6. Giulia Panozzo - @SequinsNSearch #brightonseo
7. Giulia Panozzo - @SequinsNSearch #brightonseo
8. Giulia Panozzo - @SequinsNSearch #brightonseo
9. Giulia Panozzo - @SequinsNSearch #brightonseo Example ‘Let’s add a price point to the title tag!’
10. Giulia Panozzo - @SequinsNSearch #brightonseo Example ‘Let’s add a price point to the title tag!’
11. Giulia Panozzo - @SequinsNSearch #brightonseo ‘But will this help bring more traffic in?’
12. Giulia Panozzo - @SequinsNSearch #brightonseo ‘Well…’
13. Giulia Panozzo - @SequinsNSearch #brightonseo
14. Giulia Panozzo - @SequinsNSearch #brightonseo However, with Causal Impact… X
15. Giulia Panozzo - @SequinsNSearch #brightonseo Causal Impact gives you the confidence to leverage statistically significant results and drive changes at scale
16. Giulia Panozzo - @SequinsNSearch #brightonseo ‘But will this help bring more traffic in?’
17. Giulia Panozzo - @SequinsNSearch #brightonseo YES (Most likely) NO (Most likely)
18. Giulia Panozzo - @SequinsNSearch #brightonseo You can use Causal Impact on a number of domains, not only on SEO tests!
19. Giulia Panozzo - @SequinsNSearch #brightonseo Impact of feature changes on app installs
20. Giulia Panozzo - @SequinsNSearch #brightonseo Impact of influencer campaigns
21. Giulia Panozzo - @SequinsNSearch #brightonseo Impact of offline events and campaigns
22. Giulia Panozzo - @SequinsNSearch #brightonseo And almost any time series data
23. Giulia Panozzo - @SequinsNSearch #brightonseo If you run Causal Impact on R Studio, it’s free and open-source
24. Giulia Panozzo - @SequinsNSearch #brightonseo
25. Giulia Panozzo - @SequinsNSearch #brightonseo What is Causal Impact And how it can help your strategy
26. Giulia Panozzo - @SequinsNSearch #brightonseo What is Causal Impact?
27. Giulia Panozzo - @SequinsNSearch #brightonseo Powerful package to analyse data and infer the cumulative impact of a change in a time series
28. Giulia Panozzo - @SequinsNSearch #brightonseo Based on BSTS (Bayesian Structural Time Series) statistical model
29. Giulia Panozzo - @SequinsNSearch #brightonseo Uses past data to predict the outcome in the absence of the treatment (the counterfactual)
30. Giulia Panozzo - @SequinsNSearch #brightonseo Defines the impact by measuring the deviation of the actual VS predicted outcome
31. Giulia Panozzo - @SequinsNSearch #brightonseo What is Causal Impact?
32. Giulia Panozzo - @SequinsNSearch #brightonseo How can it help us in marketing?
33. Giulia Panozzo - @SequinsNSearch #brightonseo How can it help us in marketing? Example of a clear winner from a title tag change Clicks: +58% CTR: +38% Position: -15%
34. Giulia Panozzo - @SequinsNSearch #brightonseo It can validate proposed strategy changes in case of any doubts
35. Giulia Panozzo - @SequinsNSearch #brightonseo It’s great to clearly show stakeholders the impact of our team’s work
36. Giulia Panozzo - @SequinsNSearch #brightonseo It can help forecast the direction of changes at scale and help make a case for more resources
37. Giulia Panozzo - @SequinsNSearch #brightonseo How can it help us in marketing? Example of a clear loser from a title tag change
38. Giulia Panozzo - @SequinsNSearch #brightonseo By clearly identifying a winner or loser, we can understand what works and doesn’t work for our audience
39. Giulia Panozzo - @SequinsNSearch #brightonseo How to run your analysis with Causal Impact on R Studio
40. Giulia Panozzo - @SequinsNSearch #brightonseo 1. Download R Studio Download R first https://cran.r-project.org/ Download RStudio https://www.rstudio.com/products/rstudio/download/
41. Giulia Panozzo - @SequinsNSearch #brightonseo 1. Download R Studio
42. Giulia Panozzo - @SequinsNSearch #brightonseo 1.1 Install Causal Impact
43. Giulia Panozzo - @SequinsNSearch #brightonseo 2. Prepare the data
44. Giulia Panozzo - @SequinsNSearch #brightonseo 2. Prepare the data
45. Giulia Panozzo - @SequinsNSearch #brightonseo The first column is always your test group. Other columns can be used as control groups if they are a good fit
46. Giulia Panozzo - @SequinsNSearch #brightonseo The pre-period should be at least twice as long as the post-period, to allow the model to plot the actual VS predicted outcome
47. Giulia Panozzo - @SequinsNSearch #brightonseo Any column with 0 should be either removed or corrected
48. Giulia Panozzo - @SequinsNSearch #brightonseo Isolated 0 in data set Multiple 0s VS
49. Giulia Panozzo - @SequinsNSearch #brightonseo 3. Run the script!
50. Giulia Panozzo - @SequinsNSearch #brightonseo Choose file to import
51. Giulia Panozzo - @SequinsNSearch #brightonseo Check preview
52. Giulia Panozzo - @SequinsNSearch #brightonseo Set pre and post periods
53. Giulia Panozzo - @SequinsNSearch #brightonseo It’s a winner!
54. Giulia Panozzo - @SequinsNSearch #brightonseo
55. Giulia Panozzo - @SequinsNSearch #brightonseo
56. Giulia Panozzo - @SequinsNSearch #brightonseo Now give it a go! Request access to this script here
57. Giulia Panozzo - @SequinsNSearch #brightonseo What I’ve learned from (several) trials and errors…
58. Giulia Panozzo - @SequinsNSearch #brightonseo The date column should always be removed when using this script
59. Giulia Panozzo - @SequinsNSearch #brightonseo Column titles can error out if they contain special characters, spaces, capitalised letters
60. Giulia Panozzo - @SequinsNSearch #brightonseo Start small, then expand your datasets with additional controls and features once you’re comfortable with the script
61. Giulia Panozzo - @SequinsNSearch #brightonseo Statistics VS Everything Else Limitations & Takeaways
62. Giulia Panozzo - @SequinsNSearch #brightonseo 1. External events can impact your data
63. Giulia Panozzo - @SequinsNSearch #brightonseo Google algo updates
64. Giulia Panozzo - @SequinsNSearch #brightonseo Tools tracking failures
65. Giulia Panozzo - @SequinsNSearch #brightonseo Engineering releases
66. Giulia Panozzo - @SequinsNSearch #brightonseo Create a document to map internal changes & external events This will allow you to take into account any other known factors and isolate the treatment in the analysis
67. Giulia Panozzo - @SequinsNSearch #brightonseo 2. Mind the Outliers!
68. Giulia Panozzo - @SequinsNSearch #brightonseo
69. Giulia Panozzo - @SequinsNSearch #brightonseo Outliers can originate from…
70. Giulia Panozzo - @SequinsNSearch #brightonseo New product launches (within the test group)
71. Giulia Panozzo - @SequinsNSearch #brightonseo Holidays and seasonal events
72. Giulia Panozzo - @SequinsNSearch #brightonseo Results only from one page Page 1 Page 2 Page 3 Page 4 Page 5
73. Giulia Panozzo - @SequinsNSearch #brightonseo Tracking bugs
74. Giulia Panozzo - @SequinsNSearch #brightonseo You can spot outliers and rule them out by…
75. Giulia Panozzo - @SequinsNSearch #brightonseo Increasing the size of the test group
76. Giulia Panozzo - @SequinsNSearch #brightonseo Adding control groups
77. Giulia Panozzo - @SequinsNSearch #brightonseo Pre-processing your raw data
78. Giulia Panozzo - @SequinsNSearch #brightonseo
79. Giulia Panozzo - @SequinsNSearch #brightonseo
80. Giulia Panozzo - @SequinsNSearch #brightonseo
81. Giulia Panozzo - @SequinsNSearch #brightonseo 3. Beware of confirmation bias
82. Giulia Panozzo - @SequinsNSearch #brightonseo Sometimes, your test will be inconclusive, or might be a loser even when you thought it’d be an easy winner
83. Giulia Panozzo - @SequinsNSearch #brightonseo
84. Giulia Panozzo - @SequinsNSearch #brightonseo In that case, you can run the test a little longer, or repeat the test with bigger groups
85. Giulia Panozzo - @SequinsNSearch #brightonseo If it’s still inconclusive or a loser, it’s probably best to revert the change and focus on other tests
86. Giulia Panozzo - @SequinsNSearch #brightonseo References and useful resources • How we use causal impact analysis to validate campaign success - Part and Sum • Measuring No-ID Campaigns with Causal Impact - Remerge & Alicia Horsch • Causal Impact – Data Skeptic • R Studio on GitHub • The Comprehensive R Archive Network • Causal Impact for App Store Analysis - William Martin
87. #brightonseo Thank You for Listening! Slideshare.Net/GiuliaPanozzo1 @SequinsNSearch Giulia Panozzo