Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Winner or Loser? Analysing your test results with Causal Impact on R Studio

Winner or Loser? Analysing your test results with Causal Impact on R Studio

Presentation by Giulia Panozzo at Measurefest, a BrightonSEO Fringe Event, in October 2022.

'In this session you will learn how to use Causal Impact Analysis on R Studio, a powerful way to analyse test results and infer the impact of a change on a group of pages. It can be used in any areas where changes in strategy need to be justified by test results first, and it’s an invaluable tool to help your decision-making and clearly show stakeholders the impact of your team’s work.'

Giulia Panozzo

October 19, 2022
Tweet

More Decks by Giulia Panozzo

Other Decks in Marketing & SEO

Transcript

  1. #brightonseo
    Winner or Loser?
    Analysing Your Test
    Results with Causal
    Impact on R Studio
    Slideshare.Net/GiuliaPanozzo1
    @SequinsNSearch
    Giulia Panozzo

    View full-size slide

  2. Giulia Panozzo - @SequinsNSearch #brightonseo
    Pre-
    implementation
    data
    Post-
    implementation
    data

    View full-size slide

  3. Giulia Panozzo - @SequinsNSearch #brightonseo

    View full-size slide

  4. Giulia Panozzo - @SequinsNSearch #brightonseo
    Why do we care
    about Causal Impact?

    View full-size slide

  5. Giulia Panozzo - @SequinsNSearch #brightonseo
    Because Testing is Hard!

    View full-size slide

  6. Giulia Panozzo - @SequinsNSearch #brightonseo

    View full-size slide

  7. Giulia Panozzo - @SequinsNSearch #brightonseo

    View full-size slide

  8. Giulia Panozzo - @SequinsNSearch #brightonseo

    View full-size slide

  9. Giulia Panozzo - @SequinsNSearch #brightonseo
    Example
    ‘Let’s add a price point to the title tag!’

    View full-size slide

  10. Giulia Panozzo - @SequinsNSearch #brightonseo
    Example
    ‘Let’s add a price point to the title tag!’

    View full-size slide

  11. Giulia Panozzo - @SequinsNSearch #brightonseo
    ‘But will this help
    bring more traffic
    in?’

    View full-size slide

  12. Giulia Panozzo - @SequinsNSearch #brightonseo
    ‘Well…’

    View full-size slide

  13. Giulia Panozzo - @SequinsNSearch #brightonseo

    View full-size slide

  14. Giulia Panozzo - @SequinsNSearch #brightonseo
    However, with Causal Impact…
    X

    View full-size slide

  15. Giulia Panozzo - @SequinsNSearch #brightonseo
    Causal Impact gives you the
    confidence to leverage
    statistically significant results and
    drive changes at scale

    View full-size slide

  16. Giulia Panozzo - @SequinsNSearch #brightonseo
    ‘But will this help
    bring more traffic
    in?’

    View full-size slide

  17. Giulia Panozzo - @SequinsNSearch #brightonseo
    YES
    (Most likely)
    NO
    (Most likely)

    View full-size slide

  18. Giulia Panozzo - @SequinsNSearch #brightonseo
    You can use Causal Impact on a
    number of domains, not only on
    SEO tests!

    View full-size slide

  19. Giulia Panozzo - @SequinsNSearch #brightonseo
    Impact of feature changes on
    app installs

    View full-size slide

  20. Giulia Panozzo - @SequinsNSearch #brightonseo
    Impact of influencer campaigns

    View full-size slide

  21. Giulia Panozzo - @SequinsNSearch #brightonseo
    Impact of offline events and
    campaigns

    View full-size slide

  22. Giulia Panozzo - @SequinsNSearch #brightonseo
    And almost any time series data

    View full-size slide

  23. Giulia Panozzo - @SequinsNSearch #brightonseo
    If you run Causal Impact on R
    Studio,
    it’s free and open-source

    View full-size slide

  24. Giulia Panozzo - @SequinsNSearch #brightonseo

    View full-size slide

  25. Giulia Panozzo - @SequinsNSearch #brightonseo
    What is Causal Impact
    And how it can help your
    strategy

    View full-size slide

  26. Giulia Panozzo - @SequinsNSearch #brightonseo
    What is Causal Impact?

    View full-size slide

  27. Giulia Panozzo - @SequinsNSearch #brightonseo
    Powerful package to analyse
    data and infer the
    cumulative impact of a change
    in a time series

    View full-size slide

  28. Giulia Panozzo - @SequinsNSearch #brightonseo
    Based on BSTS
    (Bayesian Structural Time
    Series)
    statistical model

    View full-size slide

  29. Giulia Panozzo - @SequinsNSearch #brightonseo
    Uses past data to predict the
    outcome in the absence of the
    treatment
    (the counterfactual)

    View full-size slide

  30. Giulia Panozzo - @SequinsNSearch #brightonseo
    Defines the impact by
    measuring the deviation of the
    actual VS predicted outcome

    View full-size slide

  31. Giulia Panozzo - @SequinsNSearch #brightonseo
    What is Causal Impact?

    View full-size slide

  32. Giulia Panozzo - @SequinsNSearch #brightonseo
    How can it help us in marketing?

    View full-size slide

  33. Giulia Panozzo - @SequinsNSearch #brightonseo
    How can it help us in marketing?
    Example of a clear winner from a title tag change
    Clicks: +58% CTR: +38% Position: -15%

    View full-size slide

  34. Giulia Panozzo - @SequinsNSearch #brightonseo
    It can validate proposed strategy
    changes in case of any doubts

    View full-size slide

  35. Giulia Panozzo - @SequinsNSearch #brightonseo
    It’s great to clearly show
    stakeholders
    the impact of our team’s work

    View full-size slide

  36. Giulia Panozzo - @SequinsNSearch #brightonseo
    It can help forecast the
    direction of changes at scale and
    help
    make a case for more resources

    View full-size slide

  37. Giulia Panozzo - @SequinsNSearch #brightonseo
    How can it help us in marketing?
    Example of a clear loser from a title tag change

    View full-size slide

  38. Giulia Panozzo - @SequinsNSearch #brightonseo
    By clearly identifying a winner or
    loser, we can understand
    what works and doesn’t work
    for our audience

    View full-size slide

  39. Giulia Panozzo - @SequinsNSearch #brightonseo
    How to run your analysis
    with Causal Impact on
    R Studio

    View full-size slide

  40. Giulia Panozzo - @SequinsNSearch #brightonseo
    1. Download R Studio
    Download R first
    https://cran.r-project.org/
    Download RStudio
    https://www.rstudio.com/products/rstudio/download/

    View full-size slide

  41. Giulia Panozzo - @SequinsNSearch #brightonseo
    1. Download R Studio

    View full-size slide

  42. Giulia Panozzo - @SequinsNSearch #brightonseo
    1.1 Install Causal Impact

    View full-size slide

  43. Giulia Panozzo - @SequinsNSearch #brightonseo
    2. Prepare the data

    View full-size slide

  44. Giulia Panozzo - @SequinsNSearch #brightonseo
    2. Prepare the data

    View full-size slide

  45. Giulia Panozzo - @SequinsNSearch #brightonseo
    The first column is always
    your test group.
    Other columns can be used as
    control groups if they are a good
    fit

    View full-size slide

  46. Giulia Panozzo - @SequinsNSearch #brightonseo
    The pre-period should be at least
    twice as long as the post-period,
    to allow the model to plot the
    actual VS predicted outcome

    View full-size slide

  47. Giulia Panozzo - @SequinsNSearch #brightonseo
    Any column with 0 should be
    either removed or corrected

    View full-size slide

  48. Giulia Panozzo - @SequinsNSearch #brightonseo
    Isolated 0 in data set
    Multiple 0s
    VS

    View full-size slide

  49. Giulia Panozzo - @SequinsNSearch #brightonseo
    3. Run the script!

    View full-size slide

  50. Giulia Panozzo - @SequinsNSearch #brightonseo
    Choose file to import

    View full-size slide

  51. Giulia Panozzo - @SequinsNSearch #brightonseo
    Check preview

    View full-size slide

  52. Giulia Panozzo - @SequinsNSearch #brightonseo
    Set pre and post
    periods

    View full-size slide

  53. Giulia Panozzo - @SequinsNSearch #brightonseo
    It’s a winner!

    View full-size slide

  54. Giulia Panozzo - @SequinsNSearch #brightonseo

    View full-size slide

  55. Giulia Panozzo - @SequinsNSearch #brightonseo

    View full-size slide

  56. Giulia Panozzo - @SequinsNSearch #brightonseo
    Now give it a go!
    Request access to this
    script here

    View full-size slide

  57. Giulia Panozzo - @SequinsNSearch #brightonseo
    What I’ve learned from (several)
    trials and errors…

    View full-size slide

  58. Giulia Panozzo - @SequinsNSearch #brightonseo
    The date column should always
    be removed
    when using this script

    View full-size slide

  59. Giulia Panozzo - @SequinsNSearch #brightonseo
    Column titles can error out if
    they contain special characters,
    spaces, capitalised letters

    View full-size slide

  60. Giulia Panozzo - @SequinsNSearch #brightonseo
    Start small, then expand your
    datasets with additional controls
    and features once you’re
    comfortable with the script

    View full-size slide

  61. Giulia Panozzo - @SequinsNSearch #brightonseo
    Statistics VS Everything
    Else
    Limitations & Takeaways

    View full-size slide

  62. Giulia Panozzo - @SequinsNSearch #brightonseo
    1. External events can impact
    your data

    View full-size slide

  63. Giulia Panozzo - @SequinsNSearch #brightonseo
    Google algo
    updates

    View full-size slide

  64. Giulia Panozzo - @SequinsNSearch #brightonseo
    Tools
    tracking
    failures

    View full-size slide

  65. Giulia Panozzo - @SequinsNSearch #brightonseo
    Engineering
    releases

    View full-size slide

  66. Giulia Panozzo - @SequinsNSearch #brightonseo
    Create a document to map
    internal changes & external events
    This will allow you to take into account any
    other known factors and isolate the treatment
    in the analysis

    View full-size slide

  67. Giulia Panozzo - @SequinsNSearch #brightonseo
    2. Mind the Outliers!

    View full-size slide

  68. Giulia Panozzo - @SequinsNSearch #brightonseo

    View full-size slide

  69. Giulia Panozzo - @SequinsNSearch #brightonseo
    Outliers can
    originate from…

    View full-size slide

  70. Giulia Panozzo - @SequinsNSearch #brightonseo
    New product launches
    (within the test group)

    View full-size slide

  71. Giulia Panozzo - @SequinsNSearch #brightonseo
    Holidays and seasonal events

    View full-size slide

  72. Giulia Panozzo - @SequinsNSearch #brightonseo
    Results only from one page
    Page 1
    Page 2
    Page 3
    Page 4
    Page 5

    View full-size slide

  73. Giulia Panozzo - @SequinsNSearch #brightonseo
    Tracking bugs

    View full-size slide

  74. Giulia Panozzo - @SequinsNSearch #brightonseo
    You can spot outliers and rule them out by…

    View full-size slide

  75. Giulia Panozzo - @SequinsNSearch #brightonseo
    Increasing the size of the test group

    View full-size slide

  76. Giulia Panozzo - @SequinsNSearch #brightonseo
    Adding control groups

    View full-size slide

  77. Giulia Panozzo - @SequinsNSearch #brightonseo
    Pre-processing your raw data

    View full-size slide

  78. Giulia Panozzo - @SequinsNSearch #brightonseo

    View full-size slide

  79. Giulia Panozzo - @SequinsNSearch #brightonseo

    View full-size slide

  80. Giulia Panozzo - @SequinsNSearch #brightonseo

    View full-size slide

  81. Giulia Panozzo - @SequinsNSearch #brightonseo
    3. Beware of confirmation bias

    View full-size slide

  82. Giulia Panozzo - @SequinsNSearch #brightonseo
    Sometimes, your test will be inconclusive, or might
    be a loser even when you thought it’d be an easy
    winner

    View full-size slide

  83. Giulia Panozzo - @SequinsNSearch #brightonseo

    View full-size slide

  84. Giulia Panozzo - @SequinsNSearch #brightonseo
    In that case, you can run the test a little longer,
    or repeat the test with bigger groups

    View full-size slide

  85. Giulia Panozzo - @SequinsNSearch #brightonseo
    If it’s still inconclusive or a loser, it’s probably best
    to revert the change and focus on other tests

    View full-size slide

  86. Giulia Panozzo - @SequinsNSearch #brightonseo
    References and useful resources
    • How we use causal impact analysis to validate campaign success - Part and Sum
    • Measuring No-ID Campaigns with Causal Impact - Remerge & Alicia Horsch
    • Causal Impact – Data Skeptic
    • R Studio on GitHub
    • The Comprehensive R Archive Network
    • Causal Impact for App Store Analysis - William Martin

    View full-size slide

  87. #brightonseo
    Thank You for
    Listening!
    Slideshare.Net/GiuliaPanozzo1
    @SequinsNSearch
    Giulia Panozzo

    View full-size slide