The Ultimate Guide to Test and Learns

Presentation by Giulia Panozzo at Measurefest, a BrightonSEO Fringe event, in September 2023.

'Testing in the digital landscape is not like testing in a lab environment where all or most conditions are controlled for, so it can be tricky to execute tests and draw clear conclusions that we can apply to the entire website. In this session, you will learn how to select test and control groups for acquisition testing, how to analyse and infer statistical significance from your results and how to present your conclusions to the wider team to create a business case for meaningful changes. This has been Giulia's go-to strategy in her past few roles and has been instrumental in setting the roadmap for most of the websites she's worked on.'

See the template provided at the end here: bit.ly/46e6p09

Transcript

1. @SequinsNsearch | #Measurefest
   The Ultimate Guide
   to Test & Learns
   Giulia Panozzo
   NEUROSCIENTIVE
   speakerdeck.com/giuliapanozzo
   @SequinsNsearch
   

   View Slide

2. @SequinsNsearch | #Measurefest
   Hi Brighton! I’m Giulia
   👩💻 In-house Director of Customer
   Acquisition
   🧠 Freelance Consultant
   ✨ Figure skater, aerialist and fire
   breather
   @SequinsNsearch | #Measurefest
   

   View Slide

3. @SequinsNsearch | #Measurefest
   I’ve also done a lot of
   test and learns
   @SequinsNsearch | #Measurefest
   

   View Slide

4. @SequinsNsearch | #Measurefest
   So I usually get some
   questions
   @SequinsNsearch #Measurefest
   

   View Slide

5. @SequinsNsearch | #Measurefest
   ‘How do I select
   test & control
   groups?’
   

   View Slide

6. @SequinsNsearch | #Measurefest
   ‘What metrics
   should I track?’
   @SequinsNsearch | #Measurefest
   

   View Slide

7. @SequinsNsearch | #Measurefest
   ‘What timeframe
   & test design
   should I use?’
   

   View Slide

8. @SequinsNsearch | #Measurefest
   ‘How should I
   analyse & report
   data?’
   

   View Slide

9. @SequinsNsearch | #Measurefest
   Let’s dive into it
   

   View Slide

10. @SequinsNsearch | #Measurefest
    Hypothesis
    

    View Slide

11. @SequinsNsearch | #Measurefest
    @SequinsNsearch | #Measurefest
    What questions are you
    trying to answer?
    

    View Slide

12. @SequinsNsearch | #Measurefest
    “Updating my titles to
    include a recency signal will
    significantly improve CTRs”
    

    View Slide

13. @SequinsNsearch | #Measurefest
    

    View Slide

14. @SequinsNsearch | #Measurefest
    

    View Slide

15. @SequinsNsearch | #Measurefest
    “Updating my titles to
    include a recency signal will
    significantly improve CTRs”
    Independent variable
    Dependent variable
    

    View Slide

16. @SequinsNsearch | #Measurefest
    Some other example questions
    that might drive your
    testing strategy
    

    View Slide

17. @SequinsNsearch | #Measurefest
    ‘How can we improve our content strategy to
    increase traffic?'
    VS
    

    View Slide

18. @SequinsNsearch | #Measurefest
    ‘How do we leverage internal linking to improve
    rankings?'
    

    View Slide

19. @SequinsNsearch | #Measurefest
    ‘What on-page elements do we need to focus
    on to improve CTRs?'
    

    View Slide

20. @SequinsNsearch | #Measurefest
    ‘What kind of schema mark-up ensures us
    the most SERP real estate?'
    

    View Slide

21. @SequinsNsearch | #Measurefest
    SEOClarity Split Testing Ideas | SearchPilot Newsletter
    (free)
    SEO Case Study Database by Adam Gent (paid)
    Need inspiration?
    💡
    

    View Slide

22. @SequinsNsearch | #Measurefest
    Test Design
    

    View Slide

23. @SequinsNsearch | #Measurefest
    What are the variables I
    am testing?
    To decide, ask yourself
    

    View Slide

24. @SequinsNsearch | #Measurefest
    What kind of changes
    do I intend to make?
    To decide, ask yourself
    

    View Slide

25. @SequinsNsearch | #Measurefest
    What tools do I have
    to implement my changes?
    To decide, ask yourself
    

    View Slide

26. @SequinsNsearch | #Measurefest
    🧪 A/B Testing
    

    View Slide

27. @SequinsNsearch | #Measurefest
    🧪 A/B Testing
    🧑🔬 Multivariate
    

    View Slide

28. @SequinsNsearch | #Measurefest
    🧪 A/B Testing
    🧑🔬 Multivariate
    ⌛ Pre-post
    

    View Slide

29. @SequinsNsearch | #Measurefest
    A/B Testing
    H1
    Hero Image
    Content
    Content
    Hero Image
    H1
    A B
    

    View Slide

30. @SequinsNsearch | #Measurefest
    A/B Testing
    Both variants (A & B) are tested
    during the same timeframe
    

    View Slide

31. @SequinsNsearch | #Measurefest
    Multivariate
    H1
    Hero Image
    Content
    A
    H1
    Hero Image
    FAQ
    Content
    H1
    Hero Image
    FAQ
    Content
    B C
    

    View Slide

32. @SequinsNsearch | #Measurefest
    A/B & Multivariate Testing
    Type of test Testing
    Timeframe
    Traffic required
    A/B Same Low/Medium
    Multivariate Same High
    

    View Slide

33. @SequinsNsearch | #Measurefest
    Better suited for
    📧 CRM
    A/B & Multivariate Testing
    

    View Slide

34. @SequinsNsearch | #Measurefest
    ✏ UX & CRO
    A/B & Multivariate Testing
    

    View Slide

35. @SequinsNsearch | #Measurefest
    💸 Marketing landing pages
    A/B & Multivariate Testing
    

    View Slide

36. @SequinsNsearch | #Measurefest
    💰 Paid campaigns
    A/B & Multivariate Testing
    

    View Slide

37. @SequinsNsearch | #Measurefest
    📈 Some kinds of on page SEO
    (if running at the edge)
    A/B & Multivariate Testing
    

    View Slide

38. @SequinsNsearch | #Measurefest
    Some Technical Directions for SEO
    Source: Google Search Central Testing Guidelines
    🥸 No cloaking
    

    View Slide

39. @SequinsNsearch | #Measurefest
    🔀 302 redirects VS 301
    (or JS-based redirects)
    Source: Google Search Central Testing Guidelines
    Some Technical Directions for SEO
    

    View Slide

40. @SequinsNsearch | #Measurefest
    Source: Google Search Central Testing Guidelines
    🏆 Use rel= "canonical" attributes
    Some Technical Directions for SEO
    

    View Slide

41. @SequinsNsearch | #Measurefest
    H1
    Hero Image
    Default
    Content
    H1
    Hero Image
    FAQ Content
    July August
    Pre-post
    

    View Slide

42. @SequinsNsearch | #Measurefest
    Pre-post
    Control and Variant are
    tested in
    2 subsequent periods
    

    View Slide

43. @SequinsNsearch | #Measurefest
    🔂 Single on-page changes
    Pre-post
    Better suited for
    

    View Slide

44. @SequinsNsearch | #Measurefest
    ⁉ Groups that cannot be
    matched for other conditions
    Pre-post
    

    View Slide

45. @SequinsNsearch | #Measurefest
    🚀 Off page campaigns
    Pre-post
    

    View Slide

46. @SequinsNsearch | #Measurefest
    Test & Control Groups
    

    View Slide

47. @SequinsNsearch | #Measurefest
    How do I select my test group?
    🚥 Pages with enough traffic
    

    View Slide

48. @SequinsNsearch | #Measurefest
    How do I select my test group?
    🚥 Pages with enough traffic
    🤪 Not your outliers
    

    View Slide

49. @SequinsNsearch | #Measurefest
    How do I select my test group?
    🚥 Pages with enough traffic
    🤪 Not your outliers
    👯 Pages of the same type (e.g. PLPs)
    

    View Slide

50. @SequinsNsearch | #Measurefest
    All to help reach
    statistical significance
    

    View Slide

51. @SequinsNsearch | #Measurefest
    How do I select my control group(s)?
    Tldr: keep everything
    as close as possible
    to the test group
    

    View Slide

52. @SequinsNsearch | #Measurefest
    A few options
    (non mutually exclusive)
    

    View Slide

53. @SequinsNsearch | #Measurefest
    Same page types
    as the test group
    with similar 6-12
    months traffic
    Ideal for small scale
    websites & tests
    

    View Slide

54. @SequinsNsearch | #Measurefest
    Same pages as
    test group, but in a
    different,
    untreated market
    Ideal for
    international
    websites & tests
    

    View Slide

55. @SequinsNsearch | #Measurefest
    Overall website trend
    across same dates
    Ideal for ecom and
    large scale
    websites & tests
    

    View Slide

56. @SequinsNsearch | #Measurefest
    Timeframe
    

    View Slide

57. @SequinsNsearch | #Measurefest
    

    View Slide

58. @SequinsNsearch | #Measurefest
    For SEO tests:
    2-4 weeks
    (with
    intermediate
    check-ins)
    How long should I let my test run?
    

    View Slide

59. @SequinsNsearch | #Measurefest
    When are my ‘pre’ and ‘post’ periods?
    

    View Slide

60. @SequinsNsearch | #Measurefest
    Launch date
    For SEO tests, check when changes have been
    crawled rather than implemented
    

    View Slide

61. @SequinsNsearch | #Measurefest
    Analysis
    

    View Slide

62. @SequinsNsearch | #Measurefest
    This is tricky because it
    depends on the test design
    and the resources you have
    

    View Slide

63. @SequinsNsearch | #Measurefest
    Flying solo? 🧗
    Good old delta formulas
    1⃣
    (Post – Pre)/Pre
    (Post/Pre) -1
    %
    OR
    

    View Slide

64. @SequinsNsearch | #Measurefest
    Causal Impact Analysis
    Flying solo? 🧗
    2⃣
    

    View Slide

65. @SequinsNsearch | #Measurefest
    Then invest in tools that can make this
    easier and faster
    Got budget? 💸
    And more…
    

    View Slide

66. @SequinsNsearch | #Measurefest
    Report
    

    View Slide

67. @SequinsNsearch | #Measurefest
    

    View Slide

68. @SequinsNsearch | #Measurefest
    Provide a top-level summary
    Hypothesis Groups & Treatment
    Launch & Check-In
    Dates
    Results &
    Classification
    

    View Slide

69. @SequinsNsearch | #Measurefest
    Provide a top-level summary
    Hypothesis Groups & Treatment
    Launch & Check-In
    Dates
    Results &
    Classification
    Next Steps
    

    View Slide

70. @SequinsNsearch | #Measurefest
    Tell a story with the help of data visualisation
    Launch
    

    View Slide

71. @SequinsNsearch | #Measurefest
    Build live dashboards when you can, in order to
    easily spot trends
    

    View Slide

72. @SequinsNsearch | #Measurefest
    Make a note of anything that could have
    impacted your test in the same timeframe
    

    View Slide

73. @SequinsNsearch | #Measurefest
    Algo updates
    Other releases /
    Bugs
    4th July, Black
    Friday, etc.
    Make a note of anything that could have
    impacted your test in the same timeframe
    

    View Slide

74. @SequinsNsearch | #Measurefest
    Summing it up
    

    View Slide

75. @SequinsNsearch | #Measurefest
    Hypothesis
    Test Design
    

    View Slide

76. @SequinsNsearch | #Measurefest
    Hypothesis
    Test Design
    Test & Control Groups
    

    View Slide

77. @SequinsNsearch | #Measurefest
    Hypothesis
    Test Design
    Test & Control Groups
    Analysis
    

    View Slide

78. @SequinsNsearch | #Measurefest
    Hypothesis
    Test Design
    Test & Control Groups
    Reporting
    Analysis
    

    View Slide

79. @SequinsNsearch | #Measurefest
    Ok, we tested…
    but what about the learnings?
    

    View Slide

80. @SequinsNsearch | #Measurefest
    Create your own repository
    

    View Slide

81. @SequinsNsearch | #Measurefest
    More Learnings
    More Strategic Testing
    Better Customer Insights
    

    View Slide

82. @SequinsNsearch | #Measurefest
    Get my template here
    

    View Slide

83. @SequinsNsearch | #Measurefest
    Thank you!
    @SequinsNsearch
    /giulia-panozzo/
    neuroscientive.com
    

    View Slide