CRO does not equal A/B Testing

Slide 1

Slide 1 text

CRO ≠ A/B Testing

Slide 2

Slide 2 text

Eidra Commercial Presentation 2 We are a consultancy collective helping leaders create great change. The world is becoming even more complex for business leaders due to economic uncertainty, emerging technologies, geopolitical risk, climate change, new consumer behaviors and talent demands. We believe a new partner is needed to rise to the demands of today and tomorrow. Eidra is a consultancy collective, built upon the foundational idea that great change requires the combination of management consulting with best in breed expertise in creativity, engineering and data. We believe we are equipped to perform the task with the agility and speed needed to turn business challenges into opportunity.

Slide 3

Slide 3 text

Eidra Commercial Presentation 3 We can help you create great change. Framing the role of your business and brand to set the direction for meaningful change. Rethink positioning & purpose Finding relevant pockets of growth and acquire the capabilities to address them. Identify new growth opportunities Addressing challenges and seizing opportunities in the sustainability transition. Scale sustainability Organizing for innovation to build, scale and ship more relevant products and services. Accelerate innovation Designing the interface between your business and customers for more affinity. Optimize customer experience Optimizing market stance and boosting productivity with strategic models, enhanced sales, and cost efficiency. Define business strategy Elevating your digital presence by integrating new tools for enhanced intelligence and efficiency. Advance with emerging tech Designing your brand to better articulate who you are and where you’re going. Build your brand Optimizing how you organize and operate for better return on brand & marketing investments. Increase commerce efficiency Developing your cross-funnel communication and content and driving and measuring the results. Craft your creative communication

Slide 4

Slide 4 text

© Eidra Our expertise. Eidra Commercial Presentation 4 Strategy & Change Brand & Marketing Innovation & Experience Technology & Product Data & Commerce Define, design and build unforgettable brands and experiences that resonate with your business’ desires and unlock growth. Make clients grow by defining when, where and how to evolve their business through business or product innovation. Drive transformation, growth and competitive differentiation through technology. Grow your business through data, AI and insights, optimizing your sales and marketing channels and operations. Future-proof your business through strategy and organizational change.

Slide 5

Slide 5 text

© Eidra How we deliver new, more and better business results. Eidra Commercial Presentation 5 Venturing into uncharted business territories by launching new products, services, and business models. Redefining value propositions through technology and capturing growth potential, finding the core of tomorrow. • Products & Services • Technologies & Innovations • Brands & Business models • Markets & Eco-systems • Ventures & Acquisitions New Growing core operations by maximizing current customer value, optimizing offerings, and enhancing experiences. Leveraging technology and creativity to boost engagement, growth, and competitiveness of current business. • Customers & Segments • Loyalty & CLV • Assortment & Pricing • Experiences & Features • Revenues & Gross profit More Creating better business value by enhancing efficiency, profitability, and sustainability of operations. Optimizing processes and integrating innovative solutions to drive impactful and lasting change. • Tech-stack & Platforms • Operations & Processes • Data & Insights • Sustainability & Footprint • Profitability & Valuation Better EXECUTION DIRECTION Innovation & Experience Technology & Product Strategy& Change Brand & Marketing Data & Commerce Orchestration &Collaboration

Slide 6

Slide 6 text

© Eidra Baboee Eidra provides R&D and AI strategies to transform work processes, ensuring seamless information access across Royal BAM Group projects. Eidra’s Data & Insights Lab established a data pipeline and and AI classification engine, analyzing a million support emails to deliver crucial business insights to guide leadership forward with precision. The Eidra team created a specialized AI solution that facilitates visual searches within a database of 100,000 paintings. The Eidra AI Bloodline architecture enables the subscription-based furnishing furnishing service NORNORM to get accurate AI driven forecasts, enabling sales team to identify when companies are are planning to move and are open to exploring interior solutions.

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

eidra.com CRO ≠ A/B Testing

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

© Eidra 2025-05-26 12 Great question! Conversion Rate Optimization (CRO) is often confused with A/B testing, but they’re not the same thing. CRO is a broad strategy focused on improving the rate at which website visitors complete a desired action, like making a purchase, signing up for a newsletter, or downloading an app.

Slide 13

Slide 13 text

© Eidra 2025-05-26 13 Great question! Conversion Rate Optimization (CRO) is often confused with A/B testing, but they’re not the same thing. CRO is a broad strategy focused on improving the rate at which website visitors complete a desired action, like making a purchase, signing up for a newsletter, or downloading an app.

Slide 14

Slide 14 text

© Eidra 2025-05-26 14 Great question! Conversion Rate Optimization (CRO) is often confused with A/B testing, but they’re not the same thing. CRO is a broad strategy focused on improving the rate at which website visitors complete a desired action, like making a purchase, signing up for a newsletter, or downloading an app.

Slide 15

Slide 15 text

© Eidra 2025-05-26 15 Great question! Conversion Rate Optimization (CRO) is often confused with A/B testing, but they’re not the same thing. CRO is a broad strategy focused on improving the rate at which website visitors complete a desired action, like making a purchase, signing up for a newsletter, or downloading an app.

Slide 16

Slide 16 text

© Eidra 2025-05-26 16 Great question! Conversion Rate Optimization (CRO) is often confused with A/B testing, but they’re not the same thing. CRO is a broad strategy focused on improving the rate at which website visitors complete a desired action, like making a purchase, signing up for a newsletter, or downloading an app. A/B testing is one method within the CRO toolbox, where two or more versions of a webpage or element are tested against each other to see which performs better. However, CRO goes beyond just testing.

Slide 17

Slide 17 text

© Eidra 2025-05-26 17 Great question! Conversion Rate Optimization (CRO) is often confused with A/B testing, but they’re not the same thing. CRO is a broad strategy focused on improving the rate at which website visitors complete a desired action, like making a purchase, signing up for a newsletter, or downloading an app. A/B testing is one method within the CRO toolbox, where two or more versions of a webpage or element are tested against each other to see which performs better. However, CRO goes beyond just testing.

Slide 18

Slide 18 text

© Eidra 2025-05-26 18 Great question! Conversion Rate Optimization (CRO) is often confused with A/B testing, but they’re not the same thing. CRO is a broad strategy focused on improving the rate at which website visitors complete a desired action, like making a purchase, signing up for a newsletter, or downloading an app. A/B testing is one method within the CRO toolbox, where two or more versions of a webpage or element are tested against each other to see which performs better. However, CRO goes beyond just testing.

Slide 19

Slide 19 text

© Eidra 2025-05-26 19 Great question! Conversion Rate Optimization (CRO) is often confused with A/B testing, but they’re not the same thing. CRO is a broad strategy focused on improving the rate at which website visitors complete a desired action, like making a purchase, signing up for a newsletter, or downloading an app. A/B testing is one method within the CRO toolbox, where two or more versions of a webpage or element are tested against each other to see which performs better. However, CRO goes beyond just testing.

Slide 20

Slide 20 text

© Eidra 2025-05-26 20 Great question! Conversion Rate Optimization (CRO) is often confused with A/B testing, but they’re not the same thing. CRO is a broad strategy focused on improving the rate at which website visitors complete a desired action, like making a purchase, signing up for a newsletter, or downloading an app. A/B testing is one method within the CRO toolbox, where two or more versions of a webpage or element are tested against each other to see which performs better. However, CRO goes beyond just testing.

Slide 21

Slide 21 text

eidra.com What is in that CRO Toolbox?

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

© Eidra 2025-05-26 24 Hierarchy of Evidence Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.

Slide 25

Slide 25 text

© Eidra 2025-05-26 25 Hierarchy of Evidence Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.

Slide 26

Slide 26 text

© Eidra 2025-05-26 26 Hierarchy of Evidence Randomized Controlled Trials Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.

Slide 27

Slide 27 text

© Eidra 2025-05-26 27 Hierarchy of Evidence Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trials Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.

Slide 28

Slide 28 text

© Eidra 2025-05-26 28 Hierarchy of Evidence Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trials Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website. Risk of Bias Quality of Evidence Lower Strong Weak Higher

Slide 29

Slide 29 text

© Eidra 2025-05-26 29 Hierarchy of Evidence Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trials Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website. Not evidence Anecdotes, gut feeling, forums, competitors, Linkedin Lunatics, A/B test results websites, Awards, etc. Risk of Bias Quality of Evidence Lower Strong Weak Higher

Slide 30

Slide 30 text

© Eidra 2025-05-26 30 The progress you want to make Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trials Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website. Not evidence Anecdotes, gut feeling, forums, competitors, Linkedin Lunatics, A/B test results websites, Awards, etc. Risk of Bias Quality of Evidence Lower Strong Weak Higher

Slide 31

Slide 31 text

© Eidra The maturity levels for each driver tend to vary and can enable or hinder your growth. The weaker areas act as bottlenecks of your optimization efforts. 2025-05-26 31 Tools Process Organization Culture Focus

Slide 32

Slide 32 text

© Eidra 2025-05-26 32 From MacGyver to Booking.com/Spotify/etc. Level 1 Basic Level 2 Beginner Level 3 Intermediate Level 4 Expert Level 5 Master Focus Specific Person Team Level Digital Management Level Management Level Entire Company Culture Gut feeling & Loose ideas Random observations from web analytics data Data driven when possible Continuous data-driven decisions & Prioritizations Company-wide data-driven experimentation in your DNA Resources Local Hero Responsibilities in Team Part-time CRO Specialist Full-time CRO Specialist CRO Team Knowledge Online Marketing Basics Simple A/B Testing & Basic Analysis Deeper CRO with regular A/B Testing & statistics Excellence in CRO with advanced A/B Testing & Advanced analytics Excellence in CRO, A/B testing & analytics with full business perspective Process None Ad-Hoc Prioritized test hypothesis test execution plan Documentation of results & learnings Aligned processes used by team and long term CRO goals & strategy Site reliability engineering integrated with company processes Tools Barely any tools Basic tools & Begging IT for help Tools required for now & Workarounds to get IT help Complete set of quantitative & qualitative tools & IT friends Lean, mean optimization machine with direct IT access

Slide 33

Slide 33 text

© Eidra 2025-05-26 33 Hierarchy of Evidence – Level 1 Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trials Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

© Eidra Fix the obvious issues. 2025-05-26 37 • 404s • Internal 301s • Check if the main conversions actually work • Check your website: • In incognito • On mobile • With a 3G connection. • And witness the cookie banner, push notifications, newsletter popup, coupon code modal, and chat bubble. Did It Spark Joy? And why did the 3rd parties load without consent?

Slide 38

Slide 38 text

© Eidra 2025-05-26 38 Hierarchy of Evidence – Level 1 Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trails Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.

Slide 39

Slide 39 text

© Eidra 2025-05-26 39 Hierarchy of Evidence – Level 2 Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trials Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.

Slide 40

Slide 40 text

Slide 41

Slide 41 text

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Slide 44

Slide 44 text

Slide 45

Slide 45 text

Slide 46

Slide 46 text

Slide 47

Slide 47 text

Slide 48

Slide 48 text

© Eidra 2025-05-26 48 Hierarchy of Evidence – Level 2 Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trails Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.

Slide 49

Slide 49 text

© Eidra 2025-05-26 49 Hierarchy of Evidence – Level 3 Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trails Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.

Slide 50

Slide 50 text

© Eidra Quantitative & Qualitative data 50 • Google Analytics 4 • Hotjar or MS Clarity • Google Search Console and Bing Webmaster Tools • Text analysis on Customer Support data => has never been easier, thanks to AI. 2025-05-26

Slide 51

Slide 51 text

Slide 52

Slide 52 text

Slide 53

Slide 53 text

Slide 54

Slide 54 text

Slide 55

Slide 55 text

© Eidra 2025-05-26 55 Hierarchy of Evidence – Level 3 Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trails Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.

Slide 56

Slide 56 text

© Eidra 2025-05-26 56 Hierarchy of Evidence – Level 4 Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trails Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.

Slide 57

Slide 57 text

Slide 58

Slide 58 text

Slide 59

Slide 59 text

© Eidra 2025-05-26 59 Hierarchy of Evidence – Level 4 Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trials Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.

Slide 60

Slide 60 text

© Eidra 2025-05-26 60 Hierarchy of Evidence – Level 5 Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trials Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.

Slide 61

Slide 61 text

2025-05-26

Slide 62

Slide 62 text

2025-05-26

Slide 63

Slide 63 text

Slide 64

Slide 64 text

Slide 65

Slide 65 text

Slide 66

Slide 66 text

Slide 67

Slide 67 text

Slide 68

Slide 68 text

Slide 69

Slide 69 text

Slide 70

Slide 70 text

© Eidra 2025-05-26 70 Relax. Step by step. Identify your level and start improving. Level 1 Basic Level 2 Beginner Level 3 Intermediate Level 4 Expert Level 5 Master Focus Specific Person Team Level Digital Management Level Management Level Entire Company Culture Gut feeling & Loose ideas Random observations from web analytics data Data driven when possible Continuous data-driven decisions & Prioritizations Company-wide data-driven experimentation in your DNA Resources Local Hero Responsibilities in Team Part-time CRO Specialist Full-time CRO Specialist CRO Team Knowledge Online Marketing Basics Simple A/B Testing & Basic Analysis Deeper CRO with regular A/B Testing & statistics Excellence in CRO with advanced A/B Testing & Advanced analytics Excellence in CRO, A/B testing & analytics with full business perspective Process None Ad-Hoc Prioritized test hypothesis test execution plan Documentation of results & learnings Aligned processes used by team and long term CRO goals & strategy Site reliability engineering integrated with company processes Tools Barely any tools Basic tools & Begging IT for help Tools required for now & Workarounds to get IT help Complete set of quantitative & qualitative tools & IT friends Lean, mean optimization machine with direct IT access

Slide 71

Slide 71 text

© Eidra 2025-05-26 71 Let’s build some pyramids! Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trials Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.

Slide 72

Slide 72 text

© Eidra 2025-05-26 72 Let’s build some pyramids! Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trails Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website. Not evidence Anecdotes, gut feeling, forums, competitors, Linkedin Lunatics, A/B test results websites, Awards, etc. Risk of Bias Quality of Evidence Lower Strong Weak Higher

Slide 73

Slide 73 text

© Eidra 2025-05-26 73 Let’s build some pyramids! Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trials Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website. Not evidence Anecdotes, gut feeling, forums, competitors, Linkedin Lunatics, A/B test results websites, Awards, etc. Risk of Bias Quality of Evidence Lower Strong Weak Higher

Slide 74

Slide 74 text

Thank you! Simon Vreeman [email protected]