A/B testing for optimising a page with 12 million monthly visits

Slide 1

Slide 1 text

A/B testing for optimising a page with 12 million monthly visits Ying Wang Vodafone UK Speakerdeck.com/YingWang @yingwang_yw

Slide 2

Slide 2 text

C2 General Almost 15 years ago in Hong Kong Airport

Slide 3

Slide 3 text

C2 General 25 April 2024 3

Slide 4

Slide 4 text

C2 General What is A/B Testing?

Slide 5

Slide 5 text

C2 General 25 April 2024 5 A B

Slide 6

Slide 6 text

C2 General Why do we do A/B testing?

Slide 7

Slide 7 text

C2 General 25 April 2024 7

Slide 8

Slide 8 text

C2 General 25 April 2024 8 12M visits Helps reduce risk and seize the optimisation opportunities x 0.1% CTR =12K Hypothetical

Slide 9

Slide 9 text

C2 General How do we plan A/B testing?

Slide 10

Slide 10 text

C2 General The stages of A/B testing Planning 01 02 03 04 Objective 05 06 07 Insight Hypothesis KPIs Traffic split Audience Timing

Slide 11

Slide 11 text

C2 General Caveat

Slide 12

Slide 12 text

C2 General Caveat One change Multiple changes

Slide 13

Slide 13 text

A/B testing: animation 13 25 April 2024

Slide 14

Slide 14 text

C2 General Hero Banner: static banner vs animated banner A/B testing 25 April 2024 14 Version B with Animated imagery Version A with static imagery A B

Slide 15

Slide 15 text

C2 General Static banner vs animated banner test for Samsung new product launch Version B with Animated imagery Version A with static imagery A B

Slide 16

Slide 16 text

C2 General OBJECTIVE Adding animation during Samsung's New Product Launch to highlight and bring to life the new Samsung handsets’ feature (examine if animation is more captivating to customers) Planning the animation A/B testing

Slide 17

Slide 17 text

C2 General HYPOTHESIS Animation version banner will help draw our customers’ attention, boost engagement rates and enhance the overall user experience, resulting in more clicks and conversions Planning the animation A/B testing

Slide 18

Slide 18 text

C2 General IMPLEMENTATION Split our traffic base 50/50 so half of the audience see static version and the other half see the animated version Planning the animation A/B testing

Slide 19

Slide 19 text

C2 General KPI Primary KPI: click-through rate Secondary KPI: conversion rate Planning the animation A/B testing

Slide 20

Slide 20 text

C2 General Animation had a significant impact across Vodafone traffic base overall OUTCOME CTR 1.3% CVR(mobile) 5.2% HYPOTHESIS PROVEN

Slide 21

Slide 21 text

C2 General CONCLUSION To draw attention to important content, add visual interest by creating a more polished and sleek animation of Samsung's top range model Animation helps enhance UX, making the site more engaging and dynamic

Slide 22

Slide 22 text

C2 General Different users responded to animation differently, depending on campaign phases and user journeys Buy Now Audience CTR /CVR Pre-order Acquisition Customers Phase Outcome Retention Customers CTR /CVR

Slide 23

Slide 23 text

C2 General Animation Case Study 2023

Slide 24

Slide 24 text

C2 General 25 April 2024 24 Animated 11/12-18/12 (9 days) Static 28/11-10/12 (13 days) CTR 3.34% CTR 4.18% CTR 0.95% CTR 0.95% Demand Generation: Christmas animation Acquisition Audience Existing Audience

Slide 25

Slide 25 text

25 WIMBLEDON CONNECTS

Slide 26

Slide 26 text

C2 General 25 April 2024 26 Brand Awareness: animation implementation Mobile UI look & feel Laptop UI look & feel

Slide 27

Slide 27 text

C2 General Building brand awareness with animation . Animated video is a powerful tool for increasing brand awareness CONCLUSION

Slide 28

Slide 28 text

C2 General Market Trend Observation

Slide 29

Slide 29 text

C2 General 29 Vodafone UK Homepage Vodafone Turkey app Vodafone Turkey takes inspiration from UK website

Slide 30

Slide 30 text

C2 General More operators might be following the trend? 25 April 2024 30

Slide 31

Slide 31 text

C2 General 25 April 2024 31 Use guidelines, but stay open-minded Page Speed Ranking, Keywords Core Web Vitals Stay Open Minded

Slide 32

Slide 32 text

LESS is MORE 32 25 April 2024

Slide 33

Slide 33 text

C2 General Savings Messaging: A/B test Version B with multiple saving shouts Version A with single saving shout

Slide 34

Slide 34 text

C2 General OBJECTIVE Comparing how different ways of phrasing our message can drive better awareness and higher CTR Savings Messaging: test planning

Slide 35

Slide 35 text

C2 General HYPOTHESIS Customers want to save more – more saving details will drive better clicks and conversions Savings Messaging: test planning

Slide 36

Slide 36 text

C2 General IMPLEMENTATION Acquisition Audience split: 50% Version A vs 50% Version B Savings Messaging: test planning

Slide 37

Slide 37 text

C2 General KPI Primary KPI: click-through rate Secondary KPI: conversation rate Savings Messaging: test planning

Slide 38

Slide 38 text

C2 General Customers appreciate simpler messaging 25 April 2024 38 OUTCOME CTR 31.4% CVR 3.2% Version A with single saving messaging

Slide 39

Slide 39 text

C2 General . 25 April 2024 39 . Audience spent less time on monthly price messaging before clicking through Version A with single saving shout Version B with multiple saving shouts Spent 33.5s to click through on Version A Spent 31s to click through on Version B

Slide 40

Slide 40 text

C2 General CONCLUSION Customers are sensitive to clear saving messaging; they seem to react less positively to multiple financial savings call-outs in a single piece of copy Clear, concise marketing works best

Slide 41

Slide 41 text

C2 General Monthly Price vs Overall Savings: A/B test Version B with Monthly Saving Headline Version A with Overall Saving Headline Ver. C with Overall Saving Headline Monthly saving sub-copy Control A B C

Slide 42

Slide 42 text

C2 General Monthly price drives better engagement 25 April 2024 42 CTR 131.41% Version B with Monthly price headline OUTCOME

Slide 43

Slide 43 text

C2 General CONCLUSION A clear increase in CTR through the hero banner; variant B backed up the hypothesis that monthly price messaging drives better engagement Customers appreciate clearer messaging

Slide 44

Slide 44 text

There's no single, one- size-fits-all strategy 44 25 April 2024

Slide 45

Slide 45 text

C2 General Black Friday: countdown timer/double CTA test 25 April 2024 A B Version A without countdown timer Version B with countdown timer

Slide 46

Slide 46 text

C2 General OBJECTIVE Test countdown timer during BF period to help create a sense of urgency – gives users greater confidence in future seasonal offerings Black Friday: A/B test planning

Slide 47

Slide 47 text

C2 General Black Friday: A/B test outcome 47 CTR 12% CTR 2.8% B OUTCOME

Slide 48

Slide 48 text

C2 General CONCLUSION For the two separate CTAs there was a marked difference in CTR – BF-specific CTA demonstrates countdown timer can help the site and encourage clicks, but when and how to use it is the key. Timing is key for countdown timer

Slide 49

Slide 49 text

C2 General 25 April 2024 49 Summary Key Takeaways A/B Testing Basics Animation Best practice as guidance Less is more NO single strategy fits all Learn from each experience

Slide 50

Slide 50 text

C2 General 25 April 2024 50 THANK YOU!