A/B testing for optimising a page with 12 million monthly visits

In this talk, as the homepage owner of Vodafone.co.uk, with monthly 12 million visits have the privilege to work on multiple, I will share the learning from A/B testing carried out and go in more deeper how to design the case, ideation till the outcome to share some interesting insights around personalisation, animation

Transcript

1. A/B testing for optimising a page with 12 million monthly visits Ying Wang Vodafone UK Speakerdeck.com/YingWang @yingwang_yw
2. C2 General Almost 15 years ago in Hong Kong Airport
3. C2 General 25 April 2024 3
4. C2 General What is A/B Testing?
5. C2 General 25 April 2024 5 A B
6. C2 General Why do we do A/B testing?
7. C2 General 25 April 2024 7
8. C2 General 25 April 2024 8 12M visits Helps reduce risk and seize the optimisation opportunities x 0.1% CTR =12K Hypothetical
9. C2 General How do we plan A/B testing?
10. C2 General The stages of A/B testing Planning 01 02 03 04 Objective 05 06 07 Insight Hypothesis KPIs Traffic split Audience Timing
11. C2 General Caveat
12. C2 General Caveat One change Multiple changes
13. A/B testing: animation 13 25 April 2024
14. C2 General Hero Banner: static banner vs animated banner A/B testing 25 April 2024 14 Version B with Animated imagery Version A with static imagery A B
15. C2 General Static banner vs animated banner test for Samsung new product launch Version B with Animated imagery Version A with static imagery A B
16. C2 General OBJECTIVE Adding animation during Samsung's New Product Launch to highlight and bring to life the new Samsung handsets’ feature (examine if animation is more captivating to customers) Planning the animation A/B testing
17. C2 General HYPOTHESIS Animation version banner will help draw our customers’ attention, boost engagement rates and enhance the overall user experience, resulting in more clicks and conversions Planning the animation A/B testing
18. C2 General IMPLEMENTATION Split our traffic base 50/50 so half of the audience see static version and the other half see the animated version Planning the animation A/B testing
19. C2 General KPI Primary KPI: click-through rate Secondary KPI: conversion rate Planning the animation A/B testing
20. C2 General Animation had a significant impact across Vodafone traffic base overall OUTCOME CTR 1.3% CVR(mobile) 5.2% HYPOTHESIS PROVEN
21. C2 General CONCLUSION To draw attention to important content, add visual interest by creating a more polished and sleek animation of Samsung's top range model Animation helps enhance UX, making the site more engaging and dynamic
22. C2 General Different users responded to animation differently, depending on campaign phases and user journeys Buy Now Audience CTR /CVR Pre-order Acquisition Customers Phase Outcome Retention Customers CTR /CVR
23. C2 General Animation Case Study 2023
24. C2 General 25 April 2024 24 Animated 11/12-18/12 (9 days) Static 28/11-10/12 (13 days) CTR 3.34% CTR 4.18% CTR 0.95% CTR 0.95% Demand Generation: Christmas animation Acquisition Audience Existing Audience
25. 25 WIMBLEDON CONNECTS
26. C2 General 25 April 2024 26 Brand Awareness: animation implementation Mobile UI look & feel Laptop UI look & feel
27. C2 General Building brand awareness with animation . Animated video is a powerful tool for increasing brand awareness CONCLUSION
28. C2 General Market Trend Observation
29. C2 General 29 Vodafone UK Homepage Vodafone Turkey app Vodafone Turkey takes inspiration from UK website
30. C2 General More operators might be following the trend? 25 April 2024 30
31. C2 General 25 April 2024 31 Use guidelines, but stay open-minded Page Speed Ranking, Keywords Core Web Vitals Stay Open Minded
32. LESS is MORE 32 25 April 2024
33. C2 General Savings Messaging: A/B test Version B with multiple saving shouts Version A with single saving shout
34. C2 General OBJECTIVE Comparing how different ways of phrasing our message can drive better awareness and higher CTR Savings Messaging: test planning
35. C2 General HYPOTHESIS Customers want to save more – more saving details will drive better clicks and conversions Savings Messaging: test planning
36. C2 General IMPLEMENTATION Acquisition Audience split: 50% Version A vs 50% Version B Savings Messaging: test planning
37. C2 General KPI Primary KPI: click-through rate Secondary KPI: conversation rate Savings Messaging: test planning
38. C2 General Customers appreciate simpler messaging 25 April 2024 38 OUTCOME CTR 31.4% CVR 3.2% Version A with single saving messaging
39. C2 General . 25 April 2024 39 . Audience spent less time on monthly price messaging before clicking through Version A with single saving shout Version B with multiple saving shouts Spent 33.5s to click through on Version A Spent 31s to click through on Version B
40. C2 General CONCLUSION Customers are sensitive to clear saving messaging; they seem to react less positively to multiple financial savings call-outs in a single piece of copy Clear, concise marketing works best
41. C2 General Monthly Price vs Overall Savings: A/B test Version B with Monthly Saving Headline Version A with Overall Saving Headline Ver. C with Overall Saving Headline Monthly saving sub-copy Control A B C
42. C2 General Monthly price drives better engagement 25 April 2024 42 CTR 131.41% Version B with Monthly price headline OUTCOME
43. C2 General CONCLUSION A clear increase in CTR through the hero banner; variant B backed up the hypothesis that monthly price messaging drives better engagement Customers appreciate clearer messaging
44. There's no single, one- size-fits-all strategy 44 25 April 2024
45. C2 General Black Friday: countdown timer/double CTA test 25 April 2024 A B Version A without countdown timer Version B with countdown timer
46. C2 General OBJECTIVE Test countdown timer during BF period to help create a sense of urgency – gives users greater confidence in future seasonal offerings Black Friday: A/B test planning
47. C2 General Black Friday: A/B test outcome 47 CTR 12% CTR 2.8% B OUTCOME
48. C2 General CONCLUSION For the two separate CTAs there was a marked difference in CTR – BF-specific CTA demonstrates countdown timer can help the site and encourage clicks, but when and how to use it is the key. Timing is key for countdown timer
49. C2 General 25 April 2024 49 Summary Key Takeaways A/B Testing Basics Animation Best practice as guidance Less is more NO single strategy fits all Learn from each experience
50. C2 General 25 April 2024 50 THANK YOU!