How we A/B test at the FT: and what happened when we started testing headlines too

Talk given at Continuous Lifecycle London 2018

Transcript

1. How we A/B test at the Financial Times; and what happened when we started testing headlines too Amy Nicholson, Technical Product Manager, FT.com
2. FAQs Why is it called Ammit?
4. FAQs Why is it called Ammit? Can I test the headlines?
5. This story had a headline A
6. Hypothesis If we wrote a punnier headline for this story, more users would read it
7. What does success look like?
8. Primary metric is click through rate Click Through Rate Of all the people who landed on a page with that headline, how many of those users clicked on that story
9. Getting the most votes clicks isn’t always enough
10. If we just chased clicks we’d write something like this A B
11. A B The other headline was more playful but on-brand ?
12. I’ll tell you later.
13. FAQs Why is it called Ammit? Can I test the headlines? How long will it take?
14. Traffic landing on the feature Uplift expected with this change Quicker tests Headline test Two of the factors involved: uplift and traffic Redesign the footer Swap order in nav bar
16. FAQs Why is it called Ammit? Can I test the headlines? How long will it take? Is there traffic available for the headline test?
17. What exactly is traffic and how might it be unavailable?
18. We split all of our users into 100 buckets 100 0
19. Every FT.com user sits in a bucket...metaphorically 100 0
20. Why do you make users sit in buckets?
21. If a user is in B for both tests... A B
22. Hypothesis If we change the layout from A to B, the overall clicks on the component would increase
23. If a user is in B for both tests and clicks the story... A B A B
24. Show tests to different buckets of users if they interfere 0 Headline test Layout test 100
25. Which tests are already running?
26. What does a test look like?
27. An article test to subscribers 0 onwardJourney (subs, article) 100
28. Another article test to subscribers
29. Show tests to different buckets of users if they interfere 0 onwardJourney (subs, article) audio (subs, article) 100
30. Buckets contain anonymous and subscribers 0 onwardJourney (subs, article) audio (subs, article) 100 Barrier (paywall) test (anon, barrier)
31. Swim lanes for tests that can run in parallel 0 onwardJourney (subs, article) audio (subs, article) 100 Layout test (subs, frontpage) Barrier (paywall) test (anon, barrier)
32. What sort of test is it?
33. If the test is for subscribers then there is traffic available 0 100 Available Available onwardJourney (subs, article) audio (subs, article) Layout test (subs, frontpage) Barrier (paywall) test (anon, barrier)
34. If the test is for subscribers then there is traffic available 0 100 Available onwardJourney (subs, article) audio (subs, article) Layout test (subs, frontpage) Barrier (paywall) test (anon, barrier) Headline test (subs, frontpage)
35. But sometimes there isn’t available traffic...
36. FAQs Why is it called Ammit? Can I test the headlines? How long will it take? Is there traffic available for my headline test? Can my test overlap?
37. Overlap tests by componentising the barrier
39. There is no traffic available for another barrier test 0 100 Blurb/No Blurb
40. Overlap tests by componentising the barrier
41. Overlap tests by componentising the barrier
42. Split the traffic so each test gets 50%......will take a while 0 100 Blurb/No blurb Green buttons/Red buttons
43. Run the tests one after another…..will also take a while 0 100 Blurb/No blurb Green buttons/Red buttons
44. MVT
45. Run a multivariate test…..this will take a while too 0 100 Blurb + Green No blurb + Green Blurb + Red No blurb + Red
46. Overlap tests if the interference is likely to be small 0 100 Blurb/No blurb Green button/Red button
47. Why haven’t you been doing this before?
48. Run tests together but look at them separately A B Blurb/No blurb 10% 9% Winner: Blurb Green/Red buttons 10% 8% Winner: Green Blurb + Green buttons win
50. Looking at them as an MVT has different winners Blurb + Green Blurb + Red No blurb + Green No blurb + Red 10% 12% 12.5% 9%
52. Overlapping would speed things up but there are risks associated
53. FAQs Why is it called Ammit? Can I test the headlines? How long will it take? Is there traffic available for my headline test? Can my test overlap? Could you just quickly check my headline test is ok?
54. Is the test live?
55. A subscriber types FT.com into their browser Preflight Subscriber => frontpage
56. Preflight send user ID to Ammit Preflight Ammit
57. Ammit fetches all of the live tests Preflight Ammit Tests
58. Check that the test is there 0 100 Available onwardJourney (subs, article) audio (subs, article) Layout test (subs, frontpage) Barrier (paywall) test (anon, barrier) Headline test (subs, frontpage)
59. Are users being shown the test?
60. Which users should see the test? 0 100 Available onwardJourney (subs, article) audio (subs, article) Layout test (subs, frontpage) Barrier (paywall) test (anon, barrier) Headline test (subs, frontpage)
61. Filter tests that don’t apply to that user’s bucket 0 100 Available onwardJourney (subs, article) audio (subs, article) Layout test (subs, frontpage) Barrier (paywall) test (anon, barrier) Headline test (subs, frontpage)
62. Ammit fetches information about the user Preflight Ammit Tests User data Subscriber
63. Filter tests that don’t apply to that user’s status 0 100 audio (subs, article) Barrier (paywall) test (anon, barrier) Headline test (subs, frontpage)
64. Filter tests that don’t apply to that user’s status 0 100 audio (subs, article) Headline test (subs, frontpage)
65. Ammit sends tests back to Preflight Preflight Ammit Tests User data Subscriber audio (subs, article) Headline test (subs, frontpage)
66. Ammit assigns variant Preflight Ammit Tests User data Subscriber audio:control (subs, article) Headline test: variant (subs, frontpage)
67. Preflight filters tests further Preflight Ammit Tests User data Subscriber Subscriber => frontpage audio:control (subs, article) Headline test: variant (subs, frontpage)
68. Preflight filters tests further Preflight Ammit Tests User data Subscriber Frontpage app Subscriber => frontpage Headline test:variant (subs, frontpage)
69. Are users seeing the correct variant?
70. Frontpage app made up of lots of story slots Teaser
71. Test and variant comes from Preflight Headline test on? User in test? In variant? Preflight Frontpage app Headline test:variant (subs, frontpage)
72. It’s not fun getting up at 3am to turn the headline test on.
73. Test is always turned on Headline test on? User in test? In variant? Preflight Frontpage app Headline test:variant (subs, frontpage) Headline test on? User in test? In variant?
74. Each test starts when there are 2 headlines for 1 story User in test? In variant? 2 x headlines? Elasticsearch
75. Alternative headlines come from the editors
76. Alternative headlines come from the editors
77. Decide which headline to show
78. You’d see headline B - the variant A B
79. FAQs Why is it called Ammit? Can I test the headlines? How long will it take? Is there traffic available for my headline test? Can my test overlap? Could you just quickly check my headline test is ok? Which variant is winning?
80. Which variant is winning?
81. Is the coin fair or weighted?
82. How you should run a frequentist test Estimate duration of test based on how much data you’d need to reach statistical significance Collect data When test duration is over…. check results Significant Not significant
83. FAQs Why is it called Ammit? Can I test this? How long will it take? Is there traffic available for my test? Can my test overlap? Could you just quickly check my headline test is ok? Which variant is winning? Which variant won?
84. An event fires when the user sees the variant headline eventType: pageView timestamp: 15/5/2018 08:55 userID: user-123
85. An event fires when the user clicks the variant headline eventType: pageView timestamp: 15/5/2018 08:55 userID: user-123 eventType: click timestamp: 15/5/2018 08:56 userID: user-123 storyID: story-456
86. Where do these events go? eventType: pageView timestamp: 15/5/2018 08:55 userID: user-123 eventType: click timestamp: 15/5/2018 08:56 userID: user-123 storyID: story-456 Redshift
87. Replace Redshift with smaller, faster Volt eventType: pageView timestamp: 15/5/2018 08:55 userID: user-123 eventType: click timestamp: 15/5/2018 08:56 userID: user-123 storyID: story-456 Volt
88. 3. The struggle is real(-time data analysis) ~2s ago
89. 3. The struggle is real(-time data analysis)
90. B A ?
91. B
92. Not every test reaches significance….
93. FAQs Why is it called Ammit? Can I test this? How long will it take? Is there traffic available for my test? Can my test overlap? Could you just quickly check my headline test is ok? Which variant is winning? Which variant won? Was it close to significance?
94. Was it close to significance?
95. You have to wait X weeks for a test and then…. Estimate duration of test based on how much data you’d need to reach statistical significance Collect data When test duration is over…. check results Significant Not significant
96. B performed better than A If we run this experiment 100 times, on less than 5 of those occasions, the observed difference between B and A would actually have occurred by random chance
97. https://www.analyticsvidhya.com/blog/2016/06/bayesian-statistics-beginners-simple-english/
98. FAQs Why is it called Ammit? Can I test the headlines? How long will it take? Is there traffic available for my headline test? Can my test overlap? Could you just quickly check my headline test is ok? Which variant is winning? Which variant won? Was it close to significance?
99. Which headline won?
100. A B
101. A B
102. A B
103. A B
104. A B
105. A B
106. Get in touch... @amy_nicholson_ amy.nicholson@ft.com We’re hiring!