Amelia Showalter - What Drives Action Online?

Transcript

1. A/B Testing on the Obama Campaign Winning With Data
2. Introduction
3. You probably thought we had it easy…
4. You probably thought we had it easy…
5. You probably thought we had it easy…
6. You probably thought we had it easy…
7. You probably thought we had it easy…
8. You probably thought we had it easy…
9. But victory was never assured  Big-spending groups called Super-PACs were largely supporting Romney  Obama was usually ahead in the polls, but the advantage was narrow and volatile
10. You might’ve heard these stories…
11. You might’ve heard these stories…
12. You might’ve heard these stories…
13. You might’ve heard these stories…
14. You might’ve heard these stories…
15. You might’ve heard these stories…
16. What you didn’t hear All of that costs money
17. What you didn’t hear All of that costs money
18.  In 2008, Obama campaign raised $750 million The fundraising challenge
19.  In 2008, Obama campaign raised $750 million  Would not be enough in 2012 The fundraising challenge
20.  In 2008, Obama campaign raised $750 million  Would not be enough in 2012 The fundraising challenge Not impressed. $750 million?
21. The fundraising challenge  But fundraising was proving more difficult in 2012 than in 2008
22. The fundraising challenge  But fundraising was proving more difficult in 2012 than in 2008  President less available for fundraising events
23. The fundraising challenge  But fundraising was proving more difficult in 2012 than in 2008  President less available for fundraising events  In early campaign, we saw average online donation was half of what it had been in 2008
24. The fundraising challenge  But fundraising was proving more difficult in 2012 than in 2008  President less available for fundraising events  In early campaign, we saw average online donation was half of what it had been in 2008  People were giving less, and less often
25. The fundraising challenge  But fundraising was proving more difficult in 2012 than in 2008  President less available for fundraising events  In early campaign, we saw average online donation was half of what it had been in 2008  People were giving less, and less often  We had to be smarter and more innovative
26. Overview  A/B testing in Obama’s digital department  Lessons learned  Don’t trust your gut  Foster a culture of testing  Invest in your team  The big picture: make it personal
27. Winning with A/B Testing
28. What impact can testing have?
29. Testing = constant improvement  Little improvements add up  Improving 1% here and 2% there isn’t a lot at first, but over time it adds up
30. Testing = many variations
31. Testing = listening to your audience  Nearly every email was tested, often on multiple levels  We started with message and subject line tests  4 messages, 3 subject lines were tested on every national send (sometimes 6x3)  These tests went to about 20% of the list  After an hour, we would send the winner to the remainder of he list
32. Testing = listening to your audience  Nearly every email was tested, often on multiple levels
33. Example: Draft language
34. Example: Draft language
35. Example: Draft language
36. Example: Draft language
37. Example: Draft language
38. Example: Draft language
39. Example: Subject lines version Subject line v1s1 Hey v1s2 Two things: v1s3 Your turn v2s1 Hey v2s2 My opponent v2s3 You decide v3s1 Hey v3s2 Last night v3s3 Stand with me today v4s1 Hey v4s2 This is my last campaign v4s3 [NAME] v5s1 Hey v5s2 There won't be many more of these deadlines v5s3 What you saw this week v6s1 Hey v6s2 Let's win. v6s3 Midnight deadline  Each draft was tested with three subject lines  One subject line would usually be common across all drafts, to help make comparisons across messages Test sends
40. Example: Best vs. Worst Versions version Subject line donors money v1s1 Hey 263 $17,646 v1s2 Two things: 268 $18,830 v1s3 Your turn 276 $22,380 v2s1 Hey 300 $17,644 v2s2 My opponent 246 $13,795 v2s3 You decide 222 $27,185 v3s1 Hey 370 $29,976 v3s2 Last night 307 $16,945 v3s3 Stand with me today 381 $25,881 v4s1 Hey 444 $25,643 v4s2 This is my last campaign 369 $24,759 v4s3 [NAME] 514 $34,308 v5s1 Hey 353 $22,190 v5s2 There won't be many more of these deadlines 273 $22,405 v5s3 What you saw this week 263 $21,014 v6s1 Hey 363 $25,689 v6s2 Let's win. 237 $17,154 v6s3 Midnight deadline 352 $23,244 $0 $1 $2 $3 $4 ACTUAL ($3.7m) IF SENDING AVG IF SENDING WORST Full send (in millions)  $2.2 million additional revenue from sending best draft vs. worst, or $1.5 million additional from sending best vs. average Test sends
41. Test every element  After testing drafts and subject lines, we would split the remaining list and run additional tests
42. Test every element  After testing drafts and subject lines, we would split the remaining list and run additional tests  Example: Unsubscribe language Variation Recips Unsubs Unsubs per recipient Significant differences in unsubs per recipient 578,994 105 0.018% None 578,814 79 0.014% Smaller than D4 578,620 86 0.015% Smaller than D4 580,507 115 0.020% Larger than D3 and D4
43. Test every element  After testing drafts and subject lines, we would split the remaining list and run additional tests  Example: Unsubscribe language Variation Recips Unsubs Unsubs per recipient Significant differences in unsubs per recipient 578,994 105 0.018% None 578,814 79 0.014% Smaller than D4 578,620 86 0.015% Smaller than D4 580,507 115 0.020% Larger than D3 and D4
44. No, really. Test every element.  We also were always running tests in the background via personalized content
45. Tests upon tests upon tests Review: Every piece of communication was an opportunity to test A single email often had many tests attached
46. Tests upon tests upon tests  Subject & draft tests Review: Every piece of communication was an opportunity to test A single email often had many tests attached
47. Tests upon tests upon tests  Subject & draft tests  Full-list tests Review: Every piece of communication was an opportunity to test A single email often had many tests attached
48. Tests upon tests upon tests  Subject & draft tests  Full-list tests  Background personalization tests Review: Every piece of communication was an opportunity to test A single email often had many tests attached
49. The results
50. The results  Campaign raised over one billion dollars  Raised over half a billion dollars online  Over 4 million Americans donated
51. The results  Campaign raised over one billion dollars  Raised over half a billion dollars online  Over 4 million Americans donated  Recruited tens of thousands of volunteers, publicized thousands of events and rallies
52. The results  Campaign raised over one billion dollars  Raised over half a billion dollars online  Over 4 million Americans donated  Recruited tens of thousands of volunteers, publicized thousands of events and rallies  Did I mention raising >$500 million online?
53. The results  Campaign raised over one billion dollars  Raised over half a billion dollars online  Over 4 million Americans donated  Recruited tens of thousands of volunteers, publicized thousands of events and rallies  Did I mention raising >$500 million online?  Conservatively, testing probably resulted in ~$200 million in additional revenue
54. Lessons
55. Don’t Trust Your Gut Lesson #1
56. Don’t trust your gut  We don’t have all the answers  Conventional wisdom is often wrong  Long-held best practices are often wrong  You are not your audience
57. Don’t trust your gut  We don’t have all the answers  Conventional wisdom is often wrong  Long-held best practices are often wrong  You are not your audience  There was this thing called the Email Derby…
58. Don’t trust your gut  We don’t have all the answers  Conventional wisdom is often wrong  Long-held best practices are often wrong  You are not your audience  There was this thing called the Email Derby…  If even the experts are bad at predicting a winning message, it shows just how important testing is.
59. Experiments: Ugly vs. Pretty  We tried making our emails prettier
60. Experiments: Ugly vs. Pretty  We tried making our emails prettier  That failed
61. Experiments: Ugly vs. Pretty  We tried making our emails prettier  That failed  So we asked: what about ugly?
62. Experiments: Ugly vs. Pretty  We tried making our emails prettier  That failed  So we asked: what about ugly?
63. Experiments: Ugly vs. Pretty  We tried making our emails prettier  That failed  So we asked: what about ugly?  Ugly yellow highlighting got us better results
64. Foster a culture of testing Lesson #2
65. The culture of testing  Check your ego at the door
66. The culture of testing  Check your ego at the door
67. The culture of testing  Check your ego at the door  Use every opportunity to test something
68. The culture of testing  Check your ego at the door  Use every opportunity to test something  Compare against yourself, not against your competitors or “the industry”  Are you doing better this month than last month?  Are you doing better than you would have otherwise?
69. When in doubt, test  In a culture of testing, all questions are answered empirically  Example: With the ugly yellow highlighting, we worried about the novelty factor  Maybe highlighting would only work for a short time before people started ignoring it (or being irritated by it).  We decided to do a multi-stage test across three consecutive emails
70. The ugly highlighting experiment  Experimental design:  Determined through this test that novelty was indeed a factor Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 Group 7 Group 8 Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 Group 7 Group 8 Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 Group 7 Group 8 First Email Second Email Third Email
71. Keep a testing calendar
72. Keep a testing calendar  On the Obama campaign we had short-term and long-term calendars for national emails
73. Keep a testing calendar  On the Obama campaign we had short-term and long-term calendars for national emails  We added a “tests” column to plan out which tests would be attached to which emails
74. Keep a testing calendar  On the Obama campaign we had short-term and long- term calendars for national emails  We added a “tests” column to plan out which tests would be attached to which emails  If we saw blank spaces, it would remind us to think of more tests to run!  Important to do frequent brainstorming sessions
75. Circulate your test results internally  We had an internal listserv entirely for the express purpose of circulating test results
76. Circulate your test results internally  We had an internal listserv entirely for the express purpose of circulating test results  Helped get buy-in and increased familiarity with the testing process
77. Circulate your test results internally  We had an internal listserv entirely for the express purpose of circulating test results  Helped get buy-in and increased familiarity with the testing process  Prompted discussions and generated new ideas for tests
78. Invest in your team Lesson #3
79. OFA Digital Department  Grew from a small team in spring 2011 to a department of 200+ in 2012  Outbound (email, social, mobile, blog)  Ads  Front-End Development  Design  Video  Project management  Digital Analytics
80. Hire smart, diverse talent  Many Obama staffers (maybe most) had never worked in politics before  Experience less important than aptitude & passion  Diverse voices led to better content, analysis
81. Hire smart, diverse talent  Many Obama staffers (maybe most) had never worked in politics before  Experience less important than aptitude & passion  Diverse voices led to better content, analysis  Digital analytics: looked for people with strong quantitative skills, willingness to learn  Staff learned politics and programming on the job
82. Using data to create a more human experience The Big Picture
83. Big data ≠ big brother  Testing allows you to listen to your user base  Let them tell you what they like  Whether through A/B testing or behavioral segmentation, optimization gives them a better experience  Usually, the interactions that are the most human are the ones that win
84. Be human!
85. Be human!  In general, we founds shorter, less formal emails and subject lines did best.  Classic example: “Hey”
86. Be human!  In general, we founds shorter, less formal emails and subject lines did best.  Classic example: “Hey”  When we dropped a mild curse word into a subject line, it usually won  “Hell yes, I like Obamacare”  “Let’s win the damn election”  “Pretty damn cool”
87. Behavioral segmentation  Behavioral segmentation makes the experience personal  Donor vs. non-donor  High-dollar vs. low-dollar  Volunteer status  What issues do people say they care about?  After using A/B tests to create a winning message, we could tweak it slightly for various behavioral groups and get better results
88. Experiments: Personalization  Adding “drop-in sentences” that reference people’s past behavior can increase conversion rates
89. Experiments: Personalization  Adding “drop-in sentences” that reference people’s past behavior can increase conversion rates  Example: asking recent donors for more money
90. Experiments: Personalization  Adding “drop-in sentences” that reference people’s past behavior can increase conversion rates  Example: asking recent donors for more money …it's going to take a lot more of us to match them. You stepped up recently to help out -- thank you. We all need to dig a little deeper if we're going to win, so I'm asking you to pitch in again. Will you donate $25 or more today? …it's going to take a lot more of us to match them. Will you donate $25 or more today?
91. Experiments: Personalization  Adding “drop-in sentences” that reference people’s past behavior can increase conversion rates  Example: asking recent donors for more money  Added sentence significantly raised donation rate  Confirmed in several similar experiments …it's going to take a lot more of us to match them. You stepped up recently to help out -- thank you. We all need to dig a little deeper if we're going to win, so I'm asking you to pitch in again. Will you donate $25 or more today? …it's going to take a lot more of us to match them. Will you donate $25 or more today?
92. Mobilization = Human Interactions From 2012 Campaign Manager Jim Messina: “My favorite story is from a volunteer in Wisconsin 10 days out [from Election Day]. She was knocking on doors on one side of the street and the Romney campaign was knocking on doors on the other side of the street…”
93. Mobilization = Human Interactions “… [The Obama volunteer] was asked to hit two doors. One was an undecided voter and she knew exactly what to say. The other was an absentee ballot and she was told to make sure they filled it out and returned it. On the other side of the street, the Romney campaign was knocking on every single door. Most of the people weren’t home, and most of the people that were home were already supporting Barack Obama. She looked at me and said, ‘You’re using my time wisely.’ That’s what data can do.” - Obama 2012 Campaign Manager Jim Messina
94. Conclusions
95. Conclusions  Make it personal  Test everything  Never stop looking for new ideas, new voices, and new innovations