Making Email Campaigns More Effective: Send Time Optimization

Presented at DataFest Tbilisi 2019:
At Emarsys their goal is to make marketing communication truly personal. A significant part of this is sending messages at the right time. They developed a machine learning algorithm resulting in increased user engagement and shorter times between sending and reading messages. In this talk Ildiko will highlight the most important milestones of this journey: what a successful machine learning project looks like from the birth of a new idea through implementation to measuring results. She will show how her team proved the significance of sending times with a simulation based on past data to avoid costly experiments, how they prototyped several algorithms and what they learned from piloting with a few clients before releasing to all of them.

Transcript

1. MESSAGE SEND TIME OPTIMIZATION Ildi Czeller Lead Data Scientist @Emarsys @czeildi on Twitter, GitHub
2. business problem initial solution validate release piloting
3. business problem validate measure (A/B test) improve improve release measure (A/B test) piloting initial solution
4. 1. business problem 2. validate the need 3. initial solution 4. piloting 5. release
5. 2000+ client 4B+ contact personal marketing communication at scale
6. content timing channel
7. timing within day
8. _ ? what can we win?
9. 1. business problem 2. validate the need 3. initial solution 4. piloting 5. release
10. personal optimization > industry best practices?
11. observed engagement 06:00 open rate 16:00 open rate Claire 2 open /4 send = 50% 3 open /10 send = 30%
12. pattern or chance?
13. realistic simulation 06:00 open rate 16:00 open rate Claire x 4 x 10
14. pattern, not chance
15. 1. business problem 2. validate the need 3. initial solution 4. piloting 5. release
16. key challenges • few data points on user level • results expected from day 1
17. exploration exploitation Bayesian bandit algorithm
18. adapt algorithm • 2 hour intervals • 12 months’ past data research engineering
19. 1. business problem 2. validate the need 3. initial solution 4. piloting 5. release
20. convince pilots to try
21. single run shows idea
22. simulate expected results
24. wait for opens
25. avoid distorting priors 8:00: 1 email to VIP users (45%) 16:00: general newsletter (25%)
26. 1. business problem 2. validate the need 3. initial solution 4. piloting 5. release
27. solve scalability issues
28. iterate on client-facing reporting daily → weekly
29. takeaway
30. learn cheaply from realistic simulations
31. you need trust set expectations give explanations
32. the algorithm is only one component of success
33. read Chris Stucchio blog.craftlab.hu/weekend-bias- in-send-time-optimisation-ba80 176af1b9 @czeildi on Twitter