Making Email Campaigns More Effective: Send Time Optimization

Making Email Campaigns More Effective: Send Time Optimization

Presented at DataFest Tbilisi 2019:
At Emarsys their goal is to make marketing communication truly personal. A significant part of this is sending messages at the right time. They developed a machine learning algorithm resulting in increased user engagement and shorter times between sending and reading messages. In this talk Ildiko will highlight the most important milestones of this journey: what a successful machine learning project looks like from the birth of a new idea through implementation to measuring results. She will show how her team proved the significance of sending times with a simulation based on past data to avoid costly experiments, how they prototyped several algorithms and what they learned from piloting with a few clients before releasing to all of them.

D43a2505af873d7e10a7d5477fd46586?s=128

Ildikó Czeller

November 14, 2019
Tweet

Transcript

  1. MESSAGE SEND TIME OPTIMIZATION Ildi Czeller Lead Data Scientist @Emarsys

    @czeildi on Twitter, GitHub
  2. business problem initial solution validate release piloting

  3. business problem validate measure (A/B test) improve improve release measure

    (A/B test) piloting initial solution
  4. 1. business problem 2. validate the need 3. initial solution

    4. piloting 5. release
  5. 2000+ client 4B+ contact personal marketing communication at scale

  6. content timing channel

  7. timing within day

  8. _ ? what can we win?

  9. 1. business problem 2. validate the need 3. initial solution

    4. piloting 5. release
  10. personal optimization > industry best practices?

  11. observed engagement 06:00 open rate 16:00 open rate Claire 2

    open /4 send = 50% 3 open /10 send = 30%
  12. pattern or chance?

  13. realistic simulation 06:00 open rate 16:00 open rate Claire x

    4 x 10
  14. pattern, not chance

  15. 1. business problem 2. validate the need 3. initial solution

    4. piloting 5. release
  16. key challenges • few data points on user level •

    results expected from day 1
  17. exploration exploitation Bayesian bandit algorithm

  18. adapt algorithm • 2 hour intervals • 12 months’ past

    data research engineering
  19. 1. business problem 2. validate the need 3. initial solution

    4. piloting 5. release
  20. convince pilots to try

  21. single run shows idea

  22. simulate expected results

  23. None
  24. wait for opens

  25. avoid distorting priors 8:00: 1 email to VIP users (45%)

    16:00: general newsletter (25%)
  26. 1. business problem 2. validate the need 3. initial solution

    4. piloting 5. release
  27. solve scalability issues

  28. iterate on client-facing reporting daily → weekly

  29. takeaway

  30. learn cheaply from realistic simulations

  31. you need trust set expectations give explanations

  32. the algorithm is only one component of success

  33. read Chris Stucchio blog.craftlab.hu/weekend-bias- in-send-time-optimisation-ba80 176af1b9 @czeildi on Twitter