Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Making Email Campaigns More Effective: Send Time Optimization

Making Email Campaigns More Effective: Send Time Optimization

Presented at DataFest Tbilisi 2019:
At Emarsys their goal is to make marketing communication truly personal. A significant part of this is sending messages at the right time. They developed a machine learning algorithm resulting in increased user engagement and shorter times between sending and reading messages. In this talk Ildiko will highlight the most important milestones of this journey: what a successful machine learning project looks like from the birth of a new idea through implementation to measuring results. She will show how her team proved the significance of sending times with a simulation based on past data to avoid costly experiments, how they prototyped several algorithms and what they learned from piloting with a few clients before releasing to all of them.

Ildikó Czeller

November 14, 2019
Tweet

More Decks by Ildikó Czeller

Other Decks in Technology

Transcript

  1. MESSAGE
    SEND TIME
    OPTIMIZATION
    Ildi Czeller
    Lead Data Scientist @Emarsys
    @czeildi on Twitter, GitHub

    View Slide

  2. business problem
    initial solution
    validate
    release
    piloting

    View Slide

  3. business problem
    validate
    measure
    (A/B test)
    improve
    improve
    release
    measure
    (A/B test)
    piloting
    initial solution

    View Slide

  4. 1. business problem
    2. validate the need
    3. initial solution
    4. piloting
    5. release

    View Slide

  5. 2000+ client
    4B+ contact
    personal
    marketing communication
    at scale

    View Slide

  6. content
    timing
    channel

    View Slide

  7. timing within day

    View Slide

  8. _
    ?
    what can we win?

    View Slide

  9. 1. business problem
    2. validate the need
    3. initial solution
    4. piloting
    5. release

    View Slide

  10. personal optimization
    >
    industry best practices?

    View Slide

  11. observed engagement
    06:00 open rate 16:00 open rate
    Claire
    2 open /4 send =
    50%
    3 open /10 send =
    30%

    View Slide

  12. pattern
    or
    chance?

    View Slide

  13. realistic simulation
    06:00 open rate 16:00 open rate
    Claire x 4 x 10

    View Slide

  14. pattern,
    not chance

    View Slide

  15. 1. business problem
    2. validate the need
    3. initial solution
    4. piloting
    5. release

    View Slide

  16. key challenges
    ● few data points on user level
    ● results expected from day 1

    View Slide

  17. exploration
    exploitation
    Bayesian bandit
    algorithm

    View Slide

  18. adapt algorithm
    • 2 hour intervals
    • 12 months’ past data
    research engineering

    View Slide

  19. 1. business problem
    2. validate the need
    3. initial solution
    4. piloting
    5. release

    View Slide

  20. convince pilots to try

    View Slide

  21. single run shows idea

    View Slide

  22. simulate expected results

    View Slide

  23. View Slide

  24. wait for opens

    View Slide

  25. avoid distorting priors
    8:00: 1 email to VIP users (45%)
    16:00: general newsletter (25%)

    View Slide

  26. 1. business problem
    2. validate the need
    3. initial solution
    4. piloting
    5. release

    View Slide

  27. solve scalability issues

    View Slide

  28. iterate on
    client-facing reporting
    daily → weekly

    View Slide

  29. takeaway

    View Slide

  30. learn cheaply from
    realistic simulations

    View Slide

  31. you need trust
    set expectations
    give explanations

    View Slide

  32. the algorithm is
    only one component
    of success

    View Slide

  33. read Chris Stucchio
    blog.craftlab.hu/weekend-bias-
    in-send-time-optimisation-ba80
    176af1b9
    @czeildi on Twitter

    View Slide