Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Making Email Campaigns More Effective: Send Time Optimization

Making Email Campaigns More Effective: Send Time Optimization

Presented at DataFest Tbilisi 2019:
At Emarsys their goal is to make marketing communication truly personal. A significant part of this is sending messages at the right time. They developed a machine learning algorithm resulting in increased user engagement and shorter times between sending and reading messages. In this talk Ildiko will highlight the most important milestones of this journey: what a successful machine learning project looks like from the birth of a new idea through implementation to measuring results. She will show how her team proved the significance of sending times with a simulation based on past data to avoid costly experiments, how they prototyped several algorithms and what they learned from piloting with a few clients before releasing to all of them.

Ildikó Czeller

November 14, 2019
Tweet

More Decks by Ildikó Czeller

Other Decks in Technology

Transcript

  1. MESSAGE
    SEND TIME
    OPTIMIZATION
    Ildi Czeller
    Lead Data Scientist @Emarsys
    @czeildi on Twitter, GitHub

    View full-size slide

  2. business problem
    initial solution
    validate
    release
    piloting

    View full-size slide

  3. business problem
    validate
    measure
    (A/B test)
    improve
    improve
    release
    measure
    (A/B test)
    piloting
    initial solution

    View full-size slide

  4. 1. business problem
    2. validate the need
    3. initial solution
    4. piloting
    5. release

    View full-size slide

  5. 2000+ client
    4B+ contact
    personal
    marketing communication
    at scale

    View full-size slide

  6. content
    timing
    channel

    View full-size slide

  7. timing within day

    View full-size slide

  8. _
    ?
    what can we win?

    View full-size slide

  9. 1. business problem
    2. validate the need
    3. initial solution
    4. piloting
    5. release

    View full-size slide

  10. personal optimization
    >
    industry best practices?

    View full-size slide

  11. observed engagement
    06:00 open rate 16:00 open rate
    Claire
    2 open /4 send =
    50%
    3 open /10 send =
    30%

    View full-size slide

  12. pattern
    or
    chance?

    View full-size slide

  13. realistic simulation
    06:00 open rate 16:00 open rate
    Claire x 4 x 10

    View full-size slide

  14. pattern,
    not chance

    View full-size slide

  15. 1. business problem
    2. validate the need
    3. initial solution
    4. piloting
    5. release

    View full-size slide

  16. key challenges
    ● few data points on user level
    ● results expected from day 1

    View full-size slide

  17. exploration
    exploitation
    Bayesian bandit
    algorithm

    View full-size slide

  18. adapt algorithm
    • 2 hour intervals
    • 12 months’ past data
    research engineering

    View full-size slide

  19. 1. business problem
    2. validate the need
    3. initial solution
    4. piloting
    5. release

    View full-size slide

  20. convince pilots to try

    View full-size slide

  21. single run shows idea

    View full-size slide

  22. simulate expected results

    View full-size slide

  23. wait for opens

    View full-size slide

  24. avoid distorting priors
    8:00: 1 email to VIP users (45%)
    16:00: general newsletter (25%)

    View full-size slide

  25. 1. business problem
    2. validate the need
    3. initial solution
    4. piloting
    5. release

    View full-size slide

  26. solve scalability issues

    View full-size slide

  27. iterate on
    client-facing reporting
    daily → weekly

    View full-size slide

  28. learn cheaply from
    realistic simulations

    View full-size slide

  29. you need trust
    set expectations
    give explanations

    View full-size slide

  30. the algorithm is
    only one component
    of success

    View full-size slide

  31. read Chris Stucchio
    blog.craftlab.hu/weekend-bias-
    in-send-time-optimisation-ba80
    176af1b9
    @czeildi on Twitter

    View full-size slide