Increasing Message Efficiency for LINE Stickers with Logic

Koki Nagamine (LINE / OA Data Science Team / Data Scientist)

https://tech-verse.me/ja/sessions/43
https://tech-verse.me/en/sessions/43
https://tech-verse.me/ko/sessions/43

Transcript

2. Self Introduction > Koki Nagamine > Joined LINE Corporation as Data Scientist in February 2020 > Working on data analysis for LINE Official Account
3. Agenda - Background & Product Issue - Approaches for Product Issue - Increasing Message Efficiency for LINE Stickers with Logic - Summary
4. Chat Message Call LINE Official Account Support Communication between account owner and users
5. Too Many Messages from LINE Official Account
6. “User Voice” : Questionnaire User feel too many messages Too Many Many Neutral Few Too Few Useful Slightly Useful Neutral Slightly Useless Useless About Message Volume Q. Is the amount of message from LINE Official Account appropriate ? About Message Quality Q.Is the message from LINE Official Account useful? 2019.8 questionnaire by line research
7. “User Reaction” : Data Analysis More receiving, Less reaction - More messages tend to users less reaction such as open, click - Reducing wasteful messages is necessary for official account platform # of receive message Open Rate Click Rate User reaction by receive message
8. Theme of this session Increasing Message Efficiency for LINE Official Account managed by In-House Services LINE
9. Approaches for Product Issue
10. Prediction of opens by ML system to filter wasteful messages before delivery Open Score Targeting Open Score Generator OA User Score Send 1 A 0.3 No 1 B 0.4 No 2 A 0.8 Yes 2 C 0.5 No
11. Approaches for Product Issue Open Score Targeting - In simulation, Reducing 18% messages of LINE Official Account managed by In-House services But … - In fact, messages did not decrease - We need Click / CV prediction
12. Click Score Targeting / OA Score Prediction of opens, click by ML system to filter wasteful messages before delivery Open Score Generator OA User Score Send 1 A 0.3 No 1 B 0.8 Yes 2 A 0.8 Yes 2 C 0.5 No Click Score Generator OA User Score Send 1 A 0.1 No 1 B 0.8 Yes 2 A 0.5 No 2 C 0.2 No OA User Score Send 1 A 0.3 No 1 B 0.4 No 2 A 0.8 Yes 2 C 0.5 No OA Score New!!
13. OA Score Simulation 1.00 1.00 0.35 0.62 0.45 0.84 delivered message count click count all deliver OpenScoreTargeting OAScore Targeting by OA Score delivered roughly 2 times more efficiently than all deliver
14. Approaches for Product Issue Open Score Targeting Click Score Targeting / OA Score - In simulation, good performance! But … - Could not obtain the understanding of In-House services
15. OA Message Cap Set policy to limit the number of In-House service OA Messages - Ongoing project - In simulation, Reducing 50% messages of LINE Official Account managed by In-House service Limit by monthly
16. Approaches for Product Issue Open Score Targeting Click Score Targeting / OA Score OA Message Cap - Ongoing project - In simulation, Reducing 50% messages of LINE Official Account managed by In-House service
17. Approaches for Product Issue Open Score Targeting Click Score Targeting / OA Score OA Message Cap LINE Stickers OA Message Update - Specific examples of LINE Official Account improvement Go To Next Slide …
18. Increasing Message Efficiency for LINE Stickers with Logic
19. LINE Sticker Official Account The largest In-House OA managed by LINE Sticker - In-House OA managed by LINE Sticker - The largest OA with 73 million followers
20. “Personal Recommend” Message Message Image - Message Type : carousel - Date : Every Friday - Content : Personal Recommend Stickers Specs
21. Current / New Logics for message delivered Current Logic New Logic OA User Score Send 1 A 0.3 No 1 B 0.8 Yes 2 A 0.8 Yes 2 C 0.5 No OA Score From buy stickers In 7 days 8 ~ 14 days Over 15 days From receive message In 7days Not deliver Not deliver Not deliver 8 ~ 14 days Not deliver Deliver Not deliver Over 15 days Deliver Deliver Deliver ML models
22. OA Score How to make OA Score using prediction of open and click OA User Score Send 1 A 0.3 No 1 B 0.8 Yes 2 A 0.8 Yes 2 C 0.5 No OA User Score Send 1 A 0.1 No 1 B 0.8 Yes 2 A 0.5 No 2 C 0.2 No OA User Score Send 1 A 0.3 No 1 B 0.4 No 2 A 0.8 Yes 2 C 0.5 No OA Score using Max Value Open Score Generator Click Score Generator
23. OA Score How to make OA Score using prediction of open and click Open Score Generator Click Score Generator OA Score [0, 0.1] (0.1, 0.2] (0.2, 0.3] (0.3, 0.4] (0.4, 0.5] (0.5, 0.6] (0.6, 0.7] (0.7, 0.8] (0.8, 0.9] (0.9, 1] [0, 0.1] (0.1, 0.2] (0.2, 0.3] (0.3, 0.4] (0.4, 0.5] (0.5, 0.6] (0.6, 0.7] (0.7, 0.8] (0.8, 0.9] (0.9, 1] [0, 0.1] (0.1, 0.2] (0.2, 0.3] (0.3, 0.4] (0.4, 0.5] (0.5, 0.6] (0.6, 0.7] (0.7, 0.8] (0.8, 0.9] (0.9, 1] [0, 0.1] (0.1, 0.2] (0.2, 0.3] (0.3, 0.4] (0.4, 0.5] (0.5, 0.6] (0.6, 0.7] (0.7, 0.8] (0.8, 0.9] (0.9, 1] ranked
24. AB test Summary - Test Target : 1. Which is better current or new? 2. How score is best threshold? - Content : Personal Recommend Stickers Sticker OA followers Delivered by current logic All deliver, simulated by OA Score Low High OA Score
25. which is better current or new? New logic outperformed current logic in Efficiency in all threshold case Current New - All threshold : 0.4 threshold : 0.5 threshold : 0.6 Group User 1,000 user 1,000 user Delivered User 383 user 1000 user 618 user 527 user 440 user Click User 5.3 user 14.8 user 1.34 user 1.26 user 1.16 user Efficiency (Click / Delivered) 1.39% 1.48% 2.17% 2.39% 2.65%
26. How score is best threshold? Chose a threshold of 0.6 for several reasons, including a business perspective Simulation vs Current Logic [0, 0.1] (0.1, 0.2] (0.2, 0.3] (0.3, 0.4] (0.4, 0.5] (0.5, 0.6] (0.6, 0.7] (0.7, 0.8] (0.8, 0.9] (0.9, 1] Delivered message Count [0, 0.1] (0.1, 0.2] (0.2, 0.3] (0.3, 0.4] (0.4, 0.5] (0.5, 0.6] (0.6, 0.7] (0.7, 0.8] (0.8, 0.9] (0.9, 1] Click Count 0.4 0.5 0.6 threshold : 0.4 threshold : 0.5 threshold : 0.6 Delivered User 1.61 1.38 1.15 Click User 2.52 2.37 2.19 Sales Conversion 3.61 3.46 3.24
27. Logic Change Result Actual value, the result was comparable to the simulation AB test Simulation Metrics Actual Value 1.15 Delivered User 1.32 2.19 Click User 2.77 3.24 Sales Conversion 4.52 Current Logic VS New Logic
28. Summary - In this session, I talked about the “Increasing Message Efficiency for LINE Stickers with Logic project” using OA Score - We will continue to take on challenges to realize OA platform that is comfortable for users - Since 2019, we have been challenging the “Increasing Message Efficiency for LINE Official Account project”, which is still ongoing
29. Thank you