Share
Embed
Increasing Message Efficiency for LINE Stickers with Logic
Koki Nagamine (LINE / OA Data Science Team / Data Scientist)
https://tech-verse.me/ja/sessions/43
https://tech-verse.me/en/sessions/43
https://tech-verse.me/ko/sessions/43
Tech-Verse2022
November 17, 2022
Transcript
29 slides
-
Self Introduction > Koki Nagamine > Joined LINE Corporation as
Data Scientist in February 2020 > Working on data analysis for LINE Official Account -
Agenda - Background & Product Issue - Approaches for Product
Issue - Increasing Message Efficiency for LINE Stickers with Logic - Summary -
“User Voice” : Questionnaire User feel too many messages Too
Many Many Neutral Few Too Few Useful Slightly Useful Neutral Slightly Useless Useless About Message Volume Q. Is the amount of message from LINE Official Account appropriate ? About Message Quality Q.Is the message from LINE Official Account useful? 2019.8 questionnaire by line research -
“User Reaction” : Data Analysis More receiving, Less reaction -
More messages tend to users less reaction such as open, click - Reducing wasteful messages is necessary for official account platform # of receive message Open Rate Click Rate User reaction by receive message -
Theme of this session Increasing Message Efficiency for LINE Official
Account managed by In-House Services LINE -
Prediction of opens by ML system to filter wasteful messages
before delivery Open Score Targeting Open Score Generator OA User Score Send 1 A 0.3 No 1 B 0.4 No 2 A 0.8 Yes 2 C 0.5 No -
Approaches for Product Issue Open Score Targeting - In simulation,
Reducing 18% messages of LINE Official Account managed by In-House services But … - In fact, messages did not decrease - We need Click / CV prediction -
Click Score Targeting / OA Score Prediction of opens, click
by ML system to filter wasteful messages before delivery Open Score Generator OA User Score Send 1 A 0.3 No 1 B 0.8 Yes 2 A 0.8 Yes 2 C 0.5 No Click Score Generator OA User Score Send 1 A 0.1 No 1 B 0.8 Yes 2 A 0.5 No 2 C 0.2 No OA User Score Send 1 A 0.3 No 1 B 0.4 No 2 A 0.8 Yes 2 C 0.5 No OA Score New!! -
OA Score Simulation 1.00 1.00 0.35 0.62 0.45 0.84 delivered
message count click count all deliver OpenScoreTargeting OAScore Targeting by OA Score delivered roughly 2 times more efficiently than all deliver -
Approaches for Product Issue Open Score Targeting Click Score Targeting
/ OA Score - In simulation, good performance! But … - Could not obtain the understanding of In-House services -
OA Message Cap Set policy to limit the number of
In-House service OA Messages - Ongoing project - In simulation, Reducing 50% messages of LINE Official Account managed by In-House service Limit by monthly -
Approaches for Product Issue Open Score Targeting Click Score Targeting
/ OA Score OA Message Cap - Ongoing project - In simulation, Reducing 50% messages of LINE Official Account managed by In-House service -
Approaches for Product Issue Open Score Targeting Click Score Targeting
/ OA Score OA Message Cap LINE Stickers OA Message Update - Specific examples of LINE Official Account improvement Go To Next Slide … -
LINE Sticker Official Account The largest In-House OA managed by
LINE Sticker - In-House OA managed by LINE Sticker - The largest OA with 73 million followers -
“Personal Recommend” Message Message Image - Message Type : carousel
- Date : Every Friday - Content : Personal Recommend Stickers Specs -
Current / New Logics for message delivered Current Logic New
Logic OA User Score Send 1 A 0.3 No 1 B 0.8 Yes 2 A 0.8 Yes 2 C 0.5 No OA Score From buy stickers In 7 days 8 ~ 14 days Over 15 days From receive message In 7days Not deliver Not deliver Not deliver 8 ~ 14 days Not deliver Deliver Not deliver Over 15 days Deliver Deliver Deliver ML models -
OA Score How to make OA Score using prediction of
open and click OA User Score Send 1 A 0.3 No 1 B 0.8 Yes 2 A 0.8 Yes 2 C 0.5 No OA User Score Send 1 A 0.1 No 1 B 0.8 Yes 2 A 0.5 No 2 C 0.2 No OA User Score Send 1 A 0.3 No 1 B 0.4 No 2 A 0.8 Yes 2 C 0.5 No OA Score using Max Value Open Score Generator Click Score Generator -
OA Score How to make OA Score using prediction of
open and click Open Score Generator Click Score Generator OA Score [0, 0.1] (0.1, 0.2] (0.2, 0.3] (0.3, 0.4] (0.4, 0.5] (0.5, 0.6] (0.6, 0.7] (0.7, 0.8] (0.8, 0.9] (0.9, 1] [0, 0.1] (0.1, 0.2] (0.2, 0.3] (0.3, 0.4] (0.4, 0.5] (0.5, 0.6] (0.6, 0.7] (0.7, 0.8] (0.8, 0.9] (0.9, 1] [0, 0.1] (0.1, 0.2] (0.2, 0.3] (0.3, 0.4] (0.4, 0.5] (0.5, 0.6] (0.6, 0.7] (0.7, 0.8] (0.8, 0.9] (0.9, 1] [0, 0.1] (0.1, 0.2] (0.2, 0.3] (0.3, 0.4] (0.4, 0.5] (0.5, 0.6] (0.6, 0.7] (0.7, 0.8] (0.8, 0.9] (0.9, 1] ranked -
AB test Summary - Test Target : 1. Which is
better current or new? 2. How score is best threshold? - Content : Personal Recommend Stickers Sticker OA followers Delivered by current logic All deliver, simulated by OA Score Low High OA Score -
which is better current or new? New logic outperformed current
logic in Efficiency in all threshold case Current New - All threshold : 0.4 threshold : 0.5 threshold : 0.6 Group User 1,000 user 1,000 user Delivered User 383 user 1000 user 618 user 527 user 440 user Click User 5.3 user 14.8 user 1.34 user 1.26 user 1.16 user Efficiency (Click / Delivered) 1.39% 1.48% 2.17% 2.39% 2.65% -
How score is best threshold? Chose a threshold of 0.6
for several reasons, including a business perspective Simulation vs Current Logic [0, 0.1] (0.1, 0.2] (0.2, 0.3] (0.3, 0.4] (0.4, 0.5] (0.5, 0.6] (0.6, 0.7] (0.7, 0.8] (0.8, 0.9] (0.9, 1] Delivered message Count [0, 0.1] (0.1, 0.2] (0.2, 0.3] (0.3, 0.4] (0.4, 0.5] (0.5, 0.6] (0.6, 0.7] (0.7, 0.8] (0.8, 0.9] (0.9, 1] Click Count 0.4 0.5 0.6 threshold : 0.4 threshold : 0.5 threshold : 0.6 Delivered User 1.61 1.38 1.15 Click User 2.52 2.37 2.19 Sales Conversion 3.61 3.46 3.24 -
Logic Change Result Actual value, the result was comparable to
the simulation AB test Simulation Metrics Actual Value 1.15 Delivered User 1.32 2.19 Click User 2.77 3.24 Sales Conversion 4.52 Current Logic VS New Logic -
Summary - In this session, I talked about the “Increasing
Message Efficiency for LINE Stickers with Logic project” using OA Score - We will continue to take on challenges to realize OA platform that is comfortable for users - Since 2019, we have been challenging the “Increasing Message Efficiency for LINE Official Account project”, which is still ongoing