Too many LINE Official Account messages? Call data science!

Yoshiaki Nishite
LINE Data Science Team4 Senior Data Scientist
Takahiro Yoshinaga
LINE Machine Learning Solution Team Senior Data Scientist / Machine Learning Engineer
https://linedevday.linecorp.com/2020/ja/sessions/0816
https://linedevday.linecorp.com/2020/en/sessions/0816

Transcript

2. Agenda › Introduction : DataLabs › Background & Direction › Building & Deploying the Model › Results & Future Prospects
3. Introduction Data Labs
4. Organization Data Science Teams Data Platform Data Science And Engineering Center Data Management Engineering Infrastructure LPML Machine Learning Teams Data Labs
5. Data Science Teams Data Science Team 1 Data Science Teams Data Science Team 2 Data Science Team 3 Data Science Team 4
6. Machine Learning Teams etc.. Machine Leaning Teams Machine Learning Infrastructure Team Machine Learning Development Team Machine Learning Solution Team Machine Leaning Planing Team ɾɾɾ
7. Overview of LINE Official Account
8. LINE Official Account Platform Message Chat External API
9. Broadcast : One of Core Features Broadcast function enable to deliver marketing/branding message to all users who followed the account There are some filtering function, like Estimated demographic, Elapsed days from follow, Managed Audience, etc
10. In-House Services in LINE Have Sent Too Many Messages
11. “User Voice”: Questionnaire User feel receiving many messages from LINE Official Account Is the amount of message from LINE Official Account appropriate ? Is the message from LINE Official Account useful ?
12. Data Analysis Result : User’s Viewpoint More Receiving, Less Opening Users tend to open messages less when receiving them more Horizontal axis : ɹ# of receiving messages per user Vertical axis : ɹupper figure : # of users who received messages ɹlower figure : open rate ( = open / receive messages)
13. Data Analysis Result : Platform’s Viewpoint On the chat list on LINE app, LINE Official Accounts are competing user’s message open Even If OA reduces sending message, Frequently Message Received user’s open rate is still low Horizontal axis : ɹfrequency of OA’s message sending Vertical axis : ɹopen rate Color : ɹfrequency of User’s message receiving
14. Direction Controlling Volume of Messages & Maximize Open Rate
15. Solution Change rules? Relocating Resources ? Raising a price when oversending messages ? Increase the manpower of LINE Official Account Operation ?
16. Solution Change rules? Relocating Resources ? Raising a price when oversending messages ? Predicting “Open Rate” and control the volume of message delivery Increase the manpower of LINE Official Account Operation ?
17. Building & Deploying the Model with OA CMS integration
18. Open Score Generator Core ML Engine
19. Overview ML system, which is assigned a high score to users who are likely to open based on past message delivery history OA User Score Send 1 A 0.7 Yes 1 B 0.3 No 2 B 0.1 No 2 C 0.9 Yes 3 C 0.2 No … … … … Scoring Modeling Open Score Generator
20. Traffic / day 80M+ # MAU (JP) 1B+ # LINE Official Accounts (JP) 0.24M # Account-User pairs (JP) MAU: As of Jun. 2020 Accounts: As of Sep. 2020 pairs; As of Oct. 2020
21. Prediction: source features and target Features Past message delivery history and open/click history, estimated demographic, account category, etc Target Whether the message was “opened” on the message receipt date talk room ͷsample ͕΄͍͠ 100% displayed
22. Data Science utilized Open Score Domain knowledge by Data Scientist Platform viewpoint feature: # days received messages in last 1week ɹɹɹɹ(≒ Frequency) User viewpoint feature: # past requests (each / all OA) Open Score Past Requests (Each OA) vs Score past requests past requests Past Requests (All OA) vs Score Open Score Percentage # days received messages in last 1week High Score User (Score >= X) Low Score User (Score < X)
23. System Architecture
24. Overview OA Dev/B-part/DataLabs Hadoop Cluster Open Score Generator DataLabs Collect Event DataHub Messaging API B-part OA CMS OA Dev Request Import Score Send to Service-side Broadcast Collect Event Data
25. Flow Hadoop Cluster Open Score Generator DataLabs Collect Event DataHub Messaging API B-part OA CMS OA Dev Request Import Score Send to Service-side Broadcast Collect Event Data CMS → Messaging API with Open Score → Targeting broadcast
26. Spec Targeting Option in OA CMS Target User Feature Auto Targeting • High + Low (Sampling) Score User • Easy • Detailed operation is not possible Manual Targeting • High Score User • Low Score User • Hard • Detailed operation is possible
27. Spec Business logic Spec How to handle in targeting New Follower Users who follow the account for less than 7days are not scored High Score User Rare broadcast Users who have not received messages from a account for more than 7days are not scored High Score User Secrecy of Communication Not agreed accounts and users are not scored Not agreed account and users are dealt as high score user
28. Results & Future Prospects
29. Pre-simulation How much the volume of messages will be decreased ? Top 5 Internal Official Account - 43% All Internal Official Account - 18% JP - 24%
30. Results The volume of message decreased via OA CMS, However, increased via Messaging API
31. Whyʁ › API multicast & Manage audience function doesn’t filter user by OST › So, they used API & Manage audience more than before › For the short-term, they can’t achieve the business goal by OST
32. Current Challenge Reduction of wasted messages and growing internal services at the same time
33. Future Prospects
34. Future Prospects › Open Rate & CTR prediction › Contents based Prediction › Expanding the scope of OST
35. OA Score Open Rate & CTR prediction OA User Score Send 1 A 0.7 Yes 1 B 0.3 No 2 B 0.1 No 2 C 0.9 Yes 3 C 0.2 No … … … … Scoring Modeling × ×… Open Score Model Click Score Model
36. Contents based OA Score Contents based Prediction Contents A Contents B Contents C OA CMS Messaging API Register Request Response OA Score Generator Contents A 0.7 Contents B 0.4 Ranking Broadcast Contents C 0.6 Account1 Account2 Account3 Account1 Account3 Account2 A user
37. Thank you