No content

Agenda › Introduction : DataLabs › Background & Direction › Building & Deploying the Model › Results & Future Prospects

Introduction Data Labs

Organization Data Science Teams Data Platform Data Science And Engineering Center Data Management Engineering Infrastructure LPML Machine Learning Teams Data Labs

Data Science Teams Data Science Team 1 Data Science Teams Data Science Team 2 Data Science Team 3 Data Science Team 4

Machine Learning Teams etc.. Machine Leaning Teams Machine Learning Infrastructure Team Machine Learning Development Team Machine Learning Solution Team Machine Leaning Planing Team ɾɾɾ

Overview of LINE Official Account

LINE Official Account Platform Message Chat External API

Broadcast : One of Core Features Broadcast function enable to deliver marketing/branding message to all users who followed the account There are some filtering function, like Estimated demographic, Elapsed days from follow, Managed Audience, etc

In-House Services in LINE Have Sent Too Many Messages

“User Voice”: Questionnaire User feel receiving many messages from LINE Official Account Is the amount of message from LINE Official Account appropriate ? Is the message from LINE Official Account useful ?

Data Analysis Result : User’s Viewpoint More Receiving, Less Opening Users tend to open messages less when receiving them more Horizontal axis : ɹ# of receiving messages per user Vertical axis : ɹupper figure : # of users who received messages ɹlower figure : open rate ( = open / receive messages)

Data Analysis Result : Platform’s Viewpoint On the chat list on LINE app, LINE Official Accounts are competing user’s message open Even If OA reduces sending message, Frequently Message Received user’s open rate is still low Horizontal axis : ɹfrequency of OA’s message sending Vertical axis : ɹopen rate Color : ɹfrequency of User’s message receiving

Direction Controlling Volume of Messages & Maximize Open Rate

Solution Change rules? Relocating Resources ? Raising a price when oversending messages ? Increase the manpower of LINE Official Account Operation ?

Solution Change rules? Relocating Resources ? Raising a price when oversending messages ? Predicting “Open Rate” and control the volume of message delivery Increase the manpower of LINE Official Account Operation ?

Building & Deploying the Model with OA CMS integration

Open Score Generator Core ML Engine

Overview ML system, which is assigned a high score to users who are likely to open based on past message delivery history OA User Score Send 1 A 0.7 Yes 1 B 0.3 No 2 B 0.1 No 2 C 0.9 Yes 3 C 0.2 No … … … … Scoring Modeling Open Score Generator

Traffic / day 80M+ # MAU (JP) 1B+ # LINE Official Accounts (JP) 0.24M # Account-User pairs (JP) MAU: As of Jun. 2020 Accounts: As of Sep. 2020 pairs; As of Oct. 2020

Prediction: source features and target Features Past message delivery history and open/click history, estimated demographic, account category, etc Target Whether the message was “opened” on the message receipt date talk room ͷsample ͕΄͍͠ 100% displayed

Data Science utilized Open Score Domain knowledge by Data Scientist Platform viewpoint feature: # days received messages in last 1week ɹɹɹɹ(≒ Frequency) User viewpoint feature: # past requests (each / all OA) Open Score Past Requests (Each OA) vs Score past requests past requests Past Requests (All OA) vs Score Open Score Percentage # days received messages in last 1week High Score User (Score >= X) Low Score User (Score < X)

System Architecture

Overview OA Dev/B-part/DataLabs Hadoop Cluster Open Score Generator DataLabs Collect Event DataHub Messaging API B-part OA CMS OA Dev Request Import Score Send to Service-side Broadcast Collect Event Data

Flow Hadoop Cluster Open Score Generator DataLabs Collect Event DataHub Messaging API B-part OA CMS OA Dev Request Import Score Send to Service-side Broadcast Collect Event Data CMS → Messaging API with Open Score → Targeting broadcast

Spec Targeting Option in OA CMS Target User Feature Auto Targeting • High + Low (Sampling) Score User • Easy • Detailed operation is not possible Manual Targeting • High Score User • Low Score User • Hard • Detailed operation is possible

Spec Business logic Spec How to handle in targeting New Follower Users who follow the account for less than 7days are not scored High Score User Rare broadcast Users who have not received messages from a account for more than 7days are not scored High Score User Secrecy of Communication Not agreed accounts and users are not scored Not agreed account and users are dealt as high score user

Results & Future Prospects

Pre-simulation How much the volume of messages will be decreased ? Top 5 Internal Official Account - 43% All Internal Official Account - 18% JP - 24%

Results The volume of message decreased via OA CMS, However, increased via Messaging API

Whyʁ › API multicast & Manage audience function doesn’t filter user by OST › So, they used API & Manage audience more than before › For the short-term, they can’t achieve the business goal by OST

Current Challenge Reduction of wasted messages and growing internal services at the same time

Future Prospects

Future Prospects › Open Rate & CTR prediction › Contents based Prediction › Expanding the scope of OST

OA Score Open Rate & CTR prediction OA User Score Send 1 A 0.7 Yes 1 B 0.3 No 2 B 0.1 No 2 C 0.9 Yes 3 C 0.2 No … … … … Scoring Modeling × ×… Open Score Model Click Score Model

Contents based OA Score Contents based Prediction Contents A Contents B Contents C OA CMS Messaging API Register Request Response OA Score Generator Contents A 0.7 Contents B 0.4 Ranking Broadcast Contents C 0.6 Account1 Account2 Account3 Account1 Account3 Account2 A user

Thank you