Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Too many LINE Official Account messages? Call data science!

Too many LINE Official Account messages? Call data science!

Yoshiaki Nishite
LINE Data Science Team4 Senior Data Scientist
Takahiro Yoshinaga
LINE Machine Learning Solution Team Senior Data Scientist / Machine Learning Engineer
https://linedevday.linecorp.com/2020/ja/sessions/0816
https://linedevday.linecorp.com/2020/en/sessions/0816

Eebedc2ee7ff95ffb9d9102c6d4a065c?s=128

LINE DevDay 2020

November 27, 2020
Tweet

Transcript

  1. None
  2. Agenda › Introduction : DataLabs › Background & Direction ›

    Building & Deploying the Model › Results & Future Prospects
  3. Introduction Data Labs

  4. Organization Data Science Teams Data Platform Data Science And Engineering

    Center Data Management Engineering Infrastructure LPML Machine Learning Teams Data Labs
  5. Data Science Teams Data Science Team 1 Data Science Teams

    Data Science Team 2 Data Science Team 3 Data Science Team 4
  6. Machine Learning Teams etc.. Machine Leaning Teams Machine Learning Infrastructure

    Team Machine Learning Development Team Machine Learning Solution Team Machine Leaning Planing Team ɾɾɾ
  7. Overview of LINE Official Account

  8. LINE Official Account Platform Message Chat External API

  9. Broadcast : One of Core Features Broadcast function enable to

    deliver marketing/branding message to all users who followed the account There are some filtering function, like Estimated demographic, Elapsed days from follow, Managed Audience, etc
  10. In-House Services in LINE Have Sent Too Many Messages

  11. “User Voice”: Questionnaire User feel receiving many messages from LINE

    Official Account Is the amount of message from LINE Official Account appropriate ? Is the message from LINE Official Account useful ?
  12. Data Analysis Result : User’s Viewpoint More Receiving, Less Opening

    Users tend to open messages less when receiving them more Horizontal axis : ɹ# of receiving messages per user Vertical axis : ɹupper figure : # of users who received messages ɹlower figure : open rate ( = open / receive messages)
  13. Data Analysis Result : Platform’s Viewpoint On the chat list

    on LINE app, LINE Official Accounts are competing user’s message open Even If OA reduces sending message, Frequently Message Received user’s open rate is still low Horizontal axis : ɹfrequency of OA’s message sending Vertical axis : ɹopen rate Color : ɹfrequency of User’s message receiving
  14. Direction Controlling Volume of Messages & Maximize Open Rate

  15. Solution Change rules? Relocating Resources ? Raising a price when

    oversending messages ? Increase the manpower of LINE Official Account Operation ?
  16. Solution Change rules? Relocating Resources ? Raising a price when

    oversending messages ? Predicting “Open Rate” and control the volume of message delivery Increase the manpower of LINE Official Account Operation ?
  17. Building & Deploying the Model with OA CMS integration

  18. Open Score Generator Core ML Engine

  19. Overview ML system, which is assigned a high score to

    users who are likely to open based on past message delivery history OA User Score Send 1 A 0.7 Yes 1 B 0.3 No 2 B 0.1 No 2 C 0.9 Yes 3 C 0.2 No … … … … Scoring Modeling Open Score Generator
  20. Traffic / day 80M+ # MAU (JP) 1B+ # LINE

    Official Accounts (JP) 0.24M # Account-User pairs (JP) MAU: As of Jun. 2020 Accounts: As of Sep. 2020 pairs; As of Oct. 2020
  21. Prediction: source features and target Features Past message delivery history

    and open/click history, estimated demographic, account category, etc Target Whether the message was “opened” on the message receipt date talk room ͷsample ͕΄͍͠ 100% displayed
  22. Data Science utilized Open Score Domain knowledge by Data Scientist

    Platform viewpoint feature: # days received messages in last 1week ɹɹɹɹ(≒ Frequency) User viewpoint feature: # past requests (each / all OA) Open Score Past Requests (Each OA) vs Score past requests past requests Past Requests (All OA) vs Score Open Score Percentage # days received messages in last 1week High Score User (Score >= X) Low Score User (Score < X)
  23. System Architecture

  24. Overview OA Dev/B-part/DataLabs Hadoop Cluster Open Score Generator DataLabs Collect

    Event DataHub Messaging API B-part OA CMS OA Dev Request Import Score Send to Service-side Broadcast Collect Event Data
  25. Flow Hadoop Cluster Open Score Generator DataLabs Collect Event DataHub

    Messaging API B-part OA CMS OA Dev Request Import Score Send to Service-side Broadcast Collect Event Data CMS → Messaging API with Open Score → Targeting broadcast
  26. Spec Targeting Option in OA CMS Target User Feature Auto

    Targeting • High + Low (Sampling) Score User • Easy • Detailed operation is not possible Manual Targeting • High Score User • Low Score User • Hard • Detailed operation is possible
  27. Spec Business logic Spec How to handle in targeting New

    Follower Users who follow the account for less than 7days are not scored High Score User Rare broadcast Users who have not received messages from a account for more than 7days are not scored High Score User Secrecy of Communication Not agreed accounts and users are not scored Not agreed account and users are dealt as high score user
  28. Results & Future Prospects

  29. Pre-simulation How much the volume of messages will be decreased

    ? Top 5 Internal Official Account - 43% All Internal Official Account - 18% JP - 24%
  30. Results The volume of message decreased via OA CMS, However,

    increased via Messaging API
  31. Whyʁ › API multicast & Manage audience function doesn’t filter

    user by OST › So, they used API & Manage audience more than before › For the short-term, they can’t achieve the business goal by OST
  32. Current Challenge Reduction of wasted messages and growing internal services

    at the same time
  33. Future Prospects

  34. Future Prospects › Open Rate & CTR prediction › Contents

    based Prediction › Expanding the scope of OST
  35. OA Score Open Rate & CTR prediction OA User Score

    Send 1 A 0.7 Yes 1 B 0.3 No 2 B 0.1 No 2 C 0.9 Yes 3 C 0.2 No … … … … Scoring Modeling × ×… Open Score Model Click Score Model
  36. Contents based OA Score Contents based Prediction Contents A Contents

    B Contents C OA CMS Messaging API Register Request Response OA Score Generator Contents A 0.7 Contents B 0.4 Ranking Broadcast Contents C 0.6 Account1 Account2 Account3 Account1 Account3 Account2 A user
  37. Thank you