__ɼᰈવ༗ਓ㘸AIෆೳ፤ိ၏ߦ᭖ Data Scientist /Johnson Wu @LINE Taiwan

Hello! - Johnson Wu - CSIE (B01, R05) - Intern @ Microsoft Research Asia - Poetry book writer (indirect) - Data Scientist @ LINE Taiwan 2019 @LINE Fukuoka

Imagination about Digital Marketing in LINE: *OUFSOFU DFMFCSJUZ 4PDJBM NFEJB "EWFSUJTFNFOU

Our actual thought of Marketing: Content & Experience Social & Relationship Commerce & Sales Management Data Advertising & Promotion • Talent Mgt. • Finance Mgt. • Quality Mgt. • CRM • Channels & Partners • Sales Automation & Intelligence • CDP/DMP • Business Intelligence • Data Governance • Customer Exp. • KOL • Community & Review • Content • Experience Mgt. • SEO • Marketing • Advertising • PR Align with user journey!

Look into our problems with… (A)AARRR Funnel ! How to let people visit my content? How to let people have action/buy? How to make people come back and buy? How to make people start to buy? How to let people refer new users to your content? Acquisition Activation Retention Revenue Referral 1.25 2.5 3.75 5

Commercial problems with the AARRR Real Cases in LINE Taiwan How to recommend OA push to users with limited number of pushes? How to attract most possibly new users to the service? How to value customers and Recommend based on values? How to deliver ad promotion with the best cost-profit balance? How do we understand users for AD targeting? Acquisition Activation Retention and activation AARRR Revenue

Using knowledge graph solution for new user acquisition User Acquisition

What are active users in the service? Where should I find the new users? How to connect new users with active users? Acquisition

Users in LINE Correlate users with behaviors of other services -*/& 50%": -*/& 
 4)011*/( -*/& .64*$ ? Acquisition 0"

Attracting New users Via SmartChannel with knowledge graph technique. smart channel And graph? Acquisition

Graph embedding helps Service knowledge graph provides abundant information - GraphSAGE (inductive), GCN - Graph Attention network with deepwalk - Semantic matching based: ComplEX - Algorithm choice depends on data traits https://arxiv.org/pdf/1710.10903.pdf https://arxiv.org/pdf/1706.02216.pdf 1 2 1 2 Acquisition

How does graph work? Model predict the probability of the “link”. - Example: Graph Attention network with deepwalk 1 2 Acquisition - Users are represented as edge connections between entity with meaningful actions - Users1: article1 read -> 
 shop item2 -> join campaign 3…. - Each node (entity) is aggregated 
 by the weighted sum of neighbors.

New acquired users Case study on LINE POINTS wall : how to acquire new users? -*/&10*/54XBMMVTFST OPO-*/&10*/54XBMMVTFST 0 ffi DJBM "DDPVOU GPMMPXJOH TUBUVT ,OPXMFEHF HSBQINPEFMJOH 10*/540" $MVTUFSJOH ,OPXMFEHF HSBQIJOGFSFODF 
 $MVTUFSJOH 0" OPO-*/&10*/54 XBMMVTFST NPTUMJLFMZUP KPJO-*/&SFXBSE Compose triples Compose triples Top k similar users

- Flexible and complex decision making - Applied to LINE SHOPPING, SPOT, POINTS, MUSIC - Embeddings show clustering effect w.r.t campaign Performance Case study on LINE POINTS : effectiveness and fully data orientation - ≥ 25% of new user growth comes from AI’s decision. - Easily fit for different scenario : 
 How about recommend OA Push message to non-new users? - ≥200% CTR lift Acquisition Activation

Apply uplift model to target the persuadable Retention: How to Maximize Campaign Profit?

Who should receive OAP? How many OAP I should deliver? How to evaluate performance? Retention

How to deliver ad promotion with the best cost-profit balance? $PVOUPGDBNQBJHO .POUIPG:FBS 3FDFJWF 0"NFTTBHF 1VSDIBTF JODBNQBJHOQFSJPE &BSO -*/&$0*/10*/54 Boost the Profits with Incentives Retention

User Segments by Uplift High Low Low High Buy if treated Buy if not treated Sure Thing Sleeping Dog Lost Cause The Persuadable Retention

User Segments by Uplift EFMJWFSDPTU JODFOUJWFDPTU High Low Low High Buy if treated Buy if not treated Sure Thing Sleeping Dog Lost Cause The Persuadable Retention

User Segments by Uplift EFMJWFSZDPTU SFWFOVF High Low Low High Buy if treated Buy if not treated Sure Thing Sleeping Dog Lost Cause The Persuadable Retention

User Segments by Uplift EFMJWFSZDPTU High Low High Low Buy if treated Buy if not treated Sure Thing Sleeping Dog Lost Cause The Persuadable Retention

User Segments by Uplift EFMJWFSZDPTU *ODFOUJWFDPTU SFWFOVF High Low Low High Buy if treated Buy if not treated Sure Thing Sleeping Dog Lost Cause The Persuadable Retention

User Segments by Uplift QPTJUJWFVQMJGU OFHBUJWFVQMJGU OPVQMJGU OPVQMJGU Retention High Low Low High Buy if treated Buy if not treated Sure Thing Sleeping Dog Lost Cause The Persuadable Uplift M odel

User Segments by Uplift Retention High Low Low High Buy if treated Buy if not treated Sure Thing Sleeping Dog Lost Cause The Persuadable 3FTQPOTF.PEFM Uplift M odel

A/B Testing to Collect Lookalike User Groups Treatment Group Control Group Retention

#Control Group #Treatment Group Leverage Lookalike Groups to Estimate Uplift

User 360 Degree Features Campaign Period Observation Period, 30 days Purchase or not Purchase or not Browsing behavior Demographics Purchase history OA engagement Ad event Sticker collections Treatment Group Control Group Retention

User 360 Degree Features Campaign Period Observation Period, 30 days Purchase or not Purchase or not Browsing behavior Demographics Purchase history OA engagement Ad event Sticker collections Treatment Group Control Group If the campaign effect is significant If the sure thing exist Retention

Meta-learners, S-Learner Campaign Period Observation Period, 30 days Browsing behavior Demographics Purchase history OA engagement Ad event Sticker collections Purchase probabilities Treatment Group Control Group $MBTTJ fi FS treatment = 1 treatment = 0 Purchase or not ≈ *OUVJUJWFBOEFBTZUPUSBJO -PXWBSJBODFBOE fl FYJCMF Retention

Estimate Individual Treatment Effects Campaign Period Observation Period, 30 days Browsing behavior Demographics Purchase history OA engagement Ad event Sticker collections $MBTTJ fi FS Purchase probability Purchase probability treatment = 1 treatment = 0 candidate Retention

Estimate Individual Treatment Effects Campaign Period Observation Period, 30 days Browsing behavior Demographics Purchase history OA engagement Ad event Sticker collections $MBTTJ fi FS treatment = 1 treatment = 0 Uplift Purchase probability candidate Retention

Uplifts by Deciles 6QMJGU The Persuadable Sure Thing/Lost Cause Sleeping Dog Retention

Reduce Delivery Cost and Mute Rate in the Future 6TFS-JTU %JTDPVOU 6TFS-JTU $PJO1PJOU #BDL 6TFS-JTU -VDLZESBX Day 3 Day 2 Day 1 Campaign Period 4UJDLFS %BUB %FW Retention

Actionable and Explainable User Segmentation from RFM to CLV CRM: How to identify valuable customers?

Know Your Customers F R M Historical Unknown Time Now From Buying Profile Retention Revenue

Measure Your Customer Lifetime Value From RFM to CLV Past Now Future Measure Former Value Predict Future Value RFM CLV Retention Revenue

Measure Your Customer Lifetime Value From RFM to CLV Past Now Future RFM CLV Measure Former Value Predict Retention Revenue

Measure Former Value - RFM Modeling RFM Scores Historical data R F M Segments Retention Revenue

Measure Former Value - RFM Modeling R F M RFM Segments Historical data Clustering Retention Revenue

RFM Modeling - Clustering R F Retention Revenue

RFM Modeling - Clustering R F Retention Revenue

Measure Former Value - RFM Modeling R F M Historical data Clustering Retention Revenue

Measure Your Customer Lifetime Value From RFM to CLV Past Now Future RFM CLV Measure Former Value Predict Future Value Retention Revenue

Predict Future Value - CLV Modeling Demographic Browsing history Buying pro fi le OA interaction Feature Engineering Modeling Tags CLV High ↑ CLV Median CLV Low ↓ { > 50% High valued customers <= 50% Medium valued customers 0, Churned customers 450 days 180 days Now Cascade model Retention Revenue

Predict Future Value - CLV Modeling Feature Engineering Modeling Tags CLV High ↑ CLV Median CLV Low ↓ { > 50% High valued customers <= 50% Medium valued customers 0, Churned customers 450 days 180 days Now User Embedding Demographic Browsing history Buying pro Retention Revenue

Predict Future Value - CLV Modeling User Embedding Time of View Content Views from customers AD 1 AD n Product 1 Article 1 Article n Product n … … … Context window Retention Revenue

Explicit and explainable user segment R↑F↑M↑ R↑F↑M↓ R↑F↓M↑ R↑F↓M↓ R↓F↑M↑ R↓F↑M↓ R↓F↓M↑ R↓F↓M↓ ࠷ۙ༗ိత౔߽ৗ٬ ࠷ۙ༗ိతখࢿৗ٬ ࠷ۙ༗ိత౔߽ك٬ ࠷ۙ༗ိతখࢿك٬ ফࣦత౔߽ৗ٬ ফࣦతখࢿৗ٬ ফࣦత౔߽ك٬ ফࣦతখࢿك٬ CLV Low CLV Median CLV High ະိߴফඅ ະိதফඅ ະိෆফඅ RFM CLV Deliverables: Tags Retention Revenue

೗Կར༻໛ܕඪតʁ R↑F↑M↓ Upselling ! RFM RFM 最近有來的⼩資常客 R↑F↑M↑ RFM 最近有來的⼟豪常客 Retention Revenue

R↑F↑M↑ CLV Low Retain ! ೗Կར༻໛ܕඪតʁ RFM + CLV RFM CLV 最近有來的⼟豪常客 未來不消費 Retention Revenue

How to adopt CLV service? Overview of CLV Service Retention Revenue

Wait… is there…

Any model that could handle everything? A good start of user understanding ! #MBDLCPY Who among them are more likely to come? Which content is more likely be click by whom? Who among them are more likely persuadable? How to estimate user values? How to refer friends of users to us?

Foundation models Pertaining on user behaviors, which come from our family services log! Click prediction User segment prediction Sales/revenue prediction

Combined with MLOps: 
 Scalable Automated MarTech Solution Data Source %BUB 8BSFIPVTF Data Integration 'FBUVSF4UPSF Application "DRVJSFOFXVTFS "%5BSHFUJOH #PPTUSFWFOVF *NQSPWF3FUFOUJPO … ,(4PMVUJPO1JQFMJOF $-74PMVUJPO1JQFMJOF 6QMJGU 
 4PMVUJPO1JQFMJOF AI Solution 6TFSTFHNFOUBUJPO 
 4PMVUJPO1JQFMJOF … LINE Family Services Demographic Log

To sum up Swifter Scalable and Flexible Solution Faster Speed of Delivery Data-oriented Decision Making AI solves (almost) every Martech problem Developer Marketer YOU

Thank you 資料科學家大解密 - LINE資料工程部門介紹 - 資料科學生活 - 職場眉眉角角 Closing the Distance Share easily with friends in LINE Timeline

Thank you Closing the Distance Official Account Ads 圖⽚來源︓https://hub.line.me/

Thank you Closing the Distance Official Account Ads 圖⽚來源︓https://hub.line.me/ 如何推薦用戶會感興趣的商家？ 如何確保新聞品質？ 如何判斷假新聞? 如何判斷潛在的訂閱制流失戶？

Thank you AI-enable Applications Business Intelligence Data Dev LINE Family Services LINE TODAY LINE SHOPPING LINE SPOT LINE MUSIC LINE Sticker LINE VOOM LINE Reward Official Account Fact Checker LINE HELP TW LINE Travel Ads 獨立的資料工程部門，提供資料科學解決方案 LINE TODAY

Thank you Data Dev LINE Family Services LINE SHOPPING LINE SPOT LINE MUSIC LINE Sticker LINE VOOM LINE Reward Fact Checker LINE HELP TW LINE Travel NLP Knowledg e Graph Uplift Modeling NER Classifier Duplication Detector Auto completion Keyword Extraction Related Search Text Generation User Tagging Data Analytics Recom- mendation CLV 從報表、分析洞見到預測模型 LINE TODAY

Thank you 成員組成 • Build and optimize da ta pipeline architectur e • Assemble large, com plex data sets that m eet requirements Data Engineer Data Analyst Big data infra, SQL, ET L, message queuing • Interpret data, analyz e results using statisti cal techniques • Identify, analyze, and interpret trends or pat terns in complex data sets Statistics, Data Visualiz ation, Business Knowle dge SKILL RESPONSIBILITY • Select appropriate da tasets and data repre sentation methods • Research and imple ment appropriate ML algorithms Data Scientist Machine learning, deep learning, CV, NLP, Spe ech ML Svc Engineer • Build and scale mach ine learning infrastruc ture • Monitor model perfor mance System infrastructure d esign, DevOps

Thank you Subtitle 成員組成 • Build and optimize da ta pipeline architectur e • Assemble large, com plex data sets that m eet requirements Data Engineer Data Analyst Big data infra, SQL, ET L, message queuing • Interpret data, analyz e results using statisti cal techniques • Identify, analyze, and interpret trends or pat terns in complex data sets Statistics, Data Visualiz ation, Business Knowle dge SKILL RESPONSIBILITY Pipeline Biz • Select appropriate da tasets and data repre sentation methods • Research and imple ment appropriate ML algorithms Data Scientist Machine learning, deep learning, CV, NLP, Spe ech Model ML Svc Engineer • Build and scale mach ine learning infrastruc ture • Monitor model perfor mance System infrastructure d esign, DevOps Service

Thank you