Recommender system 3.0: Bias, Graph, and Causality

Kyung-Min Kim
LINE AI Business KR Team AI Researcher
https://linedevday.linecorp.com/2020/jp/sessions/2647
https://linedevday.linecorp.com/2020/en/sessions/2647

Transcript

2. Agenda › Introduction to RecSys 3.0 › Positional Bias Reduction › Graph › Counterfactual Evaluation
3. 1. Introduction to RecSys 3.0
4. Evolution of Personal Recommender System 2016~ Causal Inference 2006 Netflix Prize Hybrid Approach 2017~ Graph Neural Networks 2010~ Convolutional Neural Networks Recurrent Neural Networks 1997 Collaborative Filtering 2020 Current Phase1 Phase2 Phase3
5. RecSys 1.0 RecSys 2.0 RecSys 3.0 Data interpretability low low high Data size small big big Calculation complexity low high high Three Stages of Recommender System
6. Based on Understanding of Data Generation Process, Big Data Manipulation is Important in RecSys 3.0 Graph Construction Positional Bias Reduction From Statistic To Causality
7. Use Case: LINE Wallet
8. LINE Wallet Tab Shortcuts LINE Pay Modules with different content types Coupon MyCard Point Chirashi (Flyer) Examples of modules
9. 2. Positional Bias Reduction
10. Predict Which Module to Use Next Prob. of next use 80% Coupon MyCard Point Chirashi Four modules How to determine sequence of modules? Prob. of next use 95%
11. How to Predict? - Behavioral History-based Prediction Algorithm -
12. Collect Behavioral History Data of Users Behavioral history in Wallet tab (click/view) Similar users
13. Feed Data to Learning Models Transformer
14. Feed Data to Learning Models Transformer Coupon 70%
15. This is Not End
16. Common Problem in Recommender System
17. There is Underlying Secret in Our Log Click log Learn a prediction model from behavioral / click log When given an user, Behavioral log
18. Wallet Users Tend to Click the First Module Once in Two Regardless of Its Content 1 2 3 4 50% 20% 15% 15%
19. Current Position Reinforces The Prob. of Its Next Position Many click log Less click log “Likely to be placed at the top next time” “Likely to be placed at the bottom next time”
20. If you don’t compensate for the positional advantage, Better module order might be missed
21. loss function = cross entropy * module CTR
22. Compensation Makes The Module CTR Higher After Before 62% 22%
23. So, How Much Does CTR Increase?
24. Wallet Module CTR Relative CTR Gain
25. Wallet Module CTR 2/19 12% Traffic Allocation 3/31 100% Traffic Allocation
26. Japan, Taiwan, Thailand THAILAND JAPAN TAIWAN UU CTR + 37.8% Unique User CTR = # "# $%&'() $*)+* ,-" ./&01)2 # "# $%&'() $*)+* ,-" 3&),)2 UU CTR + 255.0% UU CTR + 62.5%
27. Manual Placement vs Automatic Placement Which One is Better?
28. What Happened to Wallet Tab? Sudden decrease of click count After 4/16
29. Manual Placement of New Module on The First Position New module (ShoppingStore)
30. When We Move Down The New Module.. Restoration of click count after 4/22
31. Until Today
32. Good Module, Bad Module? And Need of Item Recommendation
33. Due to The CoVID-19 Outbreak # of Coupon users # of MyCard users People don’t go outside
34. # of Chirashi Users Also Decreases # of Chirashi users
35. However, # of Points Users Increases # of Points users You can use it at home J
36. Lesson Learned? Good Points, Bad Chirashi? 99% users receive Points at the top position, 86% users receive Chirashi at the bottom position
37. Item Recommendation Should Be Done Together to Activate Other Modules
38. 3. Graph
39. LINE Coupon 500+ items, 30+ brands
40. Start of Coupon Recommendation Android & iOS 2020.05 ~
41. Personal Recommendation Use Log Recommendation Bread Lover Bugger Lover
42. Deep Learning-based Recommender System Click History Demographics . . . Image Category . . . User !" Coupon !# $"# = & & + ()*(−)- .)/ ) User Coupon
43. Cold Start Problem Item 1 *UFN
     *UFN
     *UFN
     User 1 1 1 0 ? 6TFS 2 0 1 0 ? 6TFS 3 1 0 0 ? 6TFS 4 1 0 1 ? 6TFS 5 ? ? ? ?
44. Connect Every Log into Graph! Use logs of 70 days
45. Graph-based Recommender System
46. Learn a Connection ? Who am I? Friend Family Class mate
47. Adapt More Effectively To New Users and Coupons Day 1 Day 2 Day 3
48. CTR and Diversity: Chase Two Rabbits Simultaneously
49. Past Popularity Density of Recommendation Results 32% 30% 15% 14% 5% 2% 1% 1% 29% 19% 16% 14% 8% 5% 4% 4% Past Popularity top50 Non Graph-based Model Graph-based Model top1
50. Recommendation Performance - Diversity Novelty Negative Average Popularity Diversity 2.21% 9.24% 9.20% Relative CTR Gain against Non Graph Model
51. Recommendation Performance - CTR Graph model Non Graph model 46% 34% Relative CTR Gain against Top Popular Baseline
52. Cold Start Performance – CTR on Cold User Graph model Non Graph model 9% 3% Relative CTR Gain against Top Popular Baseline
53. References [1] Tripartite Heterogeneous Graph Propagation for Large-scale Social Recommendation, RecSys 2019 Late-Breaking Result [2] Graphs, Entities, and Step Mixture, ICML 2020 Workshop on Graph Representation Learning and Beyond [3] Hop Sampling: A Simple Regularized Graph Learning for Non-Stationary Environments, KDD 2020 Workshop on Mining and Learning With Graphs [5] div2vec: Diversity-Emphasized Node Embedding, RecSys 2020 Workshop on the Impact of Recommender Systems [4] Multi-Manifold Learning for Large-scale Targeted Advertising System, KDD 2020 Workshop, AdKDD
54. 4. Counterfactual Evaluation
55. How Did We Know That The Proposed Models Will Work Well?
56. Basically, Offline Metrics (ex> Precision) != Online Metrics (ex> CTR)
57. Counterfactual Evaluation: We Can Estimate Model’s Online Performance In Offline Test CTR IPS1 IPS2 IPS3 IPS4 IPS5 CAB Model 1 1 4 3 4 2 2 1 Model 2 2 5 4 5 3 3 3 Model 3 3 3 5 3 5 5 5 Model 4 4 2 1 2 1 1 2 Model 5 5 1 2 1 4 4 4 Correlation -0.8 -0.2 -0.8 0.2 0.2 0.4 Offline evaluation metrics Rank
58. Our Future Works: AutoML - Toward Zero Management Cost -
59. Thank you