Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Recommender system 3.0: Bias, Graph, and Causality

Recommender system 3.0: Bias, Graph, and Causality

Eebedc2ee7ff95ffb9d9102c6d4a065c?s=128

LINE DevDay 2020

November 26, 2020
Tweet

Transcript

  1. None
  2. Agenda › Introduction to RecSys 3.0 › Positional Bias Reduction

    › Graph › Counterfactual Evaluation
  3. 1. Introduction to RecSys 3.0

  4. Evolution of Personal Recommender System 2016~ Causal Inference 2006 Netflix

    Prize Hybrid Approach 2017~ Graph Neural Networks 2010~ Convolutional Neural Networks Recurrent Neural Networks 1997 Collaborative Filtering 2020 Current Phase1 Phase2 Phase3
  5. RecSys 1.0 RecSys 2.0 RecSys 3.0 Data interpretability low low

    high Data size small big big Calculation complexity low high high Three Stages of Recommender System
  6. Based on Understanding of Data Generation Process, Big Data Manipulation

    is Important in RecSys 3.0 Graph Construction Positional Bias Reduction From Statistic To Causality
  7. Use Case: LINE Wallet

  8. LINE Wallet Tab Shortcuts LINE Pay Modules with different content

    types Coupon MyCard Point Chirashi (Flyer) Examples of modules
  9. 2. Positional Bias Reduction

  10. Predict Which Module to Use Next Prob. of next use

    80% Coupon MyCard Point Chirashi Four modules How to determine sequence of modules? Prob. of next use 95%
  11. How to Predict? - Behavioral History-based Prediction Algorithm -

  12. Collect Behavioral History Data of Users Behavioral history in Wallet

    tab (click/view) Similar users
  13. Feed Data to Learning Models Transformer

  14. Feed Data to Learning Models Transformer Coupon 70%

  15. This is Not End

  16. Common Problem in Recommender System

  17. There is Underlying Secret in Our Log Click log Learn

    a prediction model from behavioral / click log When given an user, Behavioral log
  18. Wallet Users Tend to Click the First Module Once in

    Two Regardless of Its Content 1 2 3 4 50% 20% 15% 15%
  19. Current Position Reinforces The Prob. of Its Next Position Many

    click log Less click log “Likely to be placed at the top next time” “Likely to be placed at the bottom next time”
  20. If you don’t compensate for the positional advantage, Better module

    order might be missed
  21. loss function = cross entropy * module CTR

  22. Compensation Makes The Module CTR Higher After Before 62% 22%

  23. So, How Much Does CTR Increase?

  24. Wallet Module CTR Relative CTR Gain

  25. Wallet Module CTR 2/19 12% Traffic Allocation 3/31 100% Traffic

    Allocation
  26. Japan, Taiwan, Thailand THAILAND JAPAN TAIWAN UU CTR + 37.8%

    Unique User CTR = # "# $%&'() $*)+* ,-" ./&01)2 # "# $%&'() $*)+* ,-" 3&),)2 UU CTR + 255.0% UU CTR + 62.5%
  27. Manual Placement vs Automatic Placement Which One is Better?

  28. What Happened to Wallet Tab? Sudden decrease of click count

    After 4/16
  29. Manual Placement of New Module on The First Position New

    module (ShoppingStore)
  30. When We Move Down The New Module.. Restoration of click

    count after 4/22
  31. Until Today

  32. Good Module, Bad Module? And Need of Item Recommendation

  33. Due to The CoVID-19 Outbreak # of Coupon users #

    of MyCard users People don’t go outside
  34. # of Chirashi Users Also Decreases # of Chirashi users

  35. However, # of Points Users Increases # of Points users

    You can use it at home J
  36. Lesson Learned? Good Points, Bad Chirashi? 99% users receive Points

    at the top position, 86% users receive Chirashi at the bottom position
  37. Item Recommendation Should Be Done Together to Activate Other Modules

  38. 3. Graph

  39. LINE Coupon 500+ items, 30+ brands

  40. Start of Coupon Recommendation Android & iOS 2020.05 ~

  41. Personal Recommendation Use Log Recommendation Bread Lover Bugger Lover

  42. Deep Learning-based Recommender System Click History Demographics . . .

    Image Category . . . User !" Coupon !# $"# = & & + ()*(−)- .)/ ) User Coupon
  43. Cold Start Problem Item 1 *UFN *UFN *UFN User 1

    1 1 0 ? 6TFS 2 0 1 0 ? 6TFS 3 1 0 0 ? 6TFS 4 1 0 1 ? 6TFS 5 ? ? ? ?
  44. Connect Every Log into Graph! Use logs of 70 days

  45. Graph-based Recommender System

  46. Learn a Connection ? Who am I? Friend Family Class

    mate
  47. Adapt More Effectively To New Users and Coupons Day 1

    Day 2 Day 3
  48. CTR and Diversity: Chase Two Rabbits Simultaneously

  49. Past Popularity Density of Recommendation Results 32% 30% 15% 14%

    5% 2% 1% 1% 29% 19% 16% 14% 8% 5% 4% 4% Past Popularity top50 Non Graph-based Model Graph-based Model top1
  50. Recommendation Performance - Diversity Novelty Negative Average Popularity Diversity 2.21%

    9.24% 9.20% Relative CTR Gain against Non Graph Model
  51. Recommendation Performance - CTR Graph model Non Graph model 46%

    34% Relative CTR Gain against Top Popular Baseline
  52. Cold Start Performance – CTR on Cold User Graph model

    Non Graph model 9% 3% Relative CTR Gain against Top Popular Baseline
  53. References [1] Tripartite Heterogeneous Graph Propagation for Large-scale Social Recommendation,

    RecSys 2019 Late-Breaking Result [2] Graphs, Entities, and Step Mixture, ICML 2020 Workshop on Graph Representation Learning and Beyond [3] Hop Sampling: A Simple Regularized Graph Learning for Non-Stationary Environments, KDD 2020 Workshop on Mining and Learning With Graphs [5] div2vec: Diversity-Emphasized Node Embedding, RecSys 2020 Workshop on the Impact of Recommender Systems [4] Multi-Manifold Learning for Large-scale Targeted Advertising System, KDD 2020 Workshop, AdKDD
  54. 4. Counterfactual Evaluation

  55. How Did We Know That The Proposed Models Will Work

    Well?
  56. Basically, Offline Metrics (ex> Precision) != Online Metrics (ex> CTR)

  57. Counterfactual Evaluation: We Can Estimate Model’s Online Performance In Offline

    Test CTR IPS1 IPS2 IPS3 IPS4 IPS5 CAB Model 1 1 4 3 4 2 2 1 Model 2 2 5 4 5 3 3 3 Model 3 3 3 5 3 5 5 5 Model 4 4 2 1 2 1 1 2 Model 5 5 1 2 1 4 4 4 Correlation -0.8 -0.2 -0.8 0.2 0.2 0.4 Offline evaluation metrics Rank
  58. Our Future Works: AutoML - Toward Zero Management Cost -

  59. Thank you