KDD 2020 Marketplace Tutorial Rishabh Part 4

1. KDD 2020 Tutorial: Advances in Recommender Systems Part A: Recommendations in a Marketplace [User & Content Understanding] Rishabh Mehrotra Ben Carterette Senior Research Scientist, Senior Research Manager, Spotify, London Spotify, New York rishabhm@spotify.com benjaminc@spotify.com 23rd August 2020 @erishabh @BenCarterette https://sites.google.com/view/kdd20-marketplace-autorecsys/
2. Recommendations in a Marketplace ❏ Stakeholders & Objectives
3. Recommendations in a Marketplace ❏ Stakeholders & Objectives ❏ Methods for Multi-Objective Recommendations
4. Recommendations in a Marketplace ❏ Stakeholders & Objectives ❏ Methods for Multi-Objective Recommendations ❏ Leveraging User & Supplier Understanding
5. Summary: Part I Introduction to Marketplaces - Traditional RecSys ML catered towards user-centric modeling - Multiple stakeholders in online marketplaces - Need to consider multiple objectives + ML models to optimize those objectives
6. Summary: Part II Stakeholders & Objectives - Multiple stakeholders in online marketplaces - different industrial case-studies - UberEats, Postmates, Etsy, AirBnb, Music, P2P lending, Crowdfunding - Multiple, often conflicting objectives - +vely correlated, neutral, -vely correlated - ML methods needed to model the interplay between objectives - Important to carefully decide what a system optimizes for
7. Summary: Part III Multi-Objective Methods for Recommendations - Flavors of multi-objective approaches available: - Multi-task learning - Scalarization - MO-Multi task learning - MO-Bandits & MO-RL - Often optimizing for multiple interaction metrics performs better for each metric than directly optimizing that metric - Not necessarily a Zero-Sum Game
8. Part IV Leveraging User, Supplier & Content Understanding Algorithmic balancing
9. Outline 1. Introduction to Marketplaces 2. Optimization Objectives in a Marketplace 3. Methods for Multi-Objective Ranking & Recommendations 4. Leveraging Consumer, Supplier & Content Understanding a. Consumer Understanding: i. Consumption diversity of users ii. Leveraging User intents iii. Quantifying and estimating user receptivity b. Supplier Understanding: i. Diversity across suppliers c. Content Understanding: i. Personalizing Reward function ii. Query Understanding 5. Industrial Applications
10. Consumption Diversity Understanding Consumption Algorithmic Effects on the Diversity of Consumption on Spotify. Ashton Anderson, Lucas Maystre, Ian Anderson, Rishabh Mehrotra, Mounia Lalmas
11. Generalist-Specialist Score (GS) Specialist Generalist Generalist Specialist Consumption Diversity: Differences across Users
12. Generalist-Specialist Score (GS) Specialist Generalist Generalist Specialist Consumption Diversity: Differences across Users Generalists & specialists exhibit different retention & conversion behaviors
13. User Intents Intent aware models
14. User Intents for Targeting Intent aware satisfaction prediction models
15. Understanding User Intents helps! Intent aware satisfaction prediction models → Multi-level models across different intents User Intents for Targeting
16. Understanding User Intents helps! Intent aware satisfaction prediction models → Multi-level models across different intents Considering intent information is crucial → 20% improvement in SAT prediction over global model User Intents for Targeting
17. User Receptivity How receptive are users?
18. User Receptivity to Divergent Recommendations Leveraged randomized data to quantify & predict user receptivity
19. User Receptivity to Divergent Recommendations Leveraged randomized data to quantify & predict user receptivity Three ways of quantifying tolerance: 1. Engagement centric 2. Effort centric 3. Emendation centric (stickiness)
20. User Receptivity to Divergent Recommendations Leveraged randomized data to quantify & predict user receptivity Three ways of quantifying tolerance: 1. Engagement centric 2. Effort centric 3. Emendation centric (stickiness) Used user features to train predictive model
21. User tolerance for trade-off decisioning - Different users are receptive to different extent - inactive users vs heavy use users User Receptivity to Divergent Recommendations
22. User tolerance for trade-off decisioning - Different users are receptive to different extent - inactive users vs heavy use users - Identified predictive features for user receptivity - Consumption diversity is NOT predictive of engagement receptivity User Receptivity to Divergent Recommendations
23. User tolerance for trade-off decisioning - Different users are receptive to different extent - inactive users vs heavy use users - Identified predictive features for user receptivity - Consumption diversity is NOT predictive of engagement receptivity - Get 70% accuracy in predicting Engagement tolerance - Receptivity predictive of future engagement User Receptivity to Divergent Recommendations
24. Part IV Leveraging User, Supplier & Content Understanding Algorithmic balancing User Understanding Implications: - Users have different consumption diversity - Consider user level heterogeneity - Identifying user intents helps better target content - perhaps esp for content from other stakeholder objectives - Understanding user receptivity helps in avoid user dissatisfaction
25. Outline 1. Introduction to Marketplaces 2. Optimization Objectives in a Marketplace 3. Methods for Multi-Objective Ranking & Recommendations 4. Leveraging Consumer, Supplier & Content Understanding a. Consumer Understanding: i. Consumption diversity of users ii. Leveraging User intents iii. Quantifying and estimating user receptivity b. Supplier Understanding: i. Diversity across suppliers ii. Spillover effect across suppliers c. Content Understanding: i. Personalizing Reward function ii. Query Understanding 5. Industrial Applications
26. Supplier Diversity Trade-off between Relevance, Diversity & Satisfaction Towards a Fair Marketplace: Counterfactual Evaluation of the trade-off between Relevance, Fairness & Satisfaction in Recommendation Systems. Rishabh Mehrotra, James McInerney, Hugues Bouchard, Mounia Lalmas, Fernando Diaz CIKM 2018
27. Let’s consider 2 stakeholders Stakeholders Relevance Exposure Fairness
28. User Relevance vs Artist Diversity
29. Relevance vs Fairness Very few sets have both high relevance & high fairness User Relevance vs Artist Diversity
30. How do we trade-off User Relevance vs Supplier Diversity?
31. Summary of Recommendation Policies Policy I: Optimizing Relevance Policy II: Optimizing Fairness Policy III: Probabilistic Policy Policy IV: Trade-off Relevance & Fairness Policy V: Guaranteed Relevance Policy VI: Adaptive Policy I Policy VI: Adaptive Policy II
32. Balance between Relevance & Artist Diversity Diversity Relevance Inflection points of good trade-off
33. diversity relevance Balance between Relevance & Artist Diversity % loss in diversity % loss in relevance % gain in satisfaction optimizing relevance (β=1) 69.1 0 0 optimizing diversity (β=0) 0 57.7 -32.2 trade-off relevance & diversity (β =0.7) 42.7 9.8 -10.2 guaranteed relevance (rel >= 0.7) 51.7 7.8 4.4 diversity affinity aware 15 21.2 12.1 Diversity affinity aware provides the best overall trade-off → personalizing diversity
34. Spillover Effect Related suppliers benefits from events on focal suppliers Inferring the Causal Impact of New Track Releases on Music Recommendation Platforms through Counterfactual Predictions Rishabh Mehrotra, Prasanta Bhattacharya, Mounia Lalmas RecSys 2020 (LBR)
35. Understanding User Intents helps! Spillover Effects across Suppliers • New track releases help not only the focal artist but also other related artists ◦ increased tendency of users to explore and discover related music
36. Understanding User Intents helps! Spillover Effects across Suppliers • New track releases help not only the focal artist but also other related artists ◦ increased tendency of users to explore and discover related music • Other artists benefit by virtue of being on the platform
37. Understanding User Intents helps! Spillover Effects across Suppliers • New track releases help not only the focal artist but also other related artists ◦ increased tendency of users to explore and discover related music • Other artists benefit by virtue of being on the platform • Implications → Interactions within Suppliers ◦ Optimizing for exposure of certain artists might help other artists ◦ Interplay between consumption across different artists
38. Part IV Leveraging User, Supplier & Content Understanding Algorithmic balancing Supplier Understanding Implications: - Fair exposure of supplier is not guaranteed - Interplay between user relevance and supplier diversity - Causal impact of one supplier’s events on related suppliers - Interaction effects across suppliers
39. Outline 1. Introduction to Marketplaces 2. Optimization Objectives in a Marketplace 3. Methods for Multi-Objective Ranking & Recommendations 4. Leveraging Consumer, Supplier & Content Understanding a. Consumer Understanding: i. Consumption diversity of users ii. Leveraging User intents iii. Quantifying and estimating user receptivity b. Supplier Understanding: i. Diversity across suppliers ii. Spillover effect across suppliers c. Content Understanding: i. User- & Content- aware Reward function ii. Query Understanding 5. Industrial Applications
40. Personalized Rewards User- & Playlist- aware bandit rewards Deriving User- and Content-specific Rewards for Contextual Bandits Paolo Dragone, Rishabh Mehrotra, Mounia Lalmas WWW 2019
41. Aggregate over playlists Consumption time of a sleep playlist is longer than average playlist consumption time. Variation across playlists
42. Aggregate over users Jazz listeners consume Jazz and other playlists for longer period than average. users. Variation across users
43. What we propose as success metrics Distribution-based reward functions
44. User- & Content- aware reward
45. User- & Content- aware reward Likely too granular, sparse, noisy, and costly to generate and maintain.
46. User- & Content- aware reward
47. Co-clustering Dhillon, Mallela & Modha, "Information-theoretic co-clustering”, KDD, 2003. Caveat no theoretical foundation for selecting the number of co-clusters apriori group = cluster group of user x playlist = co-cluster Users Playlists User groups Playlist groups Co-clustering
48. Distribution aware rewards
49. Distribution aware rewards
50. Query Understanding Shifting Consumption to Underserved Content Query Understanding for Surfacing Under-served Music Content Federico Tomasi, Rishabh Mehrotra, Aasish Pappu, Judith Bütepage, Brian Brost, Hugo Galvão and Mounia Lalmas CIKM 2020
51. Understanding User Intents helps! Query Understanding for Marketplace Users might not have any specific preferences (e.g. relaxing music) → opportunity to surface under-served content
52. Understanding User Intents helps! Query Understanding for Marketplace Users might not have any specific preferences (e.g. relaxing music) → opportunity to surface under-served content Non-focused queries: broad intent queries for which uses are more open to non-specific recommendations
53. Understanding User Intents helps! Query Understanding for Marketplace Users might not have any specific preferences (e.g. relaxing music) → opportunity to surface under-served content Non-focused queries: broad intent queries for which uses are more open to non-specific recommendations Target content groups: Casual Music and Niche Genre
54. Understanding User Intents helps! Query Understanding for Marketplace Query & user interactions features help in identifying such queries Trained models to identify such queries at scale
55. Understanding User Intents helps! Query Understanding for Marketplace Step 1: Identify non-focused queries
56. Understanding User Intents helps! Query Understanding for Marketplace Step 1: Identify non-focused queries Step 2: Surface recommendations which help other stakeholder objectives, without hurting user satisfaction
57. Understanding User Intents helps! Query Understanding for Marketplace Step 1: Identify non-focused queries Step 2: Surface recommendations which help other stakeholder objectives, without hurting user SAT Step 3: Understand trade-offs between Gain in exposure vs Loss in SAT
58. Understanding users, suppliers & content helps! Algorithmic balancing
59. Part IV: Leveraging Consumer, Supplier & Content Understanding 1. What is a task & why are they important? 2. Characterizing Tasks across interfaces: 1. desktop search 2. digital assistants 3. voice-only assistants 3. Understanding User Tasks in Web Search a. Extracting Query Intents b. Queries → Sessions → Tasks c. Search Task Understanding a. Task extraction b. Subtask extraction c. Hierarchies of tasks & subtasks d. Evaluating task extraction algorithms 5. Recommendation Systems a. Case study: Pinterest b. Case study: Spotify Take-home messages: • Understanding users helps in pushing other objectives without hurting key user metrics – Consumption diversity – User intents – User receptivity • Understanding suppliers helps develop right approaches to ensure supplier happiness: – Supplier diversity – Spillover effects • Content understanding allows us to know when to focus on what objectives