KDD 2020 Tutorial: Advances in Recommender Systems Part A: Recommendations in a Marketplace [User & Content Understanding] Rishabh Mehrotra Ben Carterette Senior Research Scientist, Senior Research Manager, Spotify, London Spotify, New York [email protected] [email protected] 23rd August 2020 @erishabh @BenCarterette https://sites.google.com/view/kdd20-marketplace-autorecsys/ 

Recommendations in a Marketplace ❏ Stakeholders & Objectives 

Recommendations in a Marketplace ❏ Stakeholders & Objectives ❏ Methods for Multi-Objective Recommendations 

Recommendations in a Marketplace ❏ Stakeholders & Objectives ❏ Methods for Multi-Objective Recommendations ❏ Leveraging User & Supplier Understanding 

Summary: Part I Introduction to Marketplaces - Traditional RecSys ML catered towards user-centric modeling - Multiple stakeholders in online marketplaces - Need to consider multiple objectives + ML models to optimize those objectives 

Summary: Part II Stakeholders & Objectives - Multiple stakeholders in online marketplaces - different industrial case-studies - UberEats, Postmates, Etsy, AirBnb, Music, P2P lending, Crowdfunding - Multiple, often conflicting objectives - +vely correlated, neutral, -vely correlated - ML methods needed to model the interplay between objectives - Important to carefully decide what a system optimizes for 

Summary: Part III Multi-Objective Methods for Recommendations - Flavors of multi-objective approaches available: - Multi-task learning - Scalarization - MO-Multi task learning - MO-Bandits & MO-RL - Often optimizing for multiple interaction metrics performs better for each metric than directly optimizing that metric - Not necessarily a Zero-Sum Game 

Part IV Leveraging User, Supplier & Content Understanding Algorithmic balancing 

Outline 1. Introduction to Marketplaces 2. Optimization Objectives in a Marketplace 3. Methods for Multi-Objective Ranking & Recommendations 4. Leveraging Consumer, Supplier & Content Understanding a. Consumer Understanding: i. Consumption diversity of users ii. Leveraging User intents iii. Quantifying and estimating user receptivity b. Supplier Understanding: i. Diversity across suppliers c. Content Understanding: i. Personalizing Reward function ii. Query Understanding 5. Industrial Applications 

Consumption Diversity Understanding Consumption Algorithmic Effects on the Diversity of Consumption on Spotify. Ashton Anderson, Lucas Maystre, Ian Anderson, Rishabh Mehrotra, Mounia Lalmas 

Generalist-Specialist Score (GS) Specialist Generalist Generalist Specialist Consumption Diversity: Differences across Users 

Generalist-Specialist Score (GS) Specialist Generalist Generalist Specialist Consumption Diversity: Differences across Users Generalists & specialists exhibit different retention & conversion behaviors 

User Intents Intent aware models 

User Intents for Targeting Intent aware satisfaction prediction models 

Understanding User Intents helps! Intent aware satisfaction prediction models → Multi-level models across different intents User Intents for Targeting 

Understanding User Intents helps! Intent aware satisfaction prediction models → Multi-level models across different intents Considering intent information is crucial → 20% improvement in SAT prediction over global model User Intents for Targeting 

User Receptivity How receptive are users? 

User Receptivity to Divergent Recommendations Leveraged randomized data to quantify & predict user receptivity 

User Receptivity to Divergent Recommendations Leveraged randomized data to quantify & predict user receptivity Three ways of quantifying tolerance: 1. Engagement centric 2. Effort centric 3. Emendation centric (stickiness) 

User Receptivity to Divergent Recommendations Leveraged randomized data to quantify & predict user receptivity Three ways of quantifying tolerance: 1. Engagement centric 2. Effort centric 3. Emendation centric (stickiness) Used user features to train predictive model 

User tolerance for trade-off decisioning - Different users are receptive to different extent - inactive users vs heavy use users User Receptivity to Divergent Recommendations 

User tolerance for trade-off decisioning - Different users are receptive to different extent - inactive users vs heavy use users - Identified predictive features for user receptivity - Consumption diversity is NOT predictive of engagement receptivity User Receptivity to Divergent Recommendations 

User tolerance for trade-off decisioning - Different users are receptive to different extent - inactive users vs heavy use users - Identified predictive features for user receptivity - Consumption diversity is NOT predictive of engagement receptivity - Get 70% accuracy in predicting Engagement tolerance - Receptivity predictive of future engagement User Receptivity to Divergent Recommendations 

Part IV Leveraging User, Supplier & Content Understanding Algorithmic balancing User Understanding Implications: - Users have different consumption diversity - Consider user level heterogeneity - Identifying user intents helps better target content - perhaps esp for content from other stakeholder objectives - Understanding user receptivity helps in avoid user dissatisfaction 

Outline 1. Introduction to Marketplaces 2. Optimization Objectives in a Marketplace 3. Methods for Multi-Objective Ranking & Recommendations 4. Leveraging Consumer, Supplier & Content Understanding a. Consumer Understanding: i. Consumption diversity of users ii. Leveraging User intents iii. Quantifying and estimating user receptivity b. Supplier Understanding: i. Diversity across suppliers ii. Spillover effect across suppliers c. Content Understanding: i. Personalizing Reward function ii. Query Understanding 5. Industrial Applications 

Supplier Diversity Trade-off between Relevance, Diversity & Satisfaction Towards a Fair Marketplace: Counterfactual Evaluation of the trade-off between Relevance, Fairness & Satisfaction in Recommendation Systems. Rishabh Mehrotra, James McInerney, Hugues Bouchard, Mounia Lalmas, Fernando Diaz CIKM 2018 

Let’s consider 2 stakeholders Stakeholders Relevance Exposure Fairness 

User Relevance vs Artist Diversity 

Relevance vs Fairness Very few sets have both high relevance & high fairness User Relevance vs Artist Diversity 

How do we trade-off User Relevance vs Supplier Diversity? 

Summary of Recommendation Policies Policy I: Optimizing Relevance Policy II: Optimizing Fairness Policy III: Probabilistic Policy Policy IV: Trade-off Relevance & Fairness Policy V: Guaranteed Relevance Policy VI: Adaptive Policy I Policy VI: Adaptive Policy II 

Balance between Relevance & Artist Diversity Diversity Relevance Inflection points of good trade-off 

diversity relevance Balance between Relevance & Artist Diversity % loss in diversity % loss in relevance % gain in satisfaction optimizing relevance (β=1) 69.1 0 0 optimizing diversity (β=0) 0 57.7 -32.2 trade-off relevance & diversity (β =0.7) 42.7 9.8 -10.2 guaranteed relevance (rel >= 0.7) 51.7 7.8 4.4 diversity affinity aware 15 21.2 12.1 Diversity affinity aware provides the best overall trade-off → personalizing diversity 

Spillover Effect Related suppliers benefits from events on focal suppliers Inferring the Causal Impact of New Track Releases on Music Recommendation Platforms through Counterfactual Predictions Rishabh Mehrotra, Prasanta Bhattacharya, Mounia Lalmas RecSys 2020 (LBR) 

Understanding User Intents helps! Spillover Effects across Suppliers ● New track releases help not only the focal artist but also other related artists ○ increased tendency of users to explore and discover related music 

Understanding User Intents helps! Spillover Effects across Suppliers ● New track releases help not only the focal artist but also other related artists ○ increased tendency of users to explore and discover related music ● Other artists benefit by virtue of being on the platform 

Understanding User Intents helps! Spillover Effects across Suppliers ● New track releases help not only the focal artist but also other related artists ○ increased tendency of users to explore and discover related music ● Other artists benefit by virtue of being on the platform ● Implications → Interactions within Suppliers ○ Optimizing for exposure of certain artists might help other artists ○ Interplay between consumption across different artists 

Part IV Leveraging User, Supplier & Content Understanding Algorithmic balancing Supplier Understanding Implications: - Fair exposure of supplier is not guaranteed - Interplay between user relevance and supplier diversity - Causal impact of one supplier’s events on related suppliers - Interaction effects across suppliers 

Outline 1. Introduction to Marketplaces 2. Optimization Objectives in a Marketplace 3. Methods for Multi-Objective Ranking & Recommendations 4. Leveraging Consumer, Supplier & Content Understanding a. Consumer Understanding: i. Consumption diversity of users ii. Leveraging User intents iii. Quantifying and estimating user receptivity b. Supplier Understanding: i. Diversity across suppliers ii. Spillover effect across suppliers c. Content Understanding: i. User- & Content- aware Reward function ii. Query Understanding 5. Industrial Applications 

Personalized Rewards User- & Playlist- aware bandit rewards Deriving User- and Content-specific Rewards for Contextual Bandits Paolo Dragone, Rishabh Mehrotra, Mounia Lalmas WWW 2019 

Aggregate over playlists Consumption time of a sleep playlist is longer than average playlist consumption time. Variation across playlists 

Aggregate over users Jazz listeners consume Jazz and other playlists for longer period than average. users. Variation across users 

What we propose as success metrics Distribution-based reward functions 

User- & Content- aware reward 

User- & Content- aware reward Likely too granular, sparse, noisy, and costly to generate and maintain. 

User- & Content- aware reward 

Co-clustering Dhillon, Mallela & Modha, "Information-theoretic co-clustering”, KDD, 2003. Caveat no theoretical foundation for selecting the number of co-clusters apriori group = cluster group of user x playlist = co-cluster Users Playlists User groups Playlist groups Co-clustering 

Distribution aware rewards 

Distribution aware rewards 

Query Understanding Shifting Consumption to Underserved Content Query Understanding for Surfacing Under-served Music Content Federico Tomasi, Rishabh Mehrotra, Aasish Pappu, Judith Bütepage, Brian Brost, Hugo Galvão and Mounia Lalmas CIKM 2020 

Understanding User Intents helps! Query Understanding for Marketplace Users might not have any specific preferences (e.g. relaxing music) → opportunity to surface under-served content 

Understanding User Intents helps! Query Understanding for Marketplace Users might not have any specific preferences (e.g. relaxing music) → opportunity to surface under-served content Non-focused queries: broad intent queries for which uses are more open to non-specific recommendations 

Understanding User Intents helps! Query Understanding for Marketplace Users might not have any specific preferences (e.g. relaxing music) → opportunity to surface under-served content Non-focused queries: broad intent queries for which uses are more open to non-specific recommendations Target content groups: Casual Music and Niche Genre 

Understanding User Intents helps! Query Understanding for Marketplace Query & user interactions features help in identifying such queries Trained models to identify such queries at scale 

Understanding User Intents helps! Query Understanding for Marketplace Step 1: Identify non-focused queries 

Understanding User Intents helps! Query Understanding for Marketplace Step 1: Identify non-focused queries Step 2: Surface recommendations which help other stakeholder objectives, without hurting user satisfaction 

Understanding User Intents helps! Query Understanding for Marketplace Step 1: Identify non-focused queries Step 2: Surface recommendations which help other stakeholder objectives, without hurting user SAT Step 3: Understand trade-offs between Gain in exposure vs Loss in SAT 

Understanding users, suppliers & content helps! Algorithmic balancing 

Part IV: Leveraging Consumer, Supplier & Content Understanding 1. What is a task & why are they important? 2. Characterizing Tasks across interfaces: 1. desktop search 2. digital assistants 3. voice-only assistants 3. Understanding User Tasks in Web Search a. Extracting Query Intents b. Queries → Sessions → Tasks c. Search Task Understanding a. Task extraction b. Subtask extraction c. Hierarchies of tasks & subtasks d. Evaluating task extraction algorithms 5. Recommendation Systems a. Case study: Pinterest b. Case study: Spotify Take-home messages: • Understanding users helps in pushing other objectives without hurting key user metrics – Consumption diversity – User intents – User receptivity • Understanding suppliers helps develop right approaches to ensure supplier happiness: – Supplier diversity – Spillover effects • Content understanding allows us to know when to focus on what objectives