Upgrade to Pro — share decks privately, control downloads, hide ads and more …

KDD 2020 Marketplace Tutorial Rishabh Part 4

KDD 2020 Marketplace Tutorial Rishabh Part 4

Rishabh Mehrotra

August 23, 2020
Tweet

More Decks by Rishabh Mehrotra

Other Decks in Technology

Transcript

  1. KDD 2020 Tutorial: Advances in Recommender Systems Part A: Recommendations

    in a Marketplace [User & Content Understanding] Rishabh Mehrotra Ben Carterette Senior Research Scientist, Senior Research Manager, Spotify, London Spotify, New York [email protected] [email protected] 23rd August 2020 @erishabh @BenCarterette https://sites.google.com/view/kdd20-marketplace-autorecsys/
  2. Recommendations in a Marketplace ❏ Stakeholders & Objectives ❏ Methods

    for Multi-Objective Recommendations ❏ Leveraging User & Supplier Understanding
  3. Summary: Part I Introduction to Marketplaces - Traditional RecSys ML

    catered towards user-centric modeling - Multiple stakeholders in online marketplaces - Need to consider multiple objectives + ML models to optimize those objectives
  4. Summary: Part II Stakeholders & Objectives - Multiple stakeholders in

    online marketplaces - different industrial case-studies - UberEats, Postmates, Etsy, AirBnb, Music, P2P lending, Crowdfunding - Multiple, often conflicting objectives - +vely correlated, neutral, -vely correlated - ML methods needed to model the interplay between objectives - Important to carefully decide what a system optimizes for
  5. Summary: Part III Multi-Objective Methods for Recommendations - Flavors of

    multi-objective approaches available: - Multi-task learning - Scalarization - MO-Multi task learning - MO-Bandits & MO-RL - Often optimizing for multiple interaction metrics performs better for each metric than directly optimizing that metric - Not necessarily a Zero-Sum Game
  6. Outline 1. Introduction to Marketplaces 2. Optimization Objectives in a

    Marketplace 3. Methods for Multi-Objective Ranking & Recommendations 4. Leveraging Consumer, Supplier & Content Understanding a. Consumer Understanding: i. Consumption diversity of users ii. Leveraging User intents iii. Quantifying and estimating user receptivity b. Supplier Understanding: i. Diversity across suppliers c. Content Understanding: i. Personalizing Reward function ii. Query Understanding 5. Industrial Applications
  7. Consumption Diversity Understanding Consumption Algorithmic Effects on the Diversity of

    Consumption on Spotify. Ashton Anderson, Lucas Maystre, Ian Anderson, Rishabh Mehrotra, Mounia Lalmas
  8. Generalist-Specialist Score (GS) Specialist Generalist Generalist Specialist Consumption Diversity: Differences

    across Users Generalists & specialists exhibit different retention & conversion behaviors
  9. Understanding User Intents helps! Intent aware satisfaction prediction models →

    Multi-level models across different intents User Intents for Targeting
  10. Understanding User Intents helps! Intent aware satisfaction prediction models →

    Multi-level models across different intents Considering intent information is crucial → 20% improvement in SAT prediction over global model User Intents for Targeting
  11. User Receptivity to Divergent Recommendations Leveraged randomized data to quantify

    & predict user receptivity Three ways of quantifying tolerance: 1. Engagement centric 2. Effort centric 3. Emendation centric (stickiness)
  12. User Receptivity to Divergent Recommendations Leveraged randomized data to quantify

    & predict user receptivity Three ways of quantifying tolerance: 1. Engagement centric 2. Effort centric 3. Emendation centric (stickiness) Used user features to train predictive model
  13. User tolerance for trade-off decisioning - Different users are receptive

    to different extent - inactive users vs heavy use users User Receptivity to Divergent Recommendations
  14. User tolerance for trade-off decisioning - Different users are receptive

    to different extent - inactive users vs heavy use users - Identified predictive features for user receptivity - Consumption diversity is NOT predictive of engagement receptivity User Receptivity to Divergent Recommendations
  15. User tolerance for trade-off decisioning - Different users are receptive

    to different extent - inactive users vs heavy use users - Identified predictive features for user receptivity - Consumption diversity is NOT predictive of engagement receptivity - Get 70% accuracy in predicting Engagement tolerance - Receptivity predictive of future engagement User Receptivity to Divergent Recommendations
  16. Part IV Leveraging User, Supplier & Content Understanding Algorithmic balancing

    User Understanding Implications: - Users have different consumption diversity - Consider user level heterogeneity - Identifying user intents helps better target content - perhaps esp for content from other stakeholder objectives - Understanding user receptivity helps in avoid user dissatisfaction
  17. Outline 1. Introduction to Marketplaces 2. Optimization Objectives in a

    Marketplace 3. Methods for Multi-Objective Ranking & Recommendations 4. Leveraging Consumer, Supplier & Content Understanding a. Consumer Understanding: i. Consumption diversity of users ii. Leveraging User intents iii. Quantifying and estimating user receptivity b. Supplier Understanding: i. Diversity across suppliers ii. Spillover effect across suppliers c. Content Understanding: i. Personalizing Reward function ii. Query Understanding 5. Industrial Applications
  18. Supplier Diversity Trade-off between Relevance, Diversity & Satisfaction Towards a

    Fair Marketplace: Counterfactual Evaluation of the trade-off between Relevance, Fairness & Satisfaction in Recommendation Systems. Rishabh Mehrotra, James McInerney, Hugues Bouchard, Mounia Lalmas, Fernando Diaz CIKM 2018
  19. Relevance vs Fairness Very few sets have both high relevance

    & high fairness User Relevance vs Artist Diversity
  20. Summary of Recommendation Policies Policy I: Optimizing Relevance Policy II:

    Optimizing Fairness Policy III: Probabilistic Policy Policy IV: Trade-off Relevance & Fairness Policy V: Guaranteed Relevance Policy VI: Adaptive Policy I Policy VI: Adaptive Policy II
  21. diversity relevance Balance between Relevance & Artist Diversity % loss

    in diversity % loss in relevance % gain in satisfaction optimizing relevance (β=1) 69.1 0 0 optimizing diversity (β=0) 0 57.7 -32.2 trade-off relevance & diversity (β =0.7) 42.7 9.8 -10.2 guaranteed relevance (rel >= 0.7) 51.7 7.8 4.4 diversity affinity aware 15 21.2 12.1 Diversity affinity aware provides the best overall trade-off → personalizing diversity
  22. Spillover Effect Related suppliers benefits from events on focal suppliers

    Inferring the Causal Impact of New Track Releases on Music Recommendation Platforms through Counterfactual Predictions Rishabh Mehrotra, Prasanta Bhattacharya, Mounia Lalmas RecSys 2020 (LBR)
  23. Understanding User Intents helps! Spillover Effects across Suppliers • New

    track releases help not only the focal artist but also other related artists ◦ increased tendency of users to explore and discover related music
  24. Understanding User Intents helps! Spillover Effects across Suppliers • New

    track releases help not only the focal artist but also other related artists ◦ increased tendency of users to explore and discover related music • Other artists benefit by virtue of being on the platform
  25. Understanding User Intents helps! Spillover Effects across Suppliers • New

    track releases help not only the focal artist but also other related artists ◦ increased tendency of users to explore and discover related music • Other artists benefit by virtue of being on the platform • Implications → Interactions within Suppliers ◦ Optimizing for exposure of certain artists might help other artists ◦ Interplay between consumption across different artists
  26. Part IV Leveraging User, Supplier & Content Understanding Algorithmic balancing

    Supplier Understanding Implications: - Fair exposure of supplier is not guaranteed - Interplay between user relevance and supplier diversity - Causal impact of one supplier’s events on related suppliers - Interaction effects across suppliers
  27. Outline 1. Introduction to Marketplaces 2. Optimization Objectives in a

    Marketplace 3. Methods for Multi-Objective Ranking & Recommendations 4. Leveraging Consumer, Supplier & Content Understanding a. Consumer Understanding: i. Consumption diversity of users ii. Leveraging User intents iii. Quantifying and estimating user receptivity b. Supplier Understanding: i. Diversity across suppliers ii. Spillover effect across suppliers c. Content Understanding: i. User- & Content- aware Reward function ii. Query Understanding 5. Industrial Applications
  28. Personalized Rewards User- & Playlist- aware bandit rewards Deriving User-

    and Content-specific Rewards for Contextual Bandits Paolo Dragone, Rishabh Mehrotra, Mounia Lalmas WWW 2019
  29. Aggregate over playlists Consumption time of a sleep playlist is

    longer than average playlist consumption time. Variation across playlists
  30. Aggregate over users Jazz listeners consume Jazz and other playlists

    for longer period than average. users. Variation across users
  31. Co-clustering Dhillon, Mallela & Modha, "Information-theoretic co-clustering”, KDD, 2003. Caveat

    no theoretical foundation for selecting the number of co-clusters apriori group = cluster group of user x playlist = co-cluster Users Playlists User groups Playlist groups Co-clustering
  32. Query Understanding Shifting Consumption to Underserved Content Query Understanding for

    Surfacing Under-served Music Content Federico Tomasi, Rishabh Mehrotra, Aasish Pappu, Judith Bütepage, Brian Brost, Hugo Galvão and Mounia Lalmas CIKM 2020
  33. Understanding User Intents helps! Query Understanding for Marketplace Users might

    not have any specific preferences (e.g. relaxing music) → opportunity to surface under-served content
  34. Understanding User Intents helps! Query Understanding for Marketplace Users might

    not have any specific preferences (e.g. relaxing music) → opportunity to surface under-served content Non-focused queries: broad intent queries for which uses are more open to non-specific recommendations
  35. Understanding User Intents helps! Query Understanding for Marketplace Users might

    not have any specific preferences (e.g. relaxing music) → opportunity to surface under-served content Non-focused queries: broad intent queries for which uses are more open to non-specific recommendations Target content groups: Casual Music and Niche Genre
  36. Understanding User Intents helps! Query Understanding for Marketplace Query &

    user interactions features help in identifying such queries Trained models to identify such queries at scale
  37. Understanding User Intents helps! Query Understanding for Marketplace Step 1:

    Identify non-focused queries Step 2: Surface recommendations which help other stakeholder objectives, without hurting user satisfaction
  38. Understanding User Intents helps! Query Understanding for Marketplace Step 1:

    Identify non-focused queries Step 2: Surface recommendations which help other stakeholder objectives, without hurting user SAT Step 3: Understand trade-offs between Gain in exposure vs Loss in SAT
  39. Part IV: Leveraging Consumer, Supplier & Content Understanding 1. What

    is a task & why are they important? 2. Characterizing Tasks across interfaces: 1. desktop search 2. digital assistants 3. voice-only assistants 3. Understanding User Tasks in Web Search a. Extracting Query Intents b. Queries → Sessions → Tasks c. Search Task Understanding a. Task extraction b. Subtask extraction c. Hierarchies of tasks & subtasks d. Evaluating task extraction algorithms 5. Recommendation Systems a. Case study: Pinterest b. Case study: Spotify Take-home messages: • Understanding users helps in pushing other objectives without hurting key user metrics – Consumption diversity – User intents – User receptivity • Understanding suppliers helps develop right approaches to ensure supplier happiness: – Supplier diversity – Spillover effects • Content understanding allows us to know when to focus on what objectives