KDD 2020 Marketplace Tutorial Rishabh Part 5

1. KDD 2020 Tutorial: Advances in Recommender Systems Part A: Recommendations in a Marketplace [Industrial Applications] Rishabh Mehrotra Ben Carterette Senior Research Scientist, Senior Research Manager, Spotify, London Spotify, New York rishabhm@spotify.com benjaminc@spotify.com 23rd August 2020 @erishabh @BenCarterette https://sites.google.com/view/kdd20-marketplace-autorecsys/
2. Recommendations in a Marketplace ❏ Stakeholders & Objectives
3. Recommendations in a Marketplace ❏ Stakeholders & Objectives ❏ Methods for Multi-Objective Recommendations
4. Recommendations in a Marketplace ❏ Stakeholders & Objectives ❏ Methods for Multi-Objective Recommendations ❏ Leveraging User & Supplier Understanding
5. Recommendations in a Marketplace ❏ Stakeholders & Objectives ❏ Methods for Multi-Objective Recommendations ❏ Leveraging User & Supplier Understanding ❏ Industrial Applications
6. Outline 1. Introduction to Marketplaces 2. Optimization Objectives in a Marketplace 3. Methods for Multi-Objective Ranking & Recommendations 4. Leveraging Consumer, Supplier & Content Understanding 5. Industrial Applications a. Recommendation with capacity constraints b. Email volume optimization at LinkedIn c. Click shaping at Yahoo d. Seller side AB testing at Facebook Marketplace e. Feedback shaping to nurture content creation at LinkedIn f. Joint optimization for profit & relevance at Etsy g. Revenue Maximization at AirBnb Search h. Joint Optimization for Music Streaming at Spotify i. Other Applications
7. Application I: Recommendation with Capacity Constraint
8. Application I: Recommendation with Capacity Constraint Items often associated with capacity constraints Konstantina, Kawale, and Banerjee. "Recommendation with capacity constraints." Proceedings of CIKM 2017
9. Application I: Recommendation with Capacity Constraint Items often associated with capacity constraints 2 key concepts: item capacity & user propensity
10. Application I: Recommendation with Capacity Constraint Items often associated with capacity constraints 2 key concepts: item capacity & user propensity Weighted objectives: 1. Loss capturing recommendations 2. Capacity loss
11. Application I: Recommendation with Capacity Constraint Items often associated with capacity constraints 2 key concepts: item capacity & user propensity Weighted objectives: 1. Loss capturing recommendations 2. Capacity loss a. Avg capacity loss:
12. Application I: Recommendation with Capacity Constraint Items often associated with capacity constraints 2 key concepts: item capacity & user propensity Weighted objectives: 1. Loss capturing recommendations 2. Capacity loss a. Avg capacity loss: b. 1[.] not suitable for optimization
13. Application I: Recommendation with Capacity Constraint Items often associated with capacity constraints 2 key concepts: item capacity & user propensity Weighted objectives: 1. Loss capturing recommendations 2. Capacity loss a. Avg capacity loss: b. 1[.] not suitable for optimization c. logistic loss of the difference as the surrogate
14. Application I: Recommendation with Capacity Constraint Items often associated with capacity constraints 2 key concepts: item capacity & user propensity Weighted objectives: 1. Loss capturing recommendations 2. Capacity loss Overall Objective:
15. Application I: Recommendation with Capacity Constraint Items often associated with capacity constraints 2 key concepts: item capacity & user propensity Weighted objectives: 1. Loss capturing recommendations 2. Capacity loss Overall Objective:
16. Application II: Email Volume Optimization at LinkedIn
17. Application II: Email Volume Optimization at LinkedIn Less emails → drop in engagement More emails → user ignores, drop in brand value Trade-offs: • Minimize # emails sent • Maximize # downstream sessions • Minimize # resulting complaints Gupta, Liang, Tseng, Vijay, Chen and Rosales. Email volume optimization at LinkedIn. KDD 2016
18. Application II: Email Volume Optimization at LinkedIn Trade-offs: • Minimize # emails sent • Maximize # downstream sessions • Minimize # resulting complaints MOO formulated as a Constrained linear programming problem (LP) → Gupta, Liang, Tseng, Vijay, Chen and Rosales. Email volume optimization at LinkedIn. KDD 2016
19. Application II: Email Volume Optimization at LinkedIn Trade-offs: • Minimize # emails sent • Maximize # downstream sessions • Minimize # resulting complaints MOO formulated as a Constrained linear programming problem (LP) → Gupta, Liang, Tseng, Vijay, Chen and Rosales. Email volume optimization at LinkedIn. KDD 2016 expected number of emails sent for serving plan z
20. Application II: Email Volume Optimization at LinkedIn Trade-offs: • Minimize # emails sent • Maximize # downstream sessions • Minimize # resulting complaints MOO formulated as a Constrained linear programming problem (LP) → Gupta, Liang, Tseng, Vijay, Chen and Rosales. Email volume optimization at LinkedIn. KDD 2016 Targets for global & local session counts
21. Application II: Email Volume Optimization at LinkedIn Trade-offs: • Minimize # emails sent • Maximize # downstream sessions • Minimize # resulting complaints MOO formulated as a Constrained linear programming problem (LP) → Gupta, Liang, Tseng, Vijay, Chen and Rosales. Email volume optimization at LinkedIn. KDD 2016 Global & local tolerance for complaints
22. Application III: Click Shaping on Yahoo!
23. Application III: Click Shaping on Yahoo! Agarwal, Deepak, et al. "Click shaping to optimize multiple objectives." KDD 2011
24. Application III: Click Shaping on Yahoo! Agarwal, Deepak, et al. "Click shaping to optimize multiple objectives." KDD 2011
25. Application IV: Seller Side AB Testing (Facebook Marketplace)
26. Application IV: Seller side AB testing (Facebook Marketplace) Ha-Thuc, Viet, et al. "A Counterfactual Framework for Seller-Side A/B Testing on Marketplaces." SIGIR 2020
27. Application V: Nurturing Content Creation on LinkedIn
28. Application V: Nurturing Content Creation on LinkedIn Ye Tu et al. Feedback Shaping: A Modeling Approach to Nurture Content Creation. KDD 2019
29. Application VI: Joint Optimization of Profit & Relevance at Etsy
30. Application V: Joint Optimization of Profit & Relevance at Etsy
31. Application VII: Revenue Maximization via AirBnb Search
32. Application VII: Revenue Maximization via AirBnb Search
33. Application VIII: Joint Optimization for Music Streaming at Spotify
34. Application VIII: Joint Optimization for Music Streaming at Spotify
35. Other Applications
36. Part I: Introduction to Marketplaces 1. What is a task & why are they important? 2. Characterizing Tasks across interfaces: 1. desktop search 2. digital assistants 3. voice-only assistants 3. Understanding User Tasks in Web Search a. Extracting Query Intents b. Queries → Sessions → Tasks c. Search Task Understanding a. Task extraction b. Subtask extraction c. Hierarchies of tasks & subtasks d. Evaluating task extraction algorithms 5. Recommendation Systems a. Case study: Pinterest b. Case study: Spotify Take-home messages: • Traditional RecSys methods are majorly user-centric • Need to explicitly consider other stakeholders • Different types of marketplaces & examples • Components: – Multi-objective methods – User & Content understanding
37. Part II: Optimization Objectives in Marketplace 1. What is a task & why are they important? 2. Characterizing Tasks across interfaces: 1. desktop search 2. digital assistants 3. voice-only assistants 3. Understanding User Tasks in Web Search a. Extracting Query Intents b. Queries → Sessions → Tasks c. Search Task Understanding a. Task extraction b. Subtask extraction c. Hierarchies of tasks & subtasks d. Evaluating task extraction algorithms 5. Recommendation Systems a. Case study: Pinterest b. Case study: Spotify Take-home messages: • Multiple stakeholders – Multiple objectives per stakeholder • Interplay between objectives: – Neutral / positive / negative • Careful consideration needed to decide which objectives to optimize for
38. Part III: Methods for Multi-Objective Recommendation 1. What is a task & why are they important? 2. Characterizing Tasks across interfaces: 1. desktop search 2. digital assistants 3. voice-only assistants 3. Understanding User Tasks in Web Search a. Extracting Query Intents b. Queries → Sessions → Tasks c. Search Task Understanding a. Task extraction b. Subtask extraction c. Hierarchies of tasks & subtasks d. Evaluating task extraction algorithms 5. Recommendation Systems a. Case study: Pinterest b. Case study: Spotify Take-home messages: • Flavors of multi-objective approaches available: – Multi-task learning – Scalarization – MO-Multi task learning – MO-Bandits & MO-RL • Often optimizing for multiple interaction metrics performs better for each metric than directly optimizing that metric • Not necessarily a Zero-Sum Game
39. Part IV: Leveraging Consumer, Supplier & Content Understanding 1. What is a task & why are they important? 2. Characterizing Tasks across interfaces: 1. desktop search 2. digital assistants 3. voice-only assistants 3. Understanding User Tasks in Web Search a. Extracting Query Intents b. Queries → Sessions → Tasks c. Search Task Understanding a. Task extraction b. Subtask extraction c. Hierarchies of tasks & subtasks d. Evaluating task extraction algorithms 5. Recommendation Systems a. Case study: Pinterest b. Case study: Spotify Take-home messages: • Understanding users helps in pushing other objectives without hurting key user metrics – Consumption diversity – User intents – User receptivity • Understanding suppliers helps develop right approaches to ensure supplier happiness: – Supplier diversity – Spillover effects • Content understanding allows us to know when to focus on what objectives
40. Thank you! Rishabh Mehrotra Ben Carterette Senior Research Scientist, Senior Research Manager, Spotify, London Spotify, New York rishabhm@spotify.com benjaminc@spotify.com https://sites.google.com/view/kdd20-marketplace-autorecsys/