KDD 2020 Tutorial: Advances in Recommender Systems Part A: Recommendations in a Marketplace [Industrial Applications] Rishabh Mehrotra Ben Carterette Senior Research Scientist, Senior Research Manager, Spotify, London Spotify, New York [email protected] [email protected] 23rd August 2020 @erishabh @BenCarterette https://sites.google.com/view/kdd20-marketplace-autorecsys/ 

Recommendations in a Marketplace ❏ Stakeholders & Objectives 

Recommendations in a Marketplace ❏ Stakeholders & Objectives ❏ Methods for Multi-Objective Recommendations 

Recommendations in a Marketplace ❏ Stakeholders & Objectives ❏ Methods for Multi-Objective Recommendations ❏ Leveraging User & Supplier Understanding 

Recommendations in a Marketplace ❏ Stakeholders & Objectives ❏ Methods for Multi-Objective Recommendations ❏ Leveraging User & Supplier Understanding ❏ Industrial Applications 

Outline 1. Introduction to Marketplaces 2. Optimization Objectives in a Marketplace 3. Methods for Multi-Objective Ranking & Recommendations 4. Leveraging Consumer, Supplier & Content Understanding 5. Industrial Applications a. Recommendation with capacity constraints b. Email volume optimization at LinkedIn c. Click shaping at Yahoo d. Seller side AB testing at Facebook Marketplace e. Feedback shaping to nurture content creation at LinkedIn f. Joint optimization for profit & relevance at Etsy g. Revenue Maximization at AirBnb Search h. Joint Optimization for Music Streaming at Spotify i. Other Applications 

Application I: Recommendation with Capacity Constraint 

Application I: Recommendation with Capacity Constraint Items often associated with capacity constraints Konstantina, Kawale, and Banerjee. "Recommendation with capacity constraints." Proceedings of CIKM 2017 

Application I: Recommendation with Capacity Constraint Items often associated with capacity constraints 2 key concepts: item capacity & user propensity 

Application I: Recommendation with Capacity Constraint Items often associated with capacity constraints 2 key concepts: item capacity & user propensity Weighted objectives: 1. Loss capturing recommendations 2. Capacity loss 

Application I: Recommendation with Capacity Constraint Items often associated with capacity constraints 2 key concepts: item capacity & user propensity Weighted objectives: 1. Loss capturing recommendations 2. Capacity loss a. Avg capacity loss: 

Application I: Recommendation with Capacity Constraint Items often associated with capacity constraints 2 key concepts: item capacity & user propensity Weighted objectives: 1. Loss capturing recommendations 2. Capacity loss a. Avg capacity loss: b. 1[.] not suitable for optimization 

Application I: Recommendation with Capacity Constraint Items often associated with capacity constraints 2 key concepts: item capacity & user propensity Weighted objectives: 1. Loss capturing recommendations 2. Capacity loss a. Avg capacity loss: b. 1[.] not suitable for optimization c. logistic loss of the difference as the surrogate 

Application I: Recommendation with Capacity Constraint Items often associated with capacity constraints 2 key concepts: item capacity & user propensity Weighted objectives: 1. Loss capturing recommendations 2. Capacity loss Overall Objective: 

Application I: Recommendation with Capacity Constraint Items often associated with capacity constraints 2 key concepts: item capacity & user propensity Weighted objectives: 1. Loss capturing recommendations 2. Capacity loss Overall Objective: 

Application II: Email Volume Optimization at LinkedIn 

Application II: Email Volume Optimization at LinkedIn Less emails → drop in engagement More emails → user ignores, drop in brand value Trade-offs: ● Minimize # emails sent ● Maximize # downstream sessions ● Minimize # resulting complaints Gupta, Liang, Tseng, Vijay, Chen and Rosales. Email volume optimization at LinkedIn. KDD 2016 

Application II: Email Volume Optimization at LinkedIn Trade-offs: ● Minimize # emails sent ● Maximize # downstream sessions ● Minimize # resulting complaints MOO formulated as a Constrained linear programming problem (LP) → Gupta, Liang, Tseng, Vijay, Chen and Rosales. Email volume optimization at LinkedIn. KDD 2016 

Application II: Email Volume Optimization at LinkedIn Trade-offs: ● Minimize # emails sent ● Maximize # downstream sessions ● Minimize # resulting complaints MOO formulated as a Constrained linear programming problem (LP) → Gupta, Liang, Tseng, Vijay, Chen and Rosales. Email volume optimization at LinkedIn. KDD 2016 expected number of emails sent for serving plan z 

Application II: Email Volume Optimization at LinkedIn Trade-offs: ● Minimize # emails sent ● Maximize # downstream sessions ● Minimize # resulting complaints MOO formulated as a Constrained linear programming problem (LP) → Gupta, Liang, Tseng, Vijay, Chen and Rosales. Email volume optimization at LinkedIn. KDD 2016 Targets for global & local session counts 

Application II: Email Volume Optimization at LinkedIn Trade-offs: ● Minimize # emails sent ● Maximize # downstream sessions ● Minimize # resulting complaints MOO formulated as a Constrained linear programming problem (LP) → Gupta, Liang, Tseng, Vijay, Chen and Rosales. Email volume optimization at LinkedIn. KDD 2016 Global & local tolerance for complaints 

Application III: Click Shaping on Yahoo! 

Application III: Click Shaping on Yahoo! Agarwal, Deepak, et al. "Click shaping to optimize multiple objectives." KDD 2011 

Application III: Click Shaping on Yahoo! Agarwal, Deepak, et al. "Click shaping to optimize multiple objectives." KDD 2011 

Application IV: Seller Side AB Testing (Facebook Marketplace) 

Application IV: Seller side AB testing (Facebook Marketplace) Ha-Thuc, Viet, et al. "A Counterfactual Framework for Seller-Side A/B Testing on Marketplaces." SIGIR 2020 

Application V: Nurturing Content Creation on LinkedIn 

Application V: Nurturing Content Creation on LinkedIn Ye Tu et al. Feedback Shaping: A Modeling Approach to Nurture Content Creation. KDD 2019 

Application VI: Joint Optimization of Profit & Relevance at Etsy 

Application V: Joint Optimization of Profit & Relevance at Etsy 

Application VII: Revenue Maximization via AirBnb Search 

Application VII: Revenue Maximization via AirBnb Search 

Application VIII: Joint Optimization for Music Streaming at Spotify 

Application VIII: Joint Optimization for Music Streaming at Spotify 

Other Applications 

Part I: Introduction to Marketplaces 1. What is a task & why are they important? 2. Characterizing Tasks across interfaces: 1. desktop search 2. digital assistants 3. voice-only assistants 3. Understanding User Tasks in Web Search a. Extracting Query Intents b. Queries → Sessions → Tasks c. Search Task Understanding a. Task extraction b. Subtask extraction c. Hierarchies of tasks & subtasks d. Evaluating task extraction algorithms 5. Recommendation Systems a. Case study: Pinterest b. Case study: Spotify Take-home messages: • Traditional RecSys methods are majorly user-centric • Need to explicitly consider other stakeholders • Different types of marketplaces & examples • Components: – Multi-objective methods – User & Content understanding 

Part II: Optimization Objectives in Marketplace 1. What is a task & why are they important? 2. Characterizing Tasks across interfaces: 1. desktop search 2. digital assistants 3. voice-only assistants 3. Understanding User Tasks in Web Search a. Extracting Query Intents b. Queries → Sessions → Tasks c. Search Task Understanding a. Task extraction b. Subtask extraction c. Hierarchies of tasks & subtasks d. Evaluating task extraction algorithms 5. Recommendation Systems a. Case study: Pinterest b. Case study: Spotify Take-home messages: • Multiple stakeholders – Multiple objectives per stakeholder • Interplay between objectives: – Neutral / positive / negative • Careful consideration needed to decide which objectives to optimize for 

Part III: Methods for Multi-Objective Recommendation 1. What is a task & why are they important? 2. Characterizing Tasks across interfaces: 1. desktop search 2. digital assistants 3. voice-only assistants 3. Understanding User Tasks in Web Search a. Extracting Query Intents b. Queries → Sessions → Tasks c. Search Task Understanding a. Task extraction b. Subtask extraction c. Hierarchies of tasks & subtasks d. Evaluating task extraction algorithms 5. Recommendation Systems a. Case study: Pinterest b. Case study: Spotify Take-home messages: • Flavors of multi-objective approaches available: – Multi-task learning – Scalarization – MO-Multi task learning – MO-Bandits & MO-RL • Often optimizing for multiple interaction metrics performs better for each metric than directly optimizing that metric • Not necessarily a Zero-Sum Game 

Part IV: Leveraging Consumer, Supplier & Content Understanding 1. What is a task & why are they important? 2. Characterizing Tasks across interfaces: 1. desktop search 2. digital assistants 3. voice-only assistants 3. Understanding User Tasks in Web Search a. Extracting Query Intents b. Queries → Sessions → Tasks c. Search Task Understanding a. Task extraction b. Subtask extraction c. Hierarchies of tasks & subtasks d. Evaluating task extraction algorithms 5. Recommendation Systems a. Case study: Pinterest b. Case study: Spotify Take-home messages: • Understanding users helps in pushing other objectives without hurting key user metrics – Consumption diversity – User intents – User receptivity • Understanding suppliers helps develop right approaches to ensure supplier happiness: – Supplier diversity – Spillover effects • Content understanding allows us to know when to focus on what objectives 

Thank you! Rishabh Mehrotra Ben Carterette Senior Research Scientist, Senior Research Manager, Spotify, London Spotify, New York [email protected] [email protected] https://sites.google.com/view/kdd20-marketplace-autorecsys/