KDD 2020 Tutorial: Advances in Recommender Systems Part A: Recommendations in a Marketplace [Stakeholder Objectives] Rishabh Mehrotra Ben Carterette Senior Research Scientist, Senior Research Manager, Spotify, London Spotify, New York [email protected] [email protected] 23rd August 2020 @erishabh @BenCarterette https://sites.google.com/view/kdd20-marketplace-autorecsys/ 

Recommendations in a Marketplace ❏ Stakeholders & Objectives 

Stakeholders in a Marketplace f( 1 , 2 , 3 , 4 ) 

Stakeholders & Objectives 1. Introduction to Marketplaces 2. Optimization Objectives in a Marketplace a. Case studies I - VII i. Stakeholders & their objectives b. Families of objectives c. Interplay between Objectives i. Correlation analysis ii. Supporting vs Competing objectives d. Some recent industrial applications i. Yahoo + LinkedIn + Etsy + Spotify 

Case Studies: Stakeholders + Objectives 

Case Study I: UberEats https://eng.uber.com/uber-eats-recommending-marketplace/ Uber Eats marketplace consists of three sides: ● Eaters ○ discover and order food through our platform ● Restaurant-partners ○ sales channel to find customers ● Delivery-partners ○ earn income by picking up food from restaurants and delivering it to eaters 

Case Study I: UberEats https://eng.uber.com/uber-eats-recommending-marketplace/ ● Not enough eaters → restaurants won’t participate ● Not enough restaurants → selection decreases ● Orders decrease → delivery partners earn less Unbalanced market demand-supply dynamic degrades the overall Uber Eats experience 

Case Study I: UberEats https://eng.uber.com/uber-eats-recommending-marketplace/ Relevant objectives: ● Eater conversion rate ● Diversity ○ Eaters can explore different types of food ● Exposure of restaurants ● Earning per delivery-partner ● Pick-up times & distances ● Delivery times 

Case Study II: On-Demand Delivery Solutions http://www.techgistics.net/blog/2016/11/30/theuber-for-x-model-and-the-complexity-of-on-demand-delivery 

Case Study II: On-Demand Delivery Solutions http://www.techgistics.net/blog/2016/11/30/theuber-for-x-model-and-the-complexity-of-on-demand-delivery Stakeholders ● Users ○ order anything from local partner shops ● Merchants ○ provides online visibility ● Delivery-partners ○ Work + earn on their own schedule 

Case Study II: On-Demand Delivery Solutions http://www.techgistics.net/blog/2016/11/30/theuber-for-x-model-and-the-complexity-of-on-demand-delivery Objectives ● Users ○ quick delivery + best prices + reliable merchants + fresh items ● Merchants ○ matching quality + exposure + minimize wastage ● Delivery-partners ○ regularity in jobs + earnings per partner + efficient drop location planning 

Case Study III: Etsy 

Case Study III: Etsy 3-sided marketplace ● Buyers ○ choose unique products among millions ○ discover products that they wouldn't buy at the first place? ○ recommend products for different occasions? ● Sellers ○ reach larger audience and potential buyers? ○ run advertising campaign more effectively? ● Platform ○ build a healthy platform? ○ speed-up buyer and seller communication? http://www.hongliangjie.com/talks/WSDM_2018-02-09.pdf 

Case Study IV: AirBnb 2-sided marketplace: ● Hosts: looking to rent their space ● Guests: looking for a place to stay Connecting hosts & guests at scale: ● 6M+ Airbnb listings worldwide ● 500M Airbnb guest arrivals ● 2M+ avg no of people staying/night ● 40K+ experiences worldwide 

Case Study IV: AirBnb Potential Objectives: ● Conversions (search → book) ○ Booked experiences + clicked but not booked ● No of bookings per host ○ Global ○ Tail-hosts ● Promote high quality bookings ● Discovering & promoting new hits earlier ● Diversity in top-8 ● Different objective for low intent users ● Helping hosts who host less often or come back from vacation https://medium.com/airbnb-engineering/machine-learning-powered-search-ranking-of-airbnb-experiences-110b4b1a0789 

Case Study V: Spotify ARTISTS FANS 

Case Study VI: P2P Lending https://www.fundingcircle.com/global/capitalmarkets/ Platform to connect investors to lend money directly to individuals and businesses 

Case Study VI: P2P Lending Funding Circle is a peer-to-peer lending marketplace that allows investors to lend money directly to small and medium-sized businesses https://www.fundingcircle.com/global/capitalmarkets/ 

Case Study VI: Funding Circle Potential Objectives: ● Quick + easy access to loans ● Return on investment ● Better matching of investors ←→ borrowers ● Lower loan default cases ● Increase borrowers 

Case Study VII: Crowdfunding 

Case Study VII: Crowdfunding Fancy tech vs health vs art $1000000 vs $7000 vs $3K Time sensitivity 

Case Study VII: Crowdfunding Potential Objectives: ● Time to success for a campaign ● Amount raised per campaign ● Interest matching for supporters ● No of campaigns donated to per supporter ● No of successful campaigns ● Success rate per category ● Time-sensitivity of campaign Li, Y., Rakesh, V., & Reddy, C. K. Project success prediction in crowdfunding environments. In Proceedings of WSDM 2016 

Stakeholders & Objectives 1. Introduction 2. Optimization Objectives in a Marketplace a. Case studies I - VII i. Stakeholders & their objectives b. Interplay between Objectives i. Correlation analysis ii. Supporting vs Competing objectives 

Families of Objectives 

Sea of Optimization Objectives Eater conversion rate Diversity Variety (in food) Exposure of restaurants Earning per delivery-partner Pick-up times & distances Delivery times Conversions (search → book) minimize wastage reach larger audiences regularity in jobs earnings per partner efficient drop location planning speed up matching Helping hosts who host less often or come back from vacation Quick + easy access to loans Return on investment Better matching of investors ←→ borrowers Lower loan default cases Increase borrowers Time to success for a campaign Booked experiences clicked but not booked No of bookings per host Global Tail-hosts Promote high quality bookings Discovering new hits Promoting new hits earlier rsity in top-8 Amount raised per campaign Interest matching for supporters No of campaigns donated to per supporter No of successful campaigns Success rate per category Time-sensitivity of campaign Different objective for low intent users quick delivery best prices reliable merchants fresh items Choose products Discovery matching quality exposure 

Families of Objectives Platform centric 04 ● Conversion rate ● Earnings per partner ● Platform health & sustainability Content aspects 03 ● Diversity ● Fairness ● Freshness Supplier Centric 02 ● Exposure ● Temporal (pick-up times, regularity in assignments) ● Earnings User Centric 01 ● Information need satisfied, Variety in recs ● Amount of funds raised ● Delivery times, Optimal prices 

How are these objectives computed? 

How are these objectives computed? ● Estimated via static computations: ○ Relevance: user-item similarity ○ Familiarity: user affinity to certain content ○ Time to success 

How are these objectives computed? ● Estimated via static computations: ○ Relevance, Familiarity, Affinity, Temporal ● Predicted via learned models: ○ p(click), p(consume) ○ p(save), p(return) 

How are these objectives computed? ● Estimated via static computations: ○ Relevance, Familiarity, Affinity, Temporal ● Predicted via learned models: ○ p(click), p(consume) ○ p(save), p(return) Single task models 

How are these objectives computed? ● Estimated via static computations: ○ Relevance, Familiarity, Affinity, Temporal ● Predicted via learned models: ○ p(skip), p(completion) ○ p(addToPlaylist), p(heart/ban) Single task models Multi-task models 

How are these objectives computed? ● Estimated via static computations: ○ Relevance, Familiarity, Affinity, Temporal ● Predicted via learned models: ○ p(click), p(consume) p(save), p(return) ● Factual (provided “as is”) ○ Margin based ○ Revenue deal based 

Interplay between Objectives 

Interplay between Objectives Relationships between objectives: ● Correlated ● Neutral ● Anti-correlated Optimizing for one, helps the other 

Interplay between Objectives Relationships between objectives: ● Correlated ● Neutral ● Anti-correlated Optimizing for one, helps the other Optimizing for one, doesn’t impact the other 

Interplay between Objectives Relationships between objectives: ● Correlated ● Neutral ● Anti-correlated Optimizing for one, helps the other Optimizing for one, doesn’t impact the other Optimizing for one, hurts the other 

Interplay between Objectives Consider 2 objectives: ● Relevance (to user) ● Fairness/diversity (of supplier) Mehrotra, McInerney, Bouchard, Lalmas, Diaz. Towards a Fair Marketplace: Counterfactual Evaluation of the trade-off between Relevance, Fairness & Satisfaction in Recommendation Systems. CIKM 2018. 

Interplay between Objectives Consider 2 objectives: ● Relevance (to user) ● Fairness/diversity (of supplier) Very few sets have both high relevance & high fairness of exposure Mehrotra, McInerney, Bouchard, Lalmas, Diaz. Towards a Fair Marketplace: Counterfactual Evaluation of the trade-off between Relevance, Fairness & Satisfaction in Recommendation Systems. CIKM 2018. 

Interplay between Objectives Consider 2 objectives: ● Relevance (to user) ● Fairness/diversity (of supplier) Very few sets have both high relevance & high fairness of exposure ● Conjecture: optimizing for relevance might hurt fairness Mehrotra, McInerney, Bouchard, Lalmas, Diaz. Towards a Fair Marketplace: Counterfactual Evaluation of the trade-off between Relevance, Fairness & Satisfaction in Recommendation Systems. CIKM 2018. 

Interplay between Objectives Consider other objectives: ● User SAT: ○ Stream time ○ Clicks ○ Items consumed ● Diversity ○ supplier diversity ● Promotional ○ promoting a set of suppliers 

Interplay between Objectives Different SAT metrics correlated with each other 

Interplay between Objectives Different SAT metrics correlated with each other Supplier diversity metric almost neutral with SAT metrics 

Interplay between Objectives Different SAT metrics correlated with each other Supplier diversity metric almost neutral with SAT metrics Promotional objective hurts SAT objectives 

Interplay between Objectives https://eng.uber.com/uber-eats-recommending-marketplace/ 

Interplay between Objectives ● Only optimizing for eater conversion → favors popular restaurants → hurts fairness ● Optimizing for marketplace fairness → recommends irrelevant restaurants → hurts eater conversation https://eng.uber.com/uber-eats-recommending-marketplace/ 

Interplay between Objectives https://medium.com/airbnb-engineering/machine-learning-powered-search-ranking-of-airbnb-experiences-110b4b1a0789 

Interplay between Objectives Optimizing for new hits → 14% booking gain for new hits → neutral overall bookings https://medium.com/airbnb-engineering/machine-learning-powered-search-ranking-of-airbnb-experiences-110b4b1a0789 

What a system optimizes is important! 

What a system optimizes is important! 

Part II: Optimization Objectives in Marketplace 1. What is a task & why are they important? 2. Characterizing Tasks across interfaces: 1. desktop search 2. digital assistants 3. voice-only assistants 3. Understanding User Tasks in Web Search a. Extracting Query Intents b. Queries → Sessions → Tasks c. Search Task Understanding a. Task extraction b. Subtask extraction c. Hierarchies of tasks & subtasks d. Evaluating task extraction algorithms 5. Recommendation Systems a. Case study: Pinterest b. Case study: Spotify Take-home messages: • Multiple stakeholders – Multiple objectives per stakeholder • Interplay between objectives: – Neutral / positive / negative • Careful consideration needed to decide which objectives to optimize for 

Schedule 08:00 - 08:10: Welcome + Introduction 08:10 - 08:30: Part I: Introduction to Marketplaces 08:30 - 09:00: Part II: Optimization Objectives in a Marketplace 09:00 - 09:30: Part III: Methods for Multi-Objective Recommendations 09:30 - 10:00: Break 10:00 - 10:30: Part III: Methods for Multi-Objective Recommendations 10:30 - 11:10: Part IV: Leveraging Consumer, Supplier & Content Understanding 11:10 - 11:40: Part V: Industrial Applications 11:40 - 11:50: Questions & Discussions