KDD 2020 Marketplace Tutorial Rishabh Part 2

1. KDD 2020 Tutorial: Advances in Recommender Systems Part A: Recommendations in a Marketplace [Stakeholder Objectives] Rishabh Mehrotra Ben Carterette Senior Research Scientist, Senior Research Manager, Spotify, London Spotify, New York rishabhm@spotify.com benjaminc@spotify.com 23rd August 2020 @erishabh @BenCarterette https://sites.google.com/view/kdd20-marketplace-autorecsys/
2. Recommendations in a Marketplace ❏ Stakeholders & Objectives
3. Stakeholders in a Marketplace f( 1 , 2 , 3 , 4 )
4. Stakeholders & Objectives 1. Introduction to Marketplaces 2. Optimization Objectives in a Marketplace a. Case studies I - VII i. Stakeholders & their objectives b. Families of objectives c. Interplay between Objectives i. Correlation analysis ii. Supporting vs Competing objectives d. Some recent industrial applications i. Yahoo + LinkedIn + Etsy + Spotify
5. Case Studies: Stakeholders + Objectives
6. Case Study I: UberEats https://eng.uber.com/uber-eats-recommending-marketplace/ Uber Eats marketplace consists of three sides: • Eaters ◦ discover and order food through our platform • Restaurant-partners ◦ sales channel to find customers • Delivery-partners ◦ earn income by picking up food from restaurants and delivering it to eaters
7. Case Study I: UberEats https://eng.uber.com/uber-eats-recommending-marketplace/ • Not enough eaters → restaurants won’t participate • Not enough restaurants → selection decreases • Orders decrease → delivery partners earn less Unbalanced market demand-supply dynamic degrades the overall Uber Eats experience
8. Case Study I: UberEats https://eng.uber.com/uber-eats-recommending-marketplace/ Relevant objectives: • Eater conversion rate • Diversity ◦ Eaters can explore different types of food • Exposure of restaurants • Earning per delivery-partner • Pick-up times & distances • Delivery times
9. Case Study II: On-Demand Delivery Solutions http://www.techgistics.net/blog/2016/11/30/theuber-for-x-model-and-the-complexity-of-on-demand-delivery
10. Case Study II: On-Demand Delivery Solutions http://www.techgistics.net/blog/2016/11/30/theuber-for-x-model-and-the-complexity-of-on-demand-delivery Stakeholders • Users ◦ order anything from local partner shops • Merchants ◦ provides online visibility • Delivery-partners ◦ Work + earn on their own schedule
11. Case Study II: On-Demand Delivery Solutions http://www.techgistics.net/blog/2016/11/30/theuber-for-x-model-and-the-complexity-of-on-demand-delivery Objectives • Users ◦ quick delivery + best prices + reliable merchants + fresh items • Merchants ◦ matching quality + exposure + minimize wastage • Delivery-partners ◦ regularity in jobs + earnings per partner + efficient drop location planning
12. Case Study III: Etsy
13. Case Study III: Etsy 3-sided marketplace • Buyers ◦ choose unique products among millions ◦ discover products that they wouldn't buy at the first place? ◦ recommend products for different occasions? • Sellers ◦ reach larger audience and potential buyers? ◦ run advertising campaign more effectively? • Platform ◦ build a healthy platform? ◦ speed-up buyer and seller communication? http://www.hongliangjie.com/talks/WSDM_2018-02-09.pdf
14. Case Study IV: AirBnb 2-sided marketplace: • Hosts: looking to rent their space • Guests: looking for a place to stay Connecting hosts & guests at scale: • 6M+ Airbnb listings worldwide • 500M Airbnb guest arrivals • 2M+ avg no of people staying/night • 40K+ experiences worldwide
15. Case Study IV: AirBnb Potential Objectives: • Conversions (search → book) ◦ Booked experiences + clicked but not booked • No of bookings per host ◦ Global ◦ Tail-hosts • Promote high quality bookings • Discovering & promoting new hits earlier • Diversity in top-8 • Different objective for low intent users • Helping hosts who host less often or come back from vacation https://medium.com/airbnb-engineering/machine-learning-powered-search-ranking-of-airbnb-experiences-110b4b1a0789
16. Case Study V: Spotify ARTISTS FANS
17. Case Study VI: P2P Lending https://www.fundingcircle.com/global/capitalmarkets/ Platform to connect investors to lend money directly to individuals and businesses
18. Case Study VI: P2P Lending Funding Circle is a peer-to-peer lending marketplace that allows investors to lend money directly to small and medium-sized businesses https://www.fundingcircle.com/global/capitalmarkets/
19. Case Study VI: Funding Circle Potential Objectives: • Quick + easy access to loans • Return on investment • Better matching of investors ←→ borrowers • Lower loan default cases • Increase borrowers
20. Case Study VII: Crowdfunding
21. Case Study VII: Crowdfunding Fancy tech vs health vs art $1000000 vs $7000 vs $3K Time sensitivity
22. Case Study VII: Crowdfunding Potential Objectives: • Time to success for a campaign • Amount raised per campaign • Interest matching for supporters • No of campaigns donated to per supporter • No of successful campaigns • Success rate per category • Time-sensitivity of campaign Li, Y., Rakesh, V., & Reddy, C. K. Project success prediction in crowdfunding environments. In Proceedings of WSDM 2016
23. Stakeholders & Objectives 1. Introduction 2. Optimization Objectives in a Marketplace a. Case studies I - VII i. Stakeholders & their objectives b. Interplay between Objectives i. Correlation analysis ii. Supporting vs Competing objectives
24. Families of Objectives
25. Sea of Optimization Objectives Eater conversion rate Diversity Variety (in food) Exposure of restaurants Earning per delivery-partner Pick-up times & distances Delivery times Conversions (search → book) minimize wastage reach larger audiences regularity in jobs earnings per partner efficient drop location planning speed up matching Helping hosts who host less often or come back from vacation Quick + easy access to loans Return on investment Better matching of investors ←→ borrowers Lower loan default cases Increase borrowers Time to success for a campaign Booked experiences clicked but not booked No of bookings per host Global Tail-hosts Promote high quality bookings Discovering new hits Promoting new hits earlier rsity in top-8 Amount raised per campaign Interest matching for supporters No of campaigns donated to per supporter No of successful campaigns Success rate per category Time-sensitivity of campaign Different objective for low intent users quick delivery best prices reliable merchants fresh items Choose products Discovery matching quality exposure
26. Families of Objectives Platform centric 04 • Conversion rate • Earnings per partner • Platform health & sustainability Content aspects 03 • Diversity • Fairness • Freshness Supplier Centric 02 • Exposure • Temporal (pick-up times, regularity in assignments) • Earnings User Centric 01 • Information need satisfied, Variety in recs • Amount of funds raised • Delivery times, Optimal prices
27. How are these objectives computed?
28. How are these objectives computed? • Estimated via static computations: ◦ Relevance: user-item similarity ◦ Familiarity: user affinity to certain content ◦ Time to success
29. How are these objectives computed? • Estimated via static computations: ◦ Relevance, Familiarity, Affinity, Temporal • Predicted via learned models: ◦ p(click), p(consume) ◦ p(save), p(return)
30. How are these objectives computed? • Estimated via static computations: ◦ Relevance, Familiarity, Affinity, Temporal • Predicted via learned models: ◦ p(click), p(consume) ◦ p(save), p(return) Single task models
31. How are these objectives computed? • Estimated via static computations: ◦ Relevance, Familiarity, Affinity, Temporal • Predicted via learned models: ◦ p(skip), p(completion) ◦ p(addToPlaylist), p(heart/ban) Single task models Multi-task models
32. How are these objectives computed? • Estimated via static computations: ◦ Relevance, Familiarity, Affinity, Temporal • Predicted via learned models: ◦ p(click), p(consume) p(save), p(return) • Factual (provided “as is”) ◦ Margin based ◦ Revenue deal based
33. Interplay between Objectives
34. Interplay between Objectives Relationships between objectives: • Correlated • Neutral • Anti-correlated Optimizing for one, helps the other
35. Interplay between Objectives Relationships between objectives: • Correlated • Neutral • Anti-correlated Optimizing for one, helps the other Optimizing for one, doesn’t impact the other
36. Interplay between Objectives Relationships between objectives: • Correlated • Neutral • Anti-correlated Optimizing for one, helps the other Optimizing for one, doesn’t impact the other Optimizing for one, hurts the other
37. Interplay between Objectives Consider 2 objectives: • Relevance (to user) • Fairness/diversity (of supplier) Mehrotra, McInerney, Bouchard, Lalmas, Diaz. Towards a Fair Marketplace: Counterfactual Evaluation of the trade-off between Relevance, Fairness & Satisfaction in Recommendation Systems. CIKM 2018.
38. Interplay between Objectives Consider 2 objectives: • Relevance (to user) • Fairness/diversity (of supplier) Very few sets have both high relevance & high fairness of exposure Mehrotra, McInerney, Bouchard, Lalmas, Diaz. Towards a Fair Marketplace: Counterfactual Evaluation of the trade-off between Relevance, Fairness & Satisfaction in Recommendation Systems. CIKM 2018.
39. Interplay between Objectives Consider 2 objectives: • Relevance (to user) • Fairness/diversity (of supplier) Very few sets have both high relevance & high fairness of exposure • Conjecture: optimizing for relevance might hurt fairness Mehrotra, McInerney, Bouchard, Lalmas, Diaz. Towards a Fair Marketplace: Counterfactual Evaluation of the trade-off between Relevance, Fairness & Satisfaction in Recommendation Systems. CIKM 2018.
40. Interplay between Objectives Consider other objectives: • User SAT: ◦ Stream time ◦ Clicks ◦ Items consumed • Diversity ◦ supplier diversity • Promotional ◦ promoting a set of suppliers
41. Interplay between Objectives Different SAT metrics correlated with each other
42. Interplay between Objectives Different SAT metrics correlated with each other Supplier diversity metric almost neutral with SAT metrics
43. Interplay between Objectives Different SAT metrics correlated with each other Supplier diversity metric almost neutral with SAT metrics Promotional objective hurts SAT objectives
44. Interplay between Objectives https://eng.uber.com/uber-eats-recommending-marketplace/
45. Interplay between Objectives • Only optimizing for eater conversion → favors popular restaurants → hurts fairness • Optimizing for marketplace fairness → recommends irrelevant restaurants → hurts eater conversation https://eng.uber.com/uber-eats-recommending-marketplace/
46. Interplay between Objectives https://medium.com/airbnb-engineering/machine-learning-powered-search-ranking-of-airbnb-experiences-110b4b1a0789
47. Interplay between Objectives Optimizing for new hits → 14% booking gain for new hits → neutral overall bookings https://medium.com/airbnb-engineering/machine-learning-powered-search-ranking-of-airbnb-experiences-110b4b1a0789
48. What a system optimizes is important!
49. What a system optimizes is important!
50. Part II: Optimization Objectives in Marketplace 1. What is a task & why are they important? 2. Characterizing Tasks across interfaces: 1. desktop search 2. digital assistants 3. voice-only assistants 3. Understanding User Tasks in Web Search a. Extracting Query Intents b. Queries → Sessions → Tasks c. Search Task Understanding a. Task extraction b. Subtask extraction c. Hierarchies of tasks & subtasks d. Evaluating task extraction algorithms 5. Recommendation Systems a. Case study: Pinterest b. Case study: Spotify Take-home messages: • Multiple stakeholders – Multiple objectives per stakeholder • Interplay between objectives: – Neutral / positive / negative • Careful consideration needed to decide which objectives to optimize for
51. Schedule 08:00 - 08:10: Welcome + Introduction 08:10 - 08:30: Part I: Introduction to Marketplaces 08:30 - 09:00: Part II: Optimization Objectives in a Marketplace 09:00 - 09:30: Part III: Methods for Multi-Objective Recommendations 09:30 - 10:00: Break 10:00 - 10:30: Part III: Methods for Multi-Objective Recommendations 10:30 - 11:10: Part IV: Leveraging Consumer, Supplier & Content Understanding 11:10 - 11:40: Part V: Industrial Applications 11:40 - 11:50: Questions & Discussions