KDD 2020 Marketplace Tutorial Rishabh Part 1

Slide 1

Slide 1 text

KDD 2020 Tutorial: Advances in Recommender Systems Part A: Recommendations in a Marketplace [Introduction] Rishabh Mehrotra Ben Carterette Senior Research Scientist, Senior Research Manager, Spotify, London Spotify, New York [email protected] [email protected] 23rd August 2020 @erishabh @BenCarterette https://sites.google.com/view/kdd20-marketplace-autorecsys/

Slide 2

Slide 2 text

Introductions Rishabh Mehrotra Sr Research Scientist, Spotify, London Past: ● PhD in ML/IR from University College London ● BE (Hons) Computer Science, MSc (Hons) Mathematics, BITS Pilani, India ● Visiting researcher & intern, Microsoft Research, NYC/Redmond (2015-16) ● Co-founder, UserContext.AI ● Goldman Sachs Ben Carterette Sr Research Manager, Spotify, New York Past: ● Associate Professor, University of Delaware (2008 onwards) ● Visiting Researcher, CMU (2010) ● Visiting Researcher, Yahoo Labs (2009) ● PhD from University of Massachusetts Amherst (2008)

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Outline 1. Introduction: a. (Quick) Overview of traditional RecSys approaches b. Introduction to Marketplace c. Types & examples of marketplaces d. Recommendation in a marketplace e. Segway into the rest of the tutorial → walk-through of the upcoming sections → linking it all together 2. Optimization Objectives in a Marketplace 3. Methods for Multi-Objective Ranking & Recommendations 4. Leveraging Consumer, Supplier & Content Understanding 5. Industrial Applications

Slide 5

Slide 5 text

Outline 1. Introduction to Marketplaces 2. Optimization Objectives in a Marketplace a. Case studies I - VII b. Families of objectives c. Interplay between Objectives 3. Methods for Multi-Objective Ranking & Recommendations 4. Leveraging Consumer, Supplier & Content Understanding 5. Industrial Applications

Slide 6

Slide 6 text

Outline 1. Introduction to Marketplaces 2. Optimization Objectives in a Marketplace 3. Methods for Multi-Objective Ranking & Recommendations a. Pareto optimality b. Multi-objective models i. Scalarization ii. Multi-task Learning iii. Multi-objective bandits iv. Multi-objective RL 4. Leveraging Consumer, Supplier & Content Understanding 5. Industrial Applications

Slide 7

Slide 7 text

Outline 1. Introduction to Marketplaces 2. Optimization Objectives in a Marketplace 3. Methods for Multi-Objective Ranking & Recommendations 4. Leveraging Consumer, Supplier & Content Understanding a. Consumer Understanding: i. Consumption diversity of users ii. Quantifying and estimating user tolerance iii. Leveraging User intents b. Supplier Understanding: i. Diversity across suppliers c. Content Understanding: i. Personalizing Reward function ii. Query Understanding 5. Industrial Applications

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Schedule 08:00 - 08:10: Welcome + Introduction 08:10 - 08:30: Part I: Introduction to Marketplaces 08:30 - 09:00: Part II: Optimization Objectives in a Marketplace 09:00 - 09:30: Part III: Methods for Multi-Objective Recommendations 09:30 - 10:00: Break 10:00 - 10:30: Part III: Methods for Multi-Objective Recommendations 10:30 - 11:10: Part IV: Leveraging Consumer, Supplier & Content Understanding 11:10 - 11:40: Part V: Industrial Applications 11:40 - 11:50: Questions & Discussions

Slide 10

Slide 10 text

Outline 1. Introduction: a. (Quick) Overview of traditional RecSys approaches b. Introduction to Marketplace c. Types & examples of marketplaces d. Recommendation in a marketplace

Slide 11

Slide 11 text

Traditional RecSys Approaches

Slide 12

Slide 12 text

Approaches for RecSys Collaborative Filtering, i.e. matrix factorization

Slide 13

Slide 13 text

Approaches for RecSys Collaborative Filtering -- extended, i.e. Tensor factorization AAAI 2010: Collaborative Filtering Meets Mobile Recommendation: A User-Centered Approach

Slide 14

Slide 14 text

Approaches for RecSys Latent variable models RecSys 2015: A probabilistic model for using social networks in personalized item recommendation

Slide 15

Slide 15 text

Approaches for RecSys Neural Embeddings User Embedding

Slide 16

Slide 16 text

Approaches for RecSys Neural Embeddings User Embedding … with Side Information RecSys 2016: Meta-Prod2Vec - Product Embeddings Using Side-Information for Recommendation

Slide 17

Slide 17 text

Approaches for RecSys Neural Embeddings User Embedding … with Side Information Joint User-Item Embedding WSDM 2017: Joint Deep Modeling of Users and Items Using Reviews for Recommendation

Slide 18

Slide 18 text

Approaches for RecSys Neural Collaborative Ranking WWW 2017: Neural Collaborative Filtering

Slide 19

Slide 19 text

Approaches for RecSys Variants of Recommendation Styles: - Short vs long term - Cold start or cohort based - Multi-view & multi-interest models - Mult-task recommendation SIGIR 2012: Modeling the Impact of Short- and Long-Term Behavior on Search Personalization

Slide 20

Slide 20 text

Approaches for RecSys Variants of Recommendation Styles: - Short vs long term - Cold start & cohort based - Multi-view & multi-interest models - Mult-task recommendation SIGIR 2014: Cohort Modeling for Enhanced Personalized Search

Slide 21

Slide 21 text

Approaches for RecSys Variants of Recommendation Styles: - Short vs long term - Cold start or cohort based - Multi-view & multi-interest models - Mult-task recommendation RecSys 2013: Nonlinear Latent Factorization by Embedding Multiple User Interests

Slide 22

Slide 22 text

Approaches for RecSys Variants of Recommendation Styles: - Short vs long term - Cold start or cohort based - Multi-view & multi-interest models - Multi-task recommendation KDD 2018: Perceive Your Users in Depth: Learning Universal User Representations from Multiple E-commerce Tasks

Slide 23

Slide 23 text

Approaches for RecSys

Slide 24

Slide 24 text

Approaches for RecSys What do they have in common?

Slide 25

Slide 25 text

Approaches for RecSys What do they have in common? User centric focus

Slide 26

Slide 26 text

Traditional RecSys: User Centric ● User centric nature of systems: ○ Recommendations models catered to users: ■ user needs ■ user interests ■ user behavior & interactions ■ personalization ○ Evaluation approaches for user satisfaction ■ Measuring user engagement ■ Optimizing for user satisfaction ■ User centric metrics *WSDM 2018 Tutorial on metrics of user engagement; Mounia Lalmas, et al [link]

Slide 27

Slide 27 text

Is caring about the user enough?

Slide 28

Slide 28 text

Is caring about the user enough? Stakeholders

Slide 29

Slide 29 text

User centric methods is not sufficient to serve recommendations in such scenarios!

Slide 30

Slide 30 text

Introduction to Marketplaces

Slide 31

Slide 31 text

Marketplace Marketplace: Intermediaries that help facilitate economic interaction between two or more sets of agents

Slide 32

Slide 32 text

Marketplaces are everywhere! Marketplaces have existed since humans first began to engage in trade - Village markets: Vegetable & fruits & flowers - Shopping Malls - Stock markets

Slide 33

Slide 33 text

Modern Marketplace

Slide 34

Slide 34 text

Modern Marketplace

Slide 35

Slide 35 text

Modern Marketplace

Slide 36

Slide 36 text

Modern Marketplace

Slide 37

Slide 37 text

Modern Marketplace

Slide 38

Slide 38 text

Modern Marketplace

Slide 39

Slide 39 text

Modern Marketplace

Slide 40

Slide 40 text

Evolution of Marketplaces https://hackernoon.com/not-all-marketplaces-are-created-equal-tales-of-a-marketplace-founder-9fc0fb802706

Slide 41

Slide 41 text

Outline 1. Introduction: a. (Quick) Overview of traditional RecSys approaches b. Introduction to Marketplace c. Types & examples of marketplaces d. Recommendations in a marketplace

Slide 42

Slide 42 text

Marketplaces: Types + Examples

Slide 43

Slide 43 text

Marketplace Types Marketplaces can be differentiated based on many fronts: 1. Need/customer coverage (Horizontal vs Vertical) 2. Participants: who to whom? (B2B, C2C, etc) 3. Offerings classification (Information, Goods, Services, Funds) 4. Classification based on Control + Functionality (Closed, Open, Focused, Censored) 5. Management approach (Unmanaged, lightly managed, fully managed) 6. Transaction regularity (One-off, repeat transactions)

Slide 44

Slide 44 text

Marketplace Types: Need Coverage Horizontal platforms: ● facilitate exchanges in multiple different categories ○ e.g services, buying and selling stuff, etc Vertical platforms: ● focus on one problem / thing ○ E.g. Uber (driving), Airbnb (accommodation)

Slide 45

Slide 45 text

Marketplace Types: Participants Who to Whom? Participants on the online platform Platform Type Platform Use 1. B2B transaction of products or services between businesses 2. C2C transaction of products or services between customers 3. B2C transaction of products or services from business to customers 4. Crowdfunding lets people post projects and raise money through campaigns 5. eCommerce platform for two parties, e.g. seller - shopper, startup owner - investor 6. Peer-to-peer brings together users offering offline services 7. auction platforms seller lists a product; buyer with the highest bid gets the item https://syndicode.com/2017/06/28/types-of-online-marketplaces/

Slide 46

Slide 46 text

Marketplace Types Platform Type Platform Use 1. B2B transaction of products or services between businesses 2. C2C transaction of products or services between customers 3. B2C transaction of products or services from business to customers 4. Crowdfunding lets people post projects and raise money through campaigns 5. eCommerce platform for two parties, e.g. seller - shopper, startup owner - investor 6. Peer-to-peer brings together users offering offline services 7. auction platforms seller lists a product; buyer with the highest bid gets the item https://syndicode.com/2017/06/28/types-of-online-marketplaces/

Slide 47

Slide 47 text

Marketplace Types: Offerings http://wiki.rademade.com/4-typology-of-marketplaces

Slide 48

Slide 48 text

Marketplace Types: Control + Functionality Classification based on control + functionality: 1. Closed 3. Censored 2. Focused 4. Open http://wrap.warwick.ac.uk/76183/1/WRAP_JIT.r3_final.pdf

Slide 49

Slide 49 text

Marketplace Types: Management Type https://rubygarage.org/blog/types-of-online-marketplaces ● P2P transactions ● Reviews & ratings → Trust ● Lowest transaction fees

Slide 50

Slide 50 text

Marketplace Types: Management Type https://rubygarage.org/blog/types-of-online-marketplaces ● Protection for users ● Accurate content ● Technology tools

Slide 51

Slide 51 text

Marketplace Types: Management Type https://rubygarage.org/blog/types-of-online-marketplaces ● Operator mediates transactions ● Whole process managed ● Superior customer experience

Slide 52

Slide 52 text

Marketplace Types: Transaction Regularity

Slide 53

Slide 53 text

Success of marketplace ~ Matching consumers & suppliers

Slide 54

Slide 54 text

Outline 1. Introduction: a. (Quick) Overview of traditional RecSys approaches b. Introduction to Marketplace c. Types & examples of marketplaces d. Recommendations in a marketplace

Slide 55

Slide 55 text

Recommendations in a Marketplace

Slide 56

Slide 56 text

Recommendation in multi-sided Marketplace

Slide 57

Slide 57 text

Recommendation in multi-sided Marketplace Stakeholder(s) User Drivers Advertisers Campaign Hosts Platform provider

Slide 58

Slide 58 text

Recommendation in 2-sided Marketplace Stakeholder(s) User Drivers Advertisers Campaign(s) Platform provider Metrics Streams Engagement levels Reach / Depth / Retention Downstreams Other proxies of user satisfaction Exposure Audience growth Revenue LTV Diversity

Slide 59

Slide 59 text

Select an arm (i.e. card) Recommendation Strategy

Slide 60

Slide 60 text

Select an arm (i.e. card) Recommendation Strategy

Slide 61

Slide 61 text

Select an arm (i.e. card) Recommendation Strategy user-centric

Slide 62

Slide 62 text

User centric ML model is not meant to optimize for different objectives

Slide 63

Slide 63 text

Recommendation Strategy f( 1 , 2 , 3 , 4 ) Recommendation strategy = ??

Slide 64

Slide 64 text

Recommendation Strategy f( 1 , 2 , 3 , 4 )

Slide 65

Slide 65 text

Recommendation Strategy Select an arm (i.e. card) user-centric

Slide 66

Slide 66 text

user-centric supplier-centric platform economics Recommendation Strategy Select an arm (i.e. card)

Slide 67

Slide 67 text

user-centric artist-centric Spotify economics Recommendation Strategy Solution: find optimal recommendations which satisfy multiple objectives!

Slide 68

Slide 68 text

Marketplace powered Recommendations

Slide 69

Slide 69 text

Recommendations in a Marketplace SAT objectives User Understanding User Expectations

Slide 70

Slide 70 text

Recommendations in a Marketplace SAT objectives User Understanding User Expectations Supplier Exposure Supplier Goals

Slide 71

Slide 71 text

Recommendations in a Marketplace SAT objectives User Understanding User Expectations Supplier Exposure Supplier Goals Platform objectives Long Term Value

Slide 72

Slide 72 text

Recommendations in a Marketplace SAT objectives User Understanding User Expectations Supplier Exposure Supplier Goals Platform objectives Long Term Value Algorithmic balancing

Slide 73

Slide 73 text

Recommendations in a Marketplace SAT objectives User Understanding User Expectations Supplier Exposure Supplier Goals Platform objectives Long Term Value Algorithmic balancing User Understanding … Content Understanding Supplier Understanding

Slide 74

Slide 74 text

Recommendations in a Marketplace SAT objectives User Understanding User Expectations Supplier Exposure Supplier Goals Platform objectives Long Term Value Algorithmic balancing Evaluation Recommendations User Understanding … Content Understanding Supplier Understanding

Slide 75

Slide 75 text

Recommendations in a Marketplace Optimization Objectives What do different stakeholders care about? - User centric objectives - Supplier centric objectives - Platform goals & objectives

Slide 76

Slide 76 text

Recommendations in a Marketplace Multi-Objective Methods How do we make multiple considerations when recommending content? - Multiple predictions - Multi-objective models

Slide 77

Slide 77 text

Recommendations in a Marketplace Leveraging User & Content Understanding Consider user traits + stakeholder objectives to personalize: - Is the user open to diverse content? - Can we surface a tail supplier here?

Slide 78

Slide 78 text

Recommendations in a Marketplace Industrial Applications How different industries are developing their systems for multi-stakeholder needs?

Slide 79

Slide 79 text

Recommendations in a Marketplace ❏ Stakeholders & Objectives

Slide 80

Slide 80 text

Recommendations in a Marketplace ❏ Stakeholders & Objectives ❏ Methods for Multi-Objective Recommendations

Slide 81

Slide 81 text

Recommendations in a Marketplace ❏ Stakeholders & Objectives ❏ Methods for Multi-Objective Recommendations ❏ Leveraging User & Supplier Understanding

Slide 82

Slide 82 text

Recommendations in a Marketplace ❏ Stakeholders & Objectives ❏ Methods for Multi-Objective Recommendations ❏ Leveraging User & Supplier Understanding ❏ Industrial Applications

Slide 83

Slide 83 text

Part I: Introduction to Marketplaces 1. What is a task & why are they important? 2. Characterizing Tasks across interfaces: 1. desktop search 2. digital assistants 3. voice-only assistants 3. Understanding User Tasks in Web Search a. Extracting Query Intents b. Queries → Sessions → Tasks c. Search Task Understanding a. Task extraction b. Subtask extraction c. Hierarchies of tasks & subtasks d. Evaluating task extraction algorithms 5. Recommendation Systems a. Case study: Pinterest b. Case study: Spotify Take-home messages: • Traditional RecSys methods are majorly user-centric • Need to explicitly consider other stakeholders • Different types of marketplaces & examples • Components: – Multi-objective methods – User & Content understanding