Beyond Accuracy: Goal-Driven Recommender Systems Design

Tamas Jambor @UCL talk at @ds_dln #strataconf 02/10/12

Transcript

1. BEYOND ACCURACY: GOAL-DRIVEN RECOMMENDER SYSTEM DESIGN Tamas Jambor (@jamborta) Data Scientist at Big Data Partnership (@bigdataexperts) Ph.D. student at UCL
2. Evaluating Recommender Systems • General error metrics • Mean absolute error (MAE) • Root mean squared error (RMSE) • General rank based metrics • Precision/Recall • Mean Reciprocal Rank (MRR) • Directly/indirectly optimised • What does improving these metrics mean?
3. Goal driven vs Metric driven design • Metric driven (error or rank based) • Metric(s) is abstract and general (e.g. RMSE) • Objective of the algorithm is abstract • Competitions (e.g. Kaggle, Netflix) encourage this approach • Goal driven • Metric(s) depends on the goal • More focused algorithm design • Fitting goal to data (not model to data)
4. Goal driven design • User/system perspective • Define and consider user satisfaction or system related objectives as priority • Internal/external goals • Consider algorithmic or non-algorithmic solutions • Time dependent goals • Identify various objectives and find the optimal solution
5. User/System perspective • System perspective • Performance • Cost • Update time • Profit margin • User perspective • User satisfaction (how to measure?) • Diversity • Novelty • Serendipity • Context
6. User vs System perspective Collaborative system-user goals Opposite system-user goals User- focused metric General metric System- focused metric System- focused metric User- focused metric General metric
7. External/Internal goal • External goals • System as a black box • Plugin any algorithm • Post/pre-filtering or independent algorithmic solution • Easier to evaluate (modularised) • Internal goals • Goal is built in the algorithm • Goal is directly optimised • Difficult to evaluate different components
8. Approach to goal-driven design Recommender system Internal goal External goal External goal data data prediction prediction
9. Time dependent goals • User side • Change in taste • Seasonal trends • System side • Cost over time (e.g. peak vs off-peak) • Demand
10. Time dependent algorithms • Internal • Exponential decay (model temporal change) • Survival analysis • Time-series model • External • Time sensitive post filtering • Control theory • Online learning approaches
11. Approach to goal-driven design Goal(s) User or System External or Internal Time dependency Performance Measures
12. User perspective Static Temporal Internal goals Diversification/long tail items (promote diverse items) Optimal Control Theory (cold-start problem) External goals Nudging and Serendipity (promote serendipitous items) Balanced Control Theory (improve prediction per user)
13. System perspective Static Temporal Internal goals Stock control (promote items that are in stock) Optimal Control Theory (estimate/maximise profit) External goals Optimised content delivery (pre-cache liked items) Balanced Control Theory (stabilise resource allocation)
14. Example: Diversification/long tail items Goal: Promote diverse items Challenges: How to measure diversity? Scope: Goal is optimised within the algorithm Algorithm: Matrix factorisation with convex optimisation Evaluation: Measure diverse items in top position
15. Example: Stock availability Goal: Recommend items that are in stock Challenges: Up-to-date stock availability? Scope: Goal is optimised within the algorithm Algorithm: Matrix factorisation with convex optimisation Evaluation: Measure waiting list for items
16. Example: Improve prediction per user Goal: Request more data for difficult users Use only useful data to train model Challenges: Define noise/signal for data points Scope: Goal is optimised over time, for each user independently Algorithm: Control theory Evaluation: Performance measured per user basis
17. Example: Resource allocation Goal: Improve resource allocation (e.g. CPU time) Maximise available resources (e.g. fixed cluster) Challenges: Define system dynamics Scope: Goal is optimised over time Algorithm: Control theory Evaluation: Stability, divergence from reference
18. Summary • Goal defines algorithm and performance measure • Multiple goals can be integrate into a single system • Modularity to evaluate goals separately (external) • Optimisation to merge goals (internal)