Complexity in Recommendations Systems Nishan Subedi Head Of Algorithms, VP of Technology Overstock.com

Goal: ● Highlight complexities with long running recommendations systems in production ● Highlight some approaches to mitigation of complexities ● Recommend experimentation practices ● Outline architecture and similarities to Learning To Rank ● Ecosystem effects of recommendations

No content

Overstock

Who am I ● Head of Algorithms, VP of Technology @ Overstock ● Products include: ○ Organic search (user interface, backend systems, machine learning) ○ Ads (auction, ranking, partner interface) ○ Recommendations Systems (machine learning & systems) ○ Pricing Science ○ CoreML (forecasts, estimates) ● Previously scientist @ Etsy on Ranking, experience with Systems Engineering

Complexity: ● Feedback & bias ○ Positional & presentation bias ○ Features reinforce biases ● Emergence ● Long term effects - deviation and drift ● ML Debt ○ Entanglement ○ Dependencies ○ Feature erosion ● Each learning algorithm can be considered an agent ● Measurements may not generalize

Experimentation and measurement challenges: ● Long-term experiments and holdouts have big engineering costs. ● Proxy metrics ● Hard to find HEART (Happiness, Engagement, Adoption, Retention, and Task-success) metrics for recommendations ● Heterogeneity in treatment effects ● Market effects: ensure similarity in market conditions between variants ● Interaction effects ● Learning about a model’s ability to learn quickly, not long term convergence

Recommendation system https://arxiv.org/abs/1907.12372

Formulation as LTR ● Re-ranking layer for recommendations ● Embedding based retrieval system ● Modeling the user and entities as embeddings ● Optimizing recommendations for specific goals ● Synthetic training data generation ● Leveraging session information https://arxiv.org/pdf/2006.11632.pdf https://static.googleusercontent.com/media/re search.google.com/en//pubs/archive/45530.p df

Recommender ecosystem

Recommender ecosystem Overall evaluation criterion (OEC)

Personalization ● Your platform is one part in the customer’s overall journey. ● Consistency in the customer’s journey ○ Consistent user representation ○ Unified training and production infrastructure ○ Using signals across multiple platforms ● Factoring the stage of the user’s journey ● Incorporating context from source

Thankyou! https://www.linkedin.com/in/nishan-subedi-0b7aa928/