Learning to Rank with Multimodal Data

This is a talk I gave with Dr. Kamelia Aryafar on how we use techniques for learning to rank at Etsy from the theory applied to production pipeline and challenges from two perspectives: (i) a single modality model using traditional text-based features, and (ii) a multimodal approach that includes both text and visual clues from Etsy listings.

1. Learning to Rank with Multimodal Data @ Etsy NYC Machine Learning Meetup Kamelia Aryafar, Senior Data Scientist, @karyafar Nishan Subedi, Senior Software Engineer, @subedinishan Search Sciences, Etsy July 2016 1
2. Etsy
3. Etsy is a global marketplace where people around the world connect, both online and offline, to make, sell and buy unique goods. 3
4. By the Numbers 1.6M active sellers AS OF DECEMBER 31, 2015 24M active buyers AS OF DECEMBER 31, 2015 $2.39B annual GMS IN 2015 35+M items for sale AS OF DECEMBER 31, 2015 Photo by Kirsty-Lyn Jameson DISCLAIMER The statistics included on the following slides are updated quarterly.
5. 819 employees around the world AS OF DECEMBER 31, 2015 9 offices in 7 countries AS OF DECEMBER 31, 2015 Photo by Emily Andrews Work and Culture DISCLAIMER The statistics included on the following slides are updated quarterly.
6. Large and Unique Seller Base 1.6M active sellers AS OF SEPTEMBER 30, 2015 95% of sellers run their Etsy shop from home 2014 ETSY SELLER SURVEY 76% consider their shop a business 2014 ETSY SELLER SURVEY Photo by Moira K. Lime DISCLAIMER The statistics included on the following slides are updated quarterly.
7. Etsy Made in Canada Photo by Jean-Michael Seminaro 24M active buyers AS OF DECEMBER 31, 2015 92% of buyers agree Etsy offers products they can't find elsewhere 2014 ETSY BUYER SURVEY DISCLAIMER The statistics included on the following slides are updated quarterly.
11. Learning To Rank 11
12. Approaches to Learning to Rank • Pointwise - For an item, predict it’s grade (implicit ordering) - Labels come from interactions with items - Possible class imbalance • Pairwise - Ranking transformed to pairwise classification or regression - Labels depend on ordering of item pair - Ability to create balanced classes • Listwise - Input is entire set of documents associated with query - Output is their ranked list - Eg. Loss is a measure of the distance of ranking generated by the model to the perfect ranking for the set of documents
13. Pairwise Learning Each training instance represents a pair of items from same set of search results in your logs. <item1, item2> Learner must learn to order item1 and item2 correctly, with respect to user preference decisions found in your logs.
14. Features
15. Label Creation for Pairwise Features {housewarming, gift, photo} - {housewarming, gift, ceramic, tile} → +1 {housewarming, gift, ceramic, tile } - {housewarming, gift, photo} → -1
16. Train Classifier (SVM)
17. Learning to Rank Pipeline
18. Multimodal Learning to Rank
20. Image vs. Text Features 20 Texture Shape Color Title Tags
21. Extracting Image Features 21
22. title Feature Engineerings Deep Learning 22
23. ImageNet 23 Photo from : http://www.image-net.org/
24. Convolutional Neural Nets (CNNs) 24 Photo from: http://cs231n.stanford.edu/
25. VERY DEEP CONVOLUTIONAL NETWORKS FOR LARGE-SCALE IMAGE RECOGNITION Karen Simonyan & Andrew Zisserman Images Don’t Lie: Transferring Deep Visual Semantic Features to Large-Scale Multimodal Learning to Rank Corey Lynch, Kamelia Aryafar & Josh Attenberg Model Specs: VGGnet
26. Transfer Learning 26 Images Don’t Lie: Transferring Deep Visual Semantic Features to Large-Scale Multimodal Learning to Rank Corey Lynch, Kamelia Aryafar & Josh Attenberg
27. Photo by Corey Lynch
28. Photo by Corey Lynch
29. Images Don’t Lie: Transferring Deep Visual Semantic Features to Large-Scale Multimodal Learning to Rank Corey Lynch, Kamelia Aryafar & Josh Attenberg
30. Images Don’t Lie: Transferring Deep Visual Semantic Features to Large-Scale Multimodal Learning to Rank Corey Lynch, Kamelia Aryafar & Josh Attenberg
31. Images Don’t Lie: Transferring Deep Visual Semantic Features to Large-Scale Multimodal Learning to Rank Corey Lynch, Kamelia Aryafar & Josh Attenberg
32. 32 Images Don’t Lie: Transferring Deep Visual Semantic Features to Large-Scale Multimodal Learning to Rank Corey Lynch, Kamelia Aryafar & Josh Attenberg
33. Images Don’t Lie: Transferring Deep Visual Semantic Features to Large-Scale Multimodal Learning to Rank Corey Lynch, Kamelia Aryafar & Josh Attenberg
34. Images Don’t Lie: Transferring Deep Visual Semantic Features to Large-Scale Multimodal Learning to Rank Corey Lynch, Kamelia Aryafar & Josh Attenberg
35. Production Pipeline 35
36. Collecting Item Pairs with Labels 36
37. Strategies for Model Application Real Time Offline Pros Can handle unseen items No cost to feature complexity Cons Latency cost ∝ complexity Query time feature computations Computations compound as considerations increase
38. Real time model evaluation Fetch Model (cache, key-value store) Apply Ranking (ranking or reranking pass) User Query Top-k results Top-k results Reranked Index Ranking Model
39. Gaining Confidence
40. Gaining Confidence
41. Performance Replays
42. Performance Replays
43. Ranking Replays
44. Offline Evaluation Metrics
45. Model Understanding: Side by Side
46. Custom Queries & Explain Logs
47. The future…
48. Further Reading
49. etsy.com/careers
50. Thanks!